aglinxinyuan commented on code in PR #4488:
URL: https://github.com/apache/texera/pull/4488#discussion_r3165150403
##########
amber/src/main/python/core/runnables/network_receiver.py:
##########
@@ -96,7 +97,17 @@ def data_handler(command: bytes, table: Table) -> int:
"Data",
lambda _: DataFrame(table),
"State",
- lambda _: StateFrame(State(table)),
+ lambda _: StateFrame(
+ deserialize_state(
+ Tuple(
+ {
+ name: table[name][0].as_py()
Review Comment:
The batch size for State is intentionally always 1. In the common case,
states are emitted around tuple batches, for example, State, Tuples, State. In
the less common case where two states are adjacent, sending them as two
separate network payloads is still correct. So even if the stream looks like
Tuples, State, State, each State should still be carried as its own single-row
payload.
In other words, even though it is possible to send multiple states over the
network in a single table, there is no need to do so. Each State can be large,
so sending them individually is a reasonable design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
