Xiao-zhen-Liu commented on code in PR #5900:
URL: https://github.com/apache/texera/pull/5900#discussion_r3493495409
##########
amber/src/main/python/core/models/state.py:
##########
@@ -25,13 +25,41 @@
class State(dict):
CONTENT = "content"
- SCHEMA = Schema(raw_schema={CONTENT: "STRING"})
+ # Loop-control bookkeeping owned by the worker runtime, NOT user state --
it
+ # never appears in the content JSON. In memory it rides on the StateFrame
+ # envelope; it is materialized/serialized as its own column (parallel to
+ # content) by to_tuple(...). from_tuple() returns the bare State; callers
+ # that need these values read the corresponding columns off the tuple.
+ LOOP_COUNTER = "loop_counter"
+ LOOP_START_ID = "loop_start_id"
+ LOOP_START_STATE_URI = "loop_start_state_uri"
Review Comment:
To clarify the question: it isn't whether `loop_start_id` can encode the URI
(agreed it can't — the URI is LoopStart's input-port path). It's whether that
write location should be in State at all.
The reason: the loop-back target is the same every iteration — same
LoopStart, same input port, same execution. So it's loop config that happens to
be constant, not per-iteration data, and State is the per-iteration payload.
A few alternatives, your call on which fits:
1. Give it to LoopEnd at setup, the way workers already receive their port
storage URIs from the scheduler (`set_up_port_storage_writer` / the mat-reader
setup). LoopEnd still does the write, but holds the address as config instead
of in every State row, and it never gets persisted to iceberg.
2. Let the controller resolve it at jump time. It already gets
`jump_to_operator_region(loop_start_id)`, and it owns the topology and
allocated the URIs, so it can resolve the target from the id and either do the
write or hand the URI back to LoopEnd.
3. Smaller option: carry it on the in-memory `StateFrame` only, never in the
persisted columns.
Not blocking this PR — for the loop PR. Which fits best, or is there a
constraint I'm missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
