aglinxinyuan opened a new issue, #7248:
URL: https://github.com/apache/texera/issues/7248
### Describe the bug
A Loop Start MERGES any unstamped, counter-0 `State` it receives into its
loop variables (`LoopStartOperator.process_state` →
`self.state.update(state)`). That merge is load-bearing: the back-edge writes
the next iteration's variables to the Loop Start's own input-port state URI
with the same "no loop" envelope (`_jump_to_loop_start` → `State.to_tuple(0)`),
so the runtime cannot distinguish the loop's own state from someone else's.
The side effect is that **any** operator emitting boundary state via the
public `produce_state_on_start` / `produce_state_on_finish` API, upstream of or
inside a loop containing a Loop Start, silently injects its keys into that
loop's variables:
| Emitted key | Effect |
|---|---|
| collides with a loop variable (e.g. the default `i`) | silently overwrites
loop control — wrong iteration count, no error |
| `table` | aborts the loop with `_reserved_name_error` at
`produce_state_on_finish` |
```
Source ──▶ statefulOp ──(0, "")──▶ LoopStart ──▶ ... ──▶ LoopEnd
│
└─ self.state.update({"i": ...}) ←
clobbers the loop counter
```
### Where it comes from
`operator.py`'s `self.state.update(state)` dates to #5700. It is not created
by #6913, but #6913 makes operator-originated boundary state a first-class flow
through loops (the Scala side now emits it with an explicit envelope), so the
hazard is easier to reach.
Raised by @Xiao-zhen-Liu in review on #6913 (`#discussion_r3696858248`). The
Loop Start side is now documented and pinned there
(`test_loopstart_merges_unstamped_state_instead_of_forwarding_it`), but the
collision itself is undefended.
Note the Loop End side is NOT affected: its inbound loop state is always
stamped by the matching Loop Start, so it can tell the two apart and forwards a
body operator's boundary state instead of consuming it.
### Possible directions
1. Namespace the loop's own variables on the envelope (a producer tag on
`StateFrame`) so a Loop Start can tell its own state from an upstream
operator's, and reject/ignore the latter.
2. Reject a merge that would overwrite an existing loop variable, with a
message naming the colliding key.
3. Document it as the contract and add an e2e for `Source → statefulOp →
LoopStart`.
### Additional context
Related: #6913, #6660, #6661, #5700.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
