Yicong-Huang commented on issue #4545: URL: https://github.com/apache/texera/issues/4545#issuecomment-4333057858
One clarification to my root-cause note above: The regression is **not** that the reconfiguration ECM gets blocked by pause. In this path the ECM is carried in-band on the data channel, but `ECM_PAUSE` only pauses normal tuple intake until the aligned ECM is handled; the worker should still process the ECM itself while paused. What #4424 broke is the marker lifetime **after** that ECM is handled. `current_internal_marker` is no longer consumed atomically on the `DataProcessor` side, so the source's internal `EndChannel` marker remains visible across the `_switch_context()` round-trips inside `_process_end_channel`. After the reconfiguration ECM handler runs, that stale marker can be observed / re-entered again, which corrupts the end-of-stream handshake. That is why this reproduces on the Python-source propagation path rather than on the Scala/CSV-source cases: the ECM delivery during pause is behaving as designed here; the broken part is the marker consume/clear semantics that #4424 changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
