Siying Dong created SPARK-60862:
-----------------------------------
Summary: Watermark propagation through reused exchanges uses zero
watermark
Key: SPARK-60862
URL: https://issues.apache.org/jira/browse/SPARK-60862
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Siying Dong
ReusedExchangeExec is a leaf physical plan node whose underlying exchange is
stored in its
child field rather than in the normal children sequence.
PropagateWatermarkSimulator does not
currently unwrap this node when traversing the plan or resolving an input
watermark.
As a result, when equivalent streaming aggregation branches reuse an exchange,
the direct
branch receives the correct event-time watermark while the reused branch
receives a zero
watermark. Downstream stateful operators also receive zero, delaying state
eviction and
watermark propagation.
The fix treats ReusedExchangeExec as a transparent wrapper in both watermark
traversal and
input-watermark lookup. A streaming SQL regression test covers a union of two
windowed
aggregation branches and verifies that both state stores receive the same
eviction watermark.
Pull request: https://github.com/apache/spark/pull/60073
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
