Jose Torres created SPARK-23703:
-----------------------------------
Summary: Collapse sequential watermarks
Key: SPARK-23703
URL: https://issues.apache.org/jira/browse/SPARK-23703
Project: Spark
Issue Type: Sub-task
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Jose Torres
When there are two sequential EventTimeWatermark nodes in a query plan, the
topmost one overrides the column tracking metadata from its children, but
leaves the nodes themselves untouched. When there is no intervening stateful
operation to consume the watermark, we should remove the lower node entirely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
