This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 88b5bd0bda9b [SPARK-60862][SS] Fix watermark propagation through
reused exchanges
88b5bd0bda9b is described below
commit 88b5bd0bda9b7a8ac54aaf12e101b9e92a90ea9b
Author: Siying Dong <[email protected]>
AuthorDate: Thu Nov 5 09:47:13 2026 +0900
[SPARK-60862][SS] Fix watermark propagation through reused exchanges
### What changes were proposed in this pull request?
This patch makes the streaming watermark propagator treat
`ReusedExchangeExec` as a
transparent physical-plan wrapper. It traverses the reused exchange's
underlying child both
when simulating watermark propagation and when looking up an input
watermark. Because the
canonical exchange may already have visited the same hidden child, the
traversal is guarded by
the child's plan ID.
It also adds a `StreamingSQLQueryTestSuite` golden test with two union
branches that share the
same windowed aggregation exchange. After advancing the watermark, the test
verifies that both
the direct and reused branches emit their closed windows.
### Why are the changes needed?
`ReusedExchangeExec` is a leaf node whose underlying exchange is stored in
its `child` field,
not in the normal `children` sequence. The watermark simulator therefore
skipped the reused
subtree and fell back to a zero watermark. In a union of two equivalent
stateful aggregation
branches, the direct branch received the correct watermark while the reused
branch remained
pinned at zero. In append mode, that branch could fail to emit closed
windows and its state could
grow without bound, causing wrong results and unbounded state growth.
### Does this PR introduce _any_ user-facing change?
Yes. Streaming queries with a stateful operator under a reused exchange now
propagate the
correct watermark through that branch instead of using zero.
### How was this patch tested?
Added `streaming/reused-exchange-watermark.sql` with generated execution
and analyzer golden
files.
Added focused `MultiStatefulOperatorsSuite` cases with exchange reuse
enabled and disabled.
They verify the presence of `ReusedExchangeExec`, both operators' eviction
watermarks, emitted
rows, and retained state rows. A negative case independently applies two
watermarks to one
streaming DataFrame and verifies that their physical nodes and UUIDs remain
distinct and that
exchange reuse does not cross them.
Ran the streaming SQL golden test and the full
`MultiStatefulOperatorsSuite` (22 tests):
```text
build/mvn -pl sql/core \
-Dtest=NoSuchJavaTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dtests=streaming/reused-exchange-watermark.sql \
test
build/mvn -pl sql/core \
-Dtest=NoSuchJavaTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dsuites=org.apache.spark.sql.streaming.MultiStatefulOperatorsSuite \
test
```
Both the execution and analyzer tests passed. As a negative control,
removing the fix makes the
execution golden test fail because both rows from the reused `right` branch
are missing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
Closes #60073 from siying/codex/fix-reused-exchange-watermark.
Authored-by: Siying Dong <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../streaming/runtime/WatermarkPropagator.scala | 8 ++
.../streaming/reused-exchange-watermark.sql.out | 134 +++++++++++++++++++++
.../inputs/streaming/reused-exchange-watermark.sql | 53 ++++++++
.../streaming/reused-exchange-watermark.sql.out | 133 ++++++++++++++++++++
.../streaming/MultiStatefulOperatorsSuite.scala | 77 +++++++++++-
5 files changed, 404 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
index e573ea0e65ef..cd9e94f7372c 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
@@ -26,6 +26,7 @@ import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec,
QueryStageExec}
import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.exchange.ReusedExchangeExec
import
org.apache.spark.sql.execution.streaming.operators.stateful.{EventTimeWatermarkExec,
StateStoreWriter}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.util.Utils
@@ -181,6 +182,8 @@ class PropagateWatermarkSimulator extends
WatermarkPropagator with Logging {
case s: InMemoryTableScanExec =>
watermarkForChild(s.relation.cachedPlan)
case a: AdaptiveSparkPlanExec => watermarkForChild(a.executedPlan)
case e: QueryStageExec => watermarkForChild(e.plan)
+ // ReusedExchangeExec is a leaf whose output watermark is owned by its
hidden child.
+ case r: ReusedExchangeExec => watermarkForChild(r.child)
case _ =>
nodeToOutputWatermark.getOrElse(child.id, {
throw new IllegalStateException(
@@ -205,6 +208,11 @@ class PropagateWatermarkSimulator extends
WatermarkPropagator with Logging {
case s: InMemoryTableScanExec =>
traverseAndSimulate(s.relation.cachedPlan)
case a: AdaptiveSparkPlanExec => traverseAndSimulate(a.executedPlan)
case e: QueryStageExec => traverseAndSimulate(e.plan)
+ case r: ReusedExchangeExec =>
+ // Its hidden child may already have been visited through the
canonical exchange.
+ if (!nodeToOutputWatermark.contains(r.child.id)) {
+ traverseAndSimulate(r.child)
+ }
case node: EventTimeWatermarkExec =>
val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
if (inputWatermarks.nonEmpty) {
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
new file mode 100644
index 000000000000..176a16ead898
--- /dev/null
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
@@ -0,0 +1,134 @@
+-- Automatically generated by StreamingSQLQueryTestSuite
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_source
+-- !query analysis
+DropTable true, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog),
default.reused_exchange_watermark_source
+
+
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_sink
+-- !query analysis
+DropTable true, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog),
default.reused_exchange_watermark_sink
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_source (
+ event_time TIMESTAMP,
+ value INT) USING parquet
+-- !query analysis
+CreateDataSourceTableCommand
`spark_catalog`.`default`.`reused_exchange_watermark_source`, false
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:00', 0),
+ (TIMESTAMP '2024-01-01 00:00:01', 1),
+ (TIMESTAMP '2024-01-01 00:00:02', 2),
+ (TIMESTAMP '2024-01-01 00:00:03', 3),
+ (TIMESTAMP '2024-01-01 00:00:04', 4)
+-- !query analysis
+InsertIntoHadoopFsRelationCommand file:[not included in
comparison]/{warehouse_dir}/reused_exchange_watermark_source, false, Parquet,
[path=file:[not included in
comparison]/{warehouse_dir}/reused_exchange_watermark_source], Append,
`spark_catalog`.`default`.`reused_exchange_watermark_source`,
org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:[not included
in comparison]/{warehouse_dir}/reused_exchange_watermark_source), [event_time,
value]
++- Project [col1#x AS event_time#x, col2#x AS value#x]
+ +- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_sink (
+ branch STRING,
+ window_start TIMESTAMP,
+ key INT,
+ event_count BIGINT) USING parquet
+-- !query analysis
+CreateDataSourceTableCommand
`spark_catalog`.`default`.`reused_exchange_watermark_sink`, false
+
+
+-- !query
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+ SELECT
+ window(event_time, '5 seconds') AS window,
+ value % 2 AS key,
+ count(*) AS event_count
+ FROM STREAM(reused_exchange_watermark_source)
+ WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+ GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+-- !query analysis
+~WithCTE
+:- ~CTERelationDef xxxx, false
+: +- ~SubqueryAlias aggregated
+: +- ~Aggregate [window#x-T0ms, (value#x % 2)], [window#x-T0ms AS
window#x-T0ms, (value#x % 2) AS key#x, count(1) AS event_count#xL]
+: +- ~Project [named_struct(start,
knownnullable(precisetimestampconversion(((precisetimestampconversion(event_time#x-T0ms,
TimestampType, LongType) - CASE WHEN
(((precisetimestampconversion(event_time#x-T0ms, TimestampType, LongType) - 0)
% 5000000) < cast(0 as bigint)) THEN
(((precisetimestampconversion(event_time#x-T0ms, TimestampType, LongType) - 0)
% 5000000) + 5000000) ELSE ((precisetimestampconversion(event_time#x-T0ms,
TimestampType, LongType) - 0) % 5000000) END) - 0), Lo [...]
+: +- ~Filter isnotnull(event_time#x-T0ms)
+: +- ~EventTimeWatermark xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx,
event_time#x: timestamp, 0 seconds
+: +- ~SubqueryAlias
spark_catalog.default.reused_exchange_watermark_source
+: +- StreamingRelation FileSource[file:[not included in
comparison]/{warehouse_dir}/reused_exchange_watermark_source], [event_time#x,
value#x], name=<Unassigned>
++- ~Union false, false
+ :- ~Project [left AS branch#x, window#x-T0ms.start AS window_start#x,
key#x, event_count#xL]
+ : +- ~SubqueryAlias aggregated
+ : +- ~CTERelationRef xxxx, true, [window#x-T0ms, key#x,
event_count#xL], true, false
+ +- ~Project [branch#x AS branch#x, window_start#x AS window_start#x, key#x
AS key#x, event_count#xL AS event_count#xL]
+ +- ~Project [right AS branch#x, window#x-T0ms.start AS window_start#x,
key#x, event_count#xL]
+ +- ~SubqueryAlias aggregated
+ +- ~CTERelationRef xxxx, true, [window#x-T0ms, key#x,
event_count#xL], true, false
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query analysis
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query analysis
+Sort [branch#x ASC NULLS FIRST, key#x ASC NULLS FIRST], true
++- Project [branch#x, window_start#x, key#x, event_count#xL]
+ +- SubqueryAlias spark_catalog.default.reused_exchange_watermark_sink
+ +- Relation
spark_catalog.default.reused_exchange_watermark_sink[branch#x,window_start#x,key#x,event_count#xL]
parquet
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:10', 10)
+-- !query analysis
+InsertIntoHadoopFsRelationCommand file:[not included in
comparison]/{warehouse_dir}/reused_exchange_watermark_source, false, Parquet,
[path=file:[not included in
comparison]/{warehouse_dir}/reused_exchange_watermark_source], Append,
`spark_catalog`.`default`.`reused_exchange_watermark_source`,
org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:[not included
in comparison]/{warehouse_dir}/reused_exchange_watermark_source), [event_time,
value]
++- Project [col1#x AS event_time#x, col2#x AS value#x]
+ +- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query analysis
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query analysis
+Sort [branch#x ASC NULLS FIRST, key#x ASC NULLS FIRST], true
++- Project [branch#x, window_start#x, key#x, event_count#xL]
+ +- SubqueryAlias spark_catalog.default.reused_exchange_watermark_sink
+ +- Relation
spark_catalog.default.reused_exchange_watermark_sink[branch#x,window_start#x,key#x,event_count#xL]
parquet
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_source
+-- !query analysis
+DropTable false, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog),
default.reused_exchange_watermark_source
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_sink
+-- !query analysis
+DropTable false, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog),
default.reused_exchange_watermark_sink
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
b/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
new file mode 100644
index 000000000000..0941b9037017
--- /dev/null
+++
b/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
@@ -0,0 +1,53 @@
+-- SPARK-60862: A reused exchange must preserve the watermark of its
underlying child.
+--SET spark.sql.exchange.reuse=true
+--SET spark.sql.shuffle.partitions=1
+
+DROP TABLE IF EXISTS reused_exchange_watermark_source;
+DROP TABLE IF EXISTS reused_exchange_watermark_sink;
+
+CREATE TABLE reused_exchange_watermark_source (
+ event_time TIMESTAMP,
+ value INT) USING parquet;
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:00', 0),
+ (TIMESTAMP '2024-01-01 00:00:01', 1),
+ (TIMESTAMP '2024-01-01 00:00:02', 2),
+ (TIMESTAMP '2024-01-01 00:00:03', 3),
+ (TIMESTAMP '2024-01-01 00:00:04', 4);
+
+CREATE TABLE reused_exchange_watermark_sink (
+ branch STRING,
+ window_start TIMESTAMP,
+ key INT,
+ event_count BIGINT) USING parquet;
+
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+ SELECT
+ window(event_time, '5 seconds') AS window,
+ value % 2 AS key,
+ count(*) AS event_count
+ FROM STREAM(reused_exchange_watermark_source)
+ WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+ GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated;
+
+REFRESH FLOW reused_exchange_watermark_flow;
+
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key;
+
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:10', 10);
+
+REFRESH FLOW reused_exchange_watermark_flow;
+
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key;
+
+DROP TABLE reused_exchange_watermark_source;
+DROP TABLE reused_exchange_watermark_sink;
diff --git
a/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
b/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
new file mode 100644
index 000000000000..c6e077c484dd
--- /dev/null
+++
b/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
@@ -0,0 +1,133 @@
+-- Automatically generated by StreamingSQLQueryTestSuite
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_source
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_sink
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_source (
+ event_time TIMESTAMP,
+ value INT) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:00', 0),
+ (TIMESTAMP '2024-01-01 00:00:01', 1),
+ (TIMESTAMP '2024-01-01 00:00:02', 2),
+ (TIMESTAMP '2024-01-01 00:00:03', 3),
+ (TIMESTAMP '2024-01-01 00:00:04', 4)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_sink (
+ branch STRING,
+ window_start TIMESTAMP,
+ key INT,
+ event_count BIGINT) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+ SELECT
+ window(event_time, '5 seconds') AS window,
+ value % 2 AS key,
+ count(*) AS event_count
+ FROM STREAM(reused_exchange_watermark_source)
+ WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+ GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query schema
+struct<branch:string,window_start:timestamp,key:int,event_count:bigint>
+-- !query output
+
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+ (TIMESTAMP '2024-01-01 00:00:10', 10)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query schema
+struct<branch:string,window_start:timestamp,key:int,event_count:bigint>
+-- !query output
+left 2024-01-01 00:00:00 0 3
+left 2024-01-01 00:00:00 1 2
+right 2024-01-01 00:00:00 0 3
+right 2024-01-01 00:00:00 1 2
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_source
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_sink
+-- !query schema
+struct<>
+-- !query output
+
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
index 92d326234b96..e1a82e69544c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
@@ -22,7 +22,8 @@ import java.sql.Timestamp
import org.scalatest.BeforeAndAfter
import org.apache.spark.sql.{DataFrame, SparkSession}
-import
org.apache.spark.sql.execution.streaming.operators.stateful.StateStoreSaveExec
+import org.apache.spark.sql.execution.exchange.ReusedExchangeExec
+import
org.apache.spark.sql.execution.streaming.operators.stateful.{EventTimeWatermarkExec,
StateStoreSaveExec}
import
org.apache.spark.sql.execution.streaming.operators.stateful.join.StreamingSymmetricHashJoinExec
import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
import
org.apache.spark.sql.execution.streaming.state.{HDFSBackedStateStoreProvider,
RocksDBStateStoreProvider, StateStore}
@@ -53,6 +54,80 @@ class MultiStatefulOperatorsSuite
}
}
+ Seq(true, false).foreach { reuseEnabled =>
+ test("SPARK-60862: watermark propagates through reused exchange - " +
+ s"exchange reuse enabled: $reuseEnabled") {
+ withSQLConf(SQLConf.EXCHANGE_REUSE_ENABLED.key -> reuseEnabled.toString)
{
+ val inputData = MemoryStream[Int]
+
+ val aggregated = inputData.toDF()
+ .withColumn("eventTime", timestamp_seconds($"value"))
+ .withWatermark("eventTime", "0 seconds")
+ .groupBy(window($"eventTime", "5 seconds").as("window"), ($"value" %
2).as("key"))
+ .agg(count("*").as("count"))
+ .select($"window.start".cast("long").as("windowStart"), $"key",
$"count")
+
+ val stream = aggregated.select(lit("left").as("branch"), $"*")
+ .union(aggregated.select(lit("right").as("branch"), $"*"))
+ .as[(String, Long, Int, Long)]
+
+ testStream(stream)(
+ AddData(inputData, 0, 1, 2, 3, 4),
+ CheckNewAnswer(),
+ AddData(inputData, 10),
+ CheckNewAnswer(
+ ("left", 0L, 0, 3L), ("left", 0L, 1, 2L),
+ ("right", 0L, 0, 3L), ("right", 0L, 1, 2L)),
+ AssertOnQuery { query =>
+ val plan = query.lastExecution.executedPlan
+ val reusedExchanges = plan.collect { case r: ReusedExchangeExec =>
r }
+ assert(reusedExchanges.nonEmpty === reuseEnabled,
+ s"Expected ReusedExchangeExec only when reuse is enabled. Plan:
$plan")
+
+ val evictionWatermarks = plan.collect {
+ case s: StateStoreSaveExec => s.eventTimeWatermarkForEviction
+ }
+ assert(evictionWatermarks === Seq(Some(10000L), Some(10000L)))
+ true
+ },
+ assertNumStateRows(Seq(1, 1))
+ )
+ }
+ }
+ }
+
+ test("SPARK-60862: independently applied watermarks remain distinct") {
+ val inputData = MemoryStream[Int]
+ val source = inputData.toDF()
+ .withColumn("eventTime", timestamp_seconds($"value"))
+
+ val left = source
+ .withWatermark("eventTime", "0 seconds")
+ .select($"value".as("leftValue"), $"eventTime".as("leftTime"))
+ val right = source
+ .withWatermark("eventTime", "0 seconds")
+ .select($"value".as("rightValue"), $"eventTime".as("rightTime"))
+ val stream = left.join(right,
+ $"leftValue" === $"rightValue" && $"leftTime" === $"rightTime")
+ .select($"leftValue", $"rightValue")
+ .as[(Int, Int)]
+
+ testStream(stream)(
+ AddData(inputData, 1, 2),
+ CheckNewAnswer((1, 1), (2, 2)),
+ AssertOnQuery { query =>
+ val plan = query.lastExecution.executedPlan
+ val watermarkNodes = plan.collect { case w: EventTimeWatermarkExec =>
w }
+ assert(watermarkNodes.size === 2, s"Expected two watermark nodes.
Plan: $plan")
+ assert(watermarkNodes.map(_.nodeId).distinct.size === 2,
+ s"Expected distinct watermark node IDs. Plan: $plan")
+ assert(!plan.exists(_.isInstanceOf[ReusedExchangeExec]),
+ s"Exchange reuse must not cross distinct watermark nodes. Plan:
$plan")
+ true
+ }
+ )
+ }
+
test("window agg -> window agg, append mode") {
val inputData = MemoryStream[Int]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]