This is an automated email from the ASF dual-hosted git repository.

HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 2bf50c585dba [SPARK-60862][SS] Fix watermark propagation through 
reused exchanges
2bf50c585dba is described below

commit 2bf50c585dba0a0507706032b4c8e570abdf5914
Author: Siying Dong <[email protected]>
AuthorDate: Thu Nov 5 09:47:13 2026 +0900

    [SPARK-60862][SS] Fix watermark propagation through reused exchanges
    
    ### What changes were proposed in this pull request?
    
    This patch makes the streaming watermark propagator treat 
`ReusedExchangeExec` as a
    transparent physical-plan wrapper. It traverses the reused exchange's 
underlying child both
    when simulating watermark propagation and when looking up an input 
watermark. Because the
    canonical exchange may already have visited the same hidden child, the 
traversal is guarded by
    the child's plan ID.
    
    It also adds a `StreamingSQLQueryTestSuite` golden test with two union 
branches that share the
    same windowed aggregation exchange. After advancing the watermark, the test 
verifies that both
    the direct and reused branches emit their closed windows.
    
    ### Why are the changes needed?
    
    `ReusedExchangeExec` is a leaf node whose underlying exchange is stored in 
its `child` field,
    not in the normal `children` sequence. The watermark simulator therefore 
skipped the reused
    subtree and fell back to a zero watermark. In a union of two equivalent 
stateful aggregation
    branches, the direct branch received the correct watermark while the reused 
branch remained
    pinned at zero. In append mode, that branch could fail to emit closed 
windows and its state could
    grow without bound, causing wrong results and unbounded state growth.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Streaming queries with a stateful operator under a reused exchange now 
propagate the
    correct watermark through that branch instead of using zero.
    
    ### How was this patch tested?
    
    Added `streaming/reused-exchange-watermark.sql` with generated execution 
and analyzer golden
    files.
    
    Added focused `MultiStatefulOperatorsSuite` cases with exchange reuse 
enabled and disabled.
    They verify the presence of `ReusedExchangeExec`, both operators' eviction 
watermarks, emitted
    rows, and retained state rows. A negative case independently applies two 
watermarks to one
    streaming DataFrame and verifies that their physical nodes and UUIDs remain 
distinct and that
    exchange reuse does not cross them.
    
    Ran the streaming SQL golden test and the full 
`MultiStatefulOperatorsSuite` (22 tests):
    
    ```text
    build/mvn -pl sql/core \
      -Dtest=NoSuchJavaTest \
      -Dsurefire.failIfNoSpecifiedTests=false \
      -Dtests=streaming/reused-exchange-watermark.sql \
      test
    
    build/mvn -pl sql/core \
      -Dtest=NoSuchJavaTest \
      -Dsurefire.failIfNoSpecifiedTests=false \
      -Dsuites=org.apache.spark.sql.streaming.MultiStatefulOperatorsSuite \
      test
    ```
    
    Both the execution and analyzer tests passed. As a negative control, 
removing the fix makes the
    execution golden test fail because both rows from the reused `right` branch 
are missing.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #60073 from siying/codex/fix-reused-exchange-watermark.
    
    Authored-by: Siying Dong <[email protected]>
    Signed-off-by: Jungtaek Lim <[email protected]>
    (cherry picked from commit 88b5bd0bda9b7a8ac54aaf12e101b9e92a90ea9b)
    Signed-off-by: Jungtaek Lim <[email protected]>
---
 .../streaming/runtime/WatermarkPropagator.scala    |   8 ++
 .../streaming/reused-exchange-watermark.sql.out    | 134 +++++++++++++++++++++
 .../inputs/streaming/reused-exchange-watermark.sql |  53 ++++++++
 .../streaming/reused-exchange-watermark.sql.out    | 133 ++++++++++++++++++++
 .../streaming/MultiStatefulOperatorsSuite.scala    |  77 +++++++++++-
 5 files changed, 404 insertions(+), 1 deletion(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
index e573ea0e65ef..cd9e94f7372c 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/WatermarkPropagator.scala
@@ -26,6 +26,7 @@ import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.execution.SparkPlan
 import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, 
QueryStageExec}
 import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
+import org.apache.spark.sql.execution.exchange.ReusedExchangeExec
 import 
org.apache.spark.sql.execution.streaming.operators.stateful.{EventTimeWatermarkExec,
 StateStoreWriter}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.util.Utils
@@ -181,6 +182,8 @@ class PropagateWatermarkSimulator extends 
WatermarkPropagator with Logging {
         case s: InMemoryTableScanExec => 
watermarkForChild(s.relation.cachedPlan)
         case a: AdaptiveSparkPlanExec => watermarkForChild(a.executedPlan)
         case e: QueryStageExec => watermarkForChild(e.plan)
+        // ReusedExchangeExec is a leaf whose output watermark is owned by its 
hidden child.
+        case r: ReusedExchangeExec => watermarkForChild(r.child)
         case _ =>
           nodeToOutputWatermark.getOrElse(child.id, {
             throw new IllegalStateException(
@@ -205,6 +208,11 @@ class PropagateWatermarkSimulator extends 
WatermarkPropagator with Logging {
         case s: InMemoryTableScanExec => 
traverseAndSimulate(s.relation.cachedPlan)
         case a: AdaptiveSparkPlanExec => traverseAndSimulate(a.executedPlan)
         case e: QueryStageExec => traverseAndSimulate(e.plan)
+        case r: ReusedExchangeExec =>
+          // Its hidden child may already have been visited through the 
canonical exchange.
+          if (!nodeToOutputWatermark.contains(r.child.id)) {
+            traverseAndSimulate(r.child)
+          }
         case node: EventTimeWatermarkExec =>
           val inputWatermarks = getInputWatermarks(node, nodeToOutputWatermark)
           if (inputWatermarks.nonEmpty) {
diff --git 
a/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
 
b/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
new file mode 100644
index 000000000000..176a16ead898
--- /dev/null
+++ 
b/sql/core/src/test/resources/sql-tests/analyzer-results/streaming/reused-exchange-watermark.sql.out
@@ -0,0 +1,134 @@
+-- Automatically generated by StreamingSQLQueryTestSuite
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_source
+-- !query analysis
+DropTable true, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog), 
default.reused_exchange_watermark_source
+
+
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_sink
+-- !query analysis
+DropTable true, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog), 
default.reused_exchange_watermark_sink
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_source (
+  event_time TIMESTAMP,
+  value INT) USING parquet
+-- !query analysis
+CreateDataSourceTableCommand 
`spark_catalog`.`default`.`reused_exchange_watermark_source`, false
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:00', 0),
+  (TIMESTAMP '2024-01-01 00:00:01', 1),
+  (TIMESTAMP '2024-01-01 00:00:02', 2),
+  (TIMESTAMP '2024-01-01 00:00:03', 3),
+  (TIMESTAMP '2024-01-01 00:00:04', 4)
+-- !query analysis
+InsertIntoHadoopFsRelationCommand file:[not included in 
comparison]/{warehouse_dir}/reused_exchange_watermark_source, false, Parquet, 
[path=file:[not included in 
comparison]/{warehouse_dir}/reused_exchange_watermark_source], Append, 
`spark_catalog`.`default`.`reused_exchange_watermark_source`, 
org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:[not included 
in comparison]/{warehouse_dir}/reused_exchange_watermark_source), [event_time, 
value]
++- Project [col1#x AS event_time#x, col2#x AS value#x]
+   +- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_sink (
+  branch STRING,
+  window_start TIMESTAMP,
+  key INT,
+  event_count BIGINT) USING parquet
+-- !query analysis
+CreateDataSourceTableCommand 
`spark_catalog`.`default`.`reused_exchange_watermark_sink`, false
+
+
+-- !query
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+  SELECT
+    window(event_time, '5 seconds') AS window,
+    value % 2 AS key,
+    count(*) AS event_count
+  FROM STREAM(reused_exchange_watermark_source)
+    WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+  GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+-- !query analysis
+~WithCTE
+:- ~CTERelationDef xxxx, false
+:  +- ~SubqueryAlias aggregated
+:     +- ~Aggregate [window#x-T0ms, (value#x % 2)], [window#x-T0ms AS 
window#x-T0ms, (value#x % 2) AS key#x, count(1) AS event_count#xL]
+:        +- ~Project [named_struct(start, 
knownnullable(precisetimestampconversion(((precisetimestampconversion(event_time#x-T0ms,
 TimestampType, LongType) - CASE WHEN 
(((precisetimestampconversion(event_time#x-T0ms, TimestampType, LongType) - 0) 
% 5000000) < cast(0 as bigint)) THEN 
(((precisetimestampconversion(event_time#x-T0ms, TimestampType, LongType) - 0) 
% 5000000) + 5000000) ELSE ((precisetimestampconversion(event_time#x-T0ms, 
TimestampType, LongType) - 0) % 5000000) END) - 0), Lo [...]
+:           +- ~Filter isnotnull(event_time#x-T0ms)
+:              +- ~EventTimeWatermark xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 
event_time#x: timestamp, 0 seconds
+:                 +- ~SubqueryAlias 
spark_catalog.default.reused_exchange_watermark_source
+:                    +- StreamingRelation FileSource[file:[not included in 
comparison]/{warehouse_dir}/reused_exchange_watermark_source], [event_time#x, 
value#x], name=<Unassigned>
++- ~Union false, false
+   :- ~Project [left AS branch#x, window#x-T0ms.start AS window_start#x, 
key#x, event_count#xL]
+   :  +- ~SubqueryAlias aggregated
+   :     +- ~CTERelationRef xxxx, true, [window#x-T0ms, key#x, 
event_count#xL], true, false
+   +- ~Project [branch#x AS branch#x, window_start#x AS window_start#x, key#x 
AS key#x, event_count#xL AS event_count#xL]
+      +- ~Project [right AS branch#x, window#x-T0ms.start AS window_start#x, 
key#x, event_count#xL]
+         +- ~SubqueryAlias aggregated
+            +- ~CTERelationRef xxxx, true, [window#x-T0ms, key#x, 
event_count#xL], true, false
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query analysis
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query analysis
+Sort [branch#x ASC NULLS FIRST, key#x ASC NULLS FIRST], true
++- Project [branch#x, window_start#x, key#x, event_count#xL]
+   +- SubqueryAlias spark_catalog.default.reused_exchange_watermark_sink
+      +- Relation 
spark_catalog.default.reused_exchange_watermark_sink[branch#x,window_start#x,key#x,event_count#xL]
 parquet
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:10', 10)
+-- !query analysis
+InsertIntoHadoopFsRelationCommand file:[not included in 
comparison]/{warehouse_dir}/reused_exchange_watermark_source, false, Parquet, 
[path=file:[not included in 
comparison]/{warehouse_dir}/reused_exchange_watermark_source], Append, 
`spark_catalog`.`default`.`reused_exchange_watermark_source`, 
org.apache.spark.sql.execution.datasources.InMemoryFileIndex(file:[not included 
in comparison]/{warehouse_dir}/reused_exchange_watermark_source), [event_time, 
value]
++- Project [col1#x AS event_time#x, col2#x AS value#x]
+   +- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query analysis
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query analysis
+Sort [branch#x ASC NULLS FIRST, key#x ASC NULLS FIRST], true
++- Project [branch#x, window_start#x, key#x, event_count#xL]
+   +- SubqueryAlias spark_catalog.default.reused_exchange_watermark_sink
+      +- Relation 
spark_catalog.default.reused_exchange_watermark_sink[branch#x,window_start#x,key#x,event_count#xL]
 parquet
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_source
+-- !query analysis
+DropTable false, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog), 
default.reused_exchange_watermark_source
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_sink
+-- !query analysis
+DropTable false, false
++- ResolvedIdentifier V2SessionCatalog(spark_catalog), 
default.reused_exchange_watermark_sink
diff --git 
a/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
 
b/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
new file mode 100644
index 000000000000..0941b9037017
--- /dev/null
+++ 
b/sql/core/src/test/resources/sql-tests/inputs/streaming/reused-exchange-watermark.sql
@@ -0,0 +1,53 @@
+-- SPARK-60862: A reused exchange must preserve the watermark of its 
underlying child.
+--SET spark.sql.exchange.reuse=true
+--SET spark.sql.shuffle.partitions=1
+
+DROP TABLE IF EXISTS reused_exchange_watermark_source;
+DROP TABLE IF EXISTS reused_exchange_watermark_sink;
+
+CREATE TABLE reused_exchange_watermark_source (
+  event_time TIMESTAMP,
+  value INT) USING parquet;
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:00', 0),
+  (TIMESTAMP '2024-01-01 00:00:01', 1),
+  (TIMESTAMP '2024-01-01 00:00:02', 2),
+  (TIMESTAMP '2024-01-01 00:00:03', 3),
+  (TIMESTAMP '2024-01-01 00:00:04', 4);
+
+CREATE TABLE reused_exchange_watermark_sink (
+  branch STRING,
+  window_start TIMESTAMP,
+  key INT,
+  event_count BIGINT) USING parquet;
+
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+  SELECT
+    window(event_time, '5 seconds') AS window,
+    value % 2 AS key,
+    count(*) AS event_count
+  FROM STREAM(reused_exchange_watermark_source)
+    WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+  GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated;
+
+REFRESH FLOW reused_exchange_watermark_flow;
+
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key;
+
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:10', 10);
+
+REFRESH FLOW reused_exchange_watermark_flow;
+
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key;
+
+DROP TABLE reused_exchange_watermark_source;
+DROP TABLE reused_exchange_watermark_sink;
diff --git 
a/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
 
b/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
new file mode 100644
index 000000000000..c6e077c484dd
--- /dev/null
+++ 
b/sql/core/src/test/resources/sql-tests/results/streaming/reused-exchange-watermark.sql.out
@@ -0,0 +1,133 @@
+-- Automatically generated by StreamingSQLQueryTestSuite
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_source
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+DROP TABLE IF EXISTS reused_exchange_watermark_sink
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_source (
+  event_time TIMESTAMP,
+  value INT) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:00', 0),
+  (TIMESTAMP '2024-01-01 00:00:01', 1),
+  (TIMESTAMP '2024-01-01 00:00:02', 2),
+  (TIMESTAMP '2024-01-01 00:00:03', 3),
+  (TIMESTAMP '2024-01-01 00:00:04', 4)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE TABLE reused_exchange_watermark_sink (
+  branch STRING,
+  window_start TIMESTAMP,
+  key INT,
+  event_count BIGINT) USING parquet
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+CREATE FLOW reused_exchange_watermark_flow AS
+INSERT INTO reused_exchange_watermark_sink BY NAME
+WITH aggregated AS (
+  SELECT
+    window(event_time, '5 seconds') AS window,
+    value % 2 AS key,
+    count(*) AS event_count
+  FROM STREAM(reused_exchange_watermark_source)
+    WATERMARK event_time DELAY OF INTERVAL 0 SECONDS
+  GROUP BY window(event_time, '5 seconds'), value % 2
+)
+SELECT 'left' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+UNION ALL
+SELECT 'right' AS branch, window.start AS window_start, key, event_count
+FROM aggregated
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query schema
+struct<branch:string,window_start:timestamp,key:int,event_count:bigint>
+-- !query output
+
+
+
+-- !query
+INSERT INTO reused_exchange_watermark_source VALUES
+  (TIMESTAMP '2024-01-01 00:00:10', 10)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+REFRESH FLOW reused_exchange_watermark_flow
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+SELECT * FROM reused_exchange_watermark_sink ORDER BY branch, key
+-- !query schema
+struct<branch:string,window_start:timestamp,key:int,event_count:bigint>
+-- !query output
+left   2024-01-01 00:00:00     0       3
+left   2024-01-01 00:00:00     1       2
+right  2024-01-01 00:00:00     0       3
+right  2024-01-01 00:00:00     1       2
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_source
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+DROP TABLE reused_exchange_watermark_sink
+-- !query schema
+struct<>
+-- !query output
+
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
index 92d326234b96..e1a82e69544c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
@@ -22,7 +22,8 @@ import java.sql.Timestamp
 import org.scalatest.BeforeAndAfter
 
 import org.apache.spark.sql.{DataFrame, SparkSession}
-import 
org.apache.spark.sql.execution.streaming.operators.stateful.StateStoreSaveExec
+import org.apache.spark.sql.execution.exchange.ReusedExchangeExec
+import 
org.apache.spark.sql.execution.streaming.operators.stateful.{EventTimeWatermarkExec,
 StateStoreSaveExec}
 import 
org.apache.spark.sql.execution.streaming.operators.stateful.join.StreamingSymmetricHashJoinExec
 import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
 import 
org.apache.spark.sql.execution.streaming.state.{HDFSBackedStateStoreProvider, 
RocksDBStateStoreProvider, StateStore}
@@ -53,6 +54,80 @@ class MultiStatefulOperatorsSuite
     }
   }
 
+  Seq(true, false).foreach { reuseEnabled =>
+    test("SPARK-60862: watermark propagates through reused exchange - " +
+        s"exchange reuse enabled: $reuseEnabled") {
+      withSQLConf(SQLConf.EXCHANGE_REUSE_ENABLED.key -> reuseEnabled.toString) 
{
+        val inputData = MemoryStream[Int]
+
+        val aggregated = inputData.toDF()
+          .withColumn("eventTime", timestamp_seconds($"value"))
+          .withWatermark("eventTime", "0 seconds")
+          .groupBy(window($"eventTime", "5 seconds").as("window"), ($"value" % 
2).as("key"))
+          .agg(count("*").as("count"))
+          .select($"window.start".cast("long").as("windowStart"), $"key", 
$"count")
+
+        val stream = aggregated.select(lit("left").as("branch"), $"*")
+          .union(aggregated.select(lit("right").as("branch"), $"*"))
+          .as[(String, Long, Int, Long)]
+
+        testStream(stream)(
+          AddData(inputData, 0, 1, 2, 3, 4),
+          CheckNewAnswer(),
+          AddData(inputData, 10),
+          CheckNewAnswer(
+            ("left", 0L, 0, 3L), ("left", 0L, 1, 2L),
+            ("right", 0L, 0, 3L), ("right", 0L, 1, 2L)),
+          AssertOnQuery { query =>
+            val plan = query.lastExecution.executedPlan
+            val reusedExchanges = plan.collect { case r: ReusedExchangeExec => 
r }
+            assert(reusedExchanges.nonEmpty === reuseEnabled,
+              s"Expected ReusedExchangeExec only when reuse is enabled. Plan: 
$plan")
+
+            val evictionWatermarks = plan.collect {
+              case s: StateStoreSaveExec => s.eventTimeWatermarkForEviction
+            }
+            assert(evictionWatermarks === Seq(Some(10000L), Some(10000L)))
+            true
+          },
+          assertNumStateRows(Seq(1, 1))
+        )
+      }
+    }
+  }
+
+  test("SPARK-60862: independently applied watermarks remain distinct") {
+    val inputData = MemoryStream[Int]
+    val source = inputData.toDF()
+      .withColumn("eventTime", timestamp_seconds($"value"))
+
+    val left = source
+      .withWatermark("eventTime", "0 seconds")
+      .select($"value".as("leftValue"), $"eventTime".as("leftTime"))
+    val right = source
+      .withWatermark("eventTime", "0 seconds")
+      .select($"value".as("rightValue"), $"eventTime".as("rightTime"))
+    val stream = left.join(right,
+      $"leftValue" === $"rightValue" && $"leftTime" === $"rightTime")
+      .select($"leftValue", $"rightValue")
+      .as[(Int, Int)]
+
+    testStream(stream)(
+      AddData(inputData, 1, 2),
+      CheckNewAnswer((1, 1), (2, 2)),
+      AssertOnQuery { query =>
+        val plan = query.lastExecution.executedPlan
+        val watermarkNodes = plan.collect { case w: EventTimeWatermarkExec => 
w }
+        assert(watermarkNodes.size === 2, s"Expected two watermark nodes. 
Plan: $plan")
+        assert(watermarkNodes.map(_.nodeId).distinct.size === 2,
+          s"Expected distinct watermark node IDs. Plan: $plan")
+        assert(!plan.exists(_.isInstanceOf[ReusedExchangeExec]),
+          s"Exchange reuse must not cross distinct watermark nodes. Plan: 
$plan")
+        true
+      }
+    )
+  }
+
   test("window agg -> window agg, append mode") {
     val inputData = MemoryStream[Int]
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to