This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e58f12d1843 [SPARK-41411][SS] Multi-Stateful Operator watermark
support bug fix
e58f12d1843 is described below
commit e58f12d1843af39ed4ac0c2ff108490ae303e7e4
Author: Wei Liu <[email protected]>
AuthorDate: Wed Dec 7 08:35:21 2022 +0900
[SPARK-41411][SS] Multi-Stateful Operator watermark support bug fix
### What changes were proposed in this pull request?
Fix a typo in passing event time watermark
to`StreamingSymmetricHashJoinExec` that causes logic errors.
### Why are the changes needed?
Bug fix.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing unit tests.
Closes #38945 from WweiL/multi-stateful-ops-fix.
Authored-by: Wei Liu <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../org/apache/spark/sql/execution/streaming/IncrementalExecution.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
index 574709d05b0..e5e4dc7d0dc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
@@ -242,7 +242,7 @@ class IncrementalExecution(
j.copy(
stateInfo = Some(nextStatefulOperationStateInfo),
eventTimeWatermarkForLateEvents =
Some(eventTimeWatermarkForLateEvents),
- eventTimeWatermarkForEviction =
Some(eventTimeWatermarkForLateEvents),
+ eventTimeWatermarkForEviction = Some(eventTimeWatermarkForEviction),
stateWatermarkPredicates =
StreamingSymmetricHashJoinHelper.getStateWatermarkPredicates(
j.left.output, j.right.output, j.leftKeys, j.rightKeys,
j.condition.full,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
