Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/19271#discussion_r139841304
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
---
@@ -114,6 +115,16 @@ class IncrementalExecution(
stateInfo = Some(nextStatefulOperationStateInfo),
batchTimestampMs = Some(offsetSeqMetadata.batchTimestampMs),
eventTimeWatermark = Some(offsetSeqMetadata.batchWatermarkMs))
+
+ case j @ StreamingSymmetricHashJoinExec(lKeys, rKeys, _, cond, _, _,
_, left, right) =>
+ j.copy(
+ stateInfo = Some(nextStatefulOperationStateInfo),
--- End diff --
That is the case **if** you don't change Spark versions. A more recent
Spark version may include new optimizer rules that may change the ordering.
Just something to think about. Would be nice to add a test with aggregation
+ join and join + aggregation.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
