vanzin commented on a change in pull request #26108: [SPARK-26154][SS]
Streaming left/right outer join should not return outer nulls for already
matched rows
URL: https://github.com/apache/spark/pull/26108#discussion_r339222389
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -139,13 +141,20 @@ case class StreamingSymmetricHashJoinExec(
rightKeys: Seq[Expression],
joinType: JoinType,
condition: Option[Expression],
+ stateFormatVersion: Int,
left: SparkPlan,
right: SparkPlan) = {
this(
leftKeys, rightKeys, joinType, JoinConditionSplitPredicates(condition,
left, right),
stateInfo = None, eventTimeWatermark = None,
- stateWatermarkPredicates = JoinStateWatermarkPredicates(), left, right)
+ stateWatermarkPredicates = JoinStateWatermarkPredicates(),
stateFormatVersion, left, right)
+ }
+
+ if (stateFormatVersion < 2 && joinType != Inner) {
+ logError(s"The query is using stream-stream outer join with state format
version" +
Review comment:
I see. I would suggest a config option to choose between error or continue,
but then I dislike adding more and more config options.
I guess the question is: in which situations will you have the new version
of Spark using the old snapshot format?
The way I understand it, it should only happen if you restart an app that
was running 2.4, now running with 3.0. The new query will pick up the state
store from the previous run and go from there.
In that situation it doesn't seem horrible to fail the query. But anyway,
I'll leave it to your judgement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
