c21 commented on a change in pull request #30076:
URL: https://github.com/apache/spark/pull/30076#discussion_r507946803
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala
##########
@@ -151,7 +151,7 @@ case class StreamingSymmetricHashJoinExec(
stateWatermarkPredicates = JoinStateWatermarkPredicates(),
stateFormatVersion, left, right)
}
- if (stateFormatVersion < 2 && joinType != Inner) {
+ if (stateFormatVersion < 2 && (joinType == LeftOuter || joinType ==
RightOuter)) {
Review comment:
@HeartSaVioR - sorry I was under impression that we don't allow state
format version 1 after 3.0 without looking into the relevant code too much. But
I found it seems that we are having a config
[spark.sql.streaming.join.stateFormatVersion](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L1345),
that allows user to use version 1.
I will add the check to fail the left semi join query with state format 1 as
outer joins here. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
