HeartSaVioR commented on a change in pull request #26108: [SPARK-26154][SS]
Streaming left/right outer join should not return outer nulls for already
matched rows
URL: https://github.com/apache/spark/pull/26108#discussion_r334712074
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1069,6 +1069,16 @@ object SQLConf {
.checkValue(v => Set(1, 2).contains(v), "Valid versions are 1 and 2")
.createWithDefault(2)
+ val STREAMING_JOIN_STATE_FORMAT_VERSION =
Review comment:
That is internal config and end users don't need to set this. This approach
has been picked to mark the value of config into metadata and restrict changing
the config once the query has been loaded from checkpoint. Please refer
FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION /
STREAMING_JOIN_STATE_FORMAT_VERSION to see how they handled.
When we deal with modification of schema for state, unless we reach a
consensus to let end users just discard checkpoint before continuing (that's
totally destructive so we tend to avoid this), we should construct a way to
deal with both schemas: that's why the concept of "versioning" comes out, and
additional interfaces/implementations are introduced to handle these versions.
So the actual code diff gets much bigger than what the patch is originally
trying to fix.
We might still be able to reduce the code diff once more via replacing new
interface/implementations with if-else, but well... I wish I can avoid it if
possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
