HeartSaVioR commented on code in PR #58562:
URL: https://github.com/apache/spark/pull/58562#discussion_r3947849035
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala:
##########
@@ -282,8 +284,40 @@ class StreamingQueryManager private[sql] (
catalogAndIdent,
catalogTable)
+ // Unsupported-operation checks run during this initial analysis, before
the normal offset-log
Review Comment:
To reviewer: This is the most unintuitive part of the code change; we should
retain backward compatibility via config in offset log, but the check is
happening on "initial analysis"
(`sparkSession.sessionState.executePlan(dataStreamWritePlan).analyzed`) which
is before the streaming engine reads the offset log and replaces the session
config with the config in the offset log.
That is why we need to do the similar trick here manually before doing the
initial analysis. To avoid the overhead of reading checkpoint, we scope it to
the case when the query has EXCEPT with streaming on the left side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
