HeartSaVioR commented on code in PR #49983:
URL: https://github.com/apache/spark/pull/49983#discussion_r1957767351
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala:
##########
@@ -180,5 +180,16 @@ object OffsetSeqMetadata extends Logging {
}
}
}
+
+ // SPARK-51187: This incorrect config is not added in the
relevantSQLConfs, but the
+ // metadata in the offset log may have this if the batch ran from Spark
3.5.4.
+ // We need to pick the value from the metadata and set it in the new
config.
+ // This also leads the further batches to have a correct config in the
offset log.
+
metadata.conf.get("spark.databricks.sql.optimizer.pruneFiltersCanPruneStreamingSubplan")
match {
Review Comment:
I'll add TODO with mentioning removal when we want to file JIRA ticket for
removal immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
