HeartSaVioR commented on a change in pull request #30162:
URL: https://github.com/apache/spark/pull/30162#discussion_r513180738
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
##########
@@ -89,10 +89,15 @@ case class OffsetSeqMetadata(
object OffsetSeqMetadata extends Logging {
private implicit val format = Serialization.formats(NoTypeHints)
+ /**
+ * These configs are related to streaming query execution and should not be
changed across
+ * batches of a streaming query. The values of these configs are persisted
into the offset
+ * log in the checkpoint position.
+ */
private val relevantSQLConfs = Seq(
SHUFFLE_PARTITIONS, STATE_STORE_PROVIDER_CLASS,
STREAMING_MULTIPLE_WATERMARK_POLICY,
FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION,
STREAMING_AGGREGATION_STATE_FORMAT_VERSION,
- STREAMING_JOIN_STATE_FORMAT_VERSION)
+ STREAMING_JOIN_STATE_FORMAT_VERSION, STATE_STORE_COMPRESSION_CODEC)
Review comment:
I'd also recommend to add the default value ("lz4") in
`relevantSQLConfDefaultValues` to make sure we don't make any possible mistake.
(That's a sort of defensive programming though.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
