viirya commented on a change in pull request #30162:
URL: https://github.com/apache/spark/pull/30162#discussion_r513203243
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
##########
@@ -89,10 +89,15 @@ case class OffsetSeqMetadata(
object OffsetSeqMetadata extends Logging {
private implicit val format = Serialization.formats(NoTypeHints)
+ /**
+ * These configs are related to streaming query execution and should not be
changed across
+ * batches of a streaming query. The values of these configs are persisted
into the offset
+ * log in the checkpoint position.
+ */
private val relevantSQLConfs = Seq(
SHUFFLE_PARTITIONS, STATE_STORE_PROVIDER_CLASS,
STREAMING_MULTIPLE_WATERMARK_POLICY,
FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION,
STREAMING_AGGREGATION_STATE_FORMAT_VERSION,
- STREAMING_JOIN_STATE_FORMAT_VERSION)
+ STREAMING_JOIN_STATE_FORMAT_VERSION, STATE_STORE_COMPRESSION_CODEC)
Review comment:
Seems good if we possibly change default value unintentionally in the
future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
