viirya commented on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-717609418
> How about we follow `spark.eventLog.compression.codec`, > > ``` > The codec to compress logged events. If this is not given, spark.io.compression.codec will be used. > ``` > > so if `spark.sql.streaming.stateStore.compression.codec` is not set, we use `spark.io.compression.codec`. Basically the new `spark.sql.streaming.stateStore.compression.codec` is default to lz4 for backward-compatible. If we let it follow `spark.io.compression.codec`, it might be unintentionally changed other codec. If users read old state store, it will cause problem. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
