viirya commented on pull request #30162: URL: https://github.com/apache/spark/pull/30162#issuecomment-717630958
> That said, you'll need to either 1) make it as a configuration but prevent the value to be changed after the query starts (like we do in state store formats) or 2) add the information to the separate metadata file and let state store read it. Probably even for the case of 2) you'll want to prevent the compression codec to be changed across the lifetime of the query - if you allow arbitrary changes of the compression codec across batches, these information should be written and referenced which would become non-trivial overhead. This is good point. I'd like to make it simple so we don't allow to change the value after the query starts. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
