HeartSaVioR commented on code in PR #54083:
URL: https://github.com/apache/spark/pull/54083#discussion_r2767437840
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -2498,6 +2512,17 @@ object RocksDBConf {
private val VERIFY_NON_EMPTY_FILES_IN_ZIP_CONF =
SQLConfEntry(VERIFY_NON_EMPTY_FILES_IN_ZIP_CONF_KEY, "true")
+ // Configuration to set the merge operator version for backward
compatibility.
+ // Version 1 (default): Uses comma "," as delimiter for StringAppendOperator
+ // Version 2: Uses empty string "" as delimiter (no delimiter, direct
concatenation)
+ //
+ // Note: this is also defined in
`SQLConf.STATE_STORE_ROCKSDB_MERGE_OPERATOR_VERSION`.
+ // These two places should be updated together.
Review Comment:
We need SQLConf to do the offset log trick. But in the meanwhile, it is
easier for us to have the same entry here to automatically take the conf into
account for RocksDBConf. We did this earlier -
`SQLConf.STATE_STORE_ROCKSDB_FORMAT_VERSION` & `FORMAT_VERSION`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
