chaoqin-li1123 commented on code in PR #47393: URL: https://github.com/apache/spark/pull/47393#discussion_r1681918789
########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreConf.scala: ########## @@ -41,6 +41,13 @@ class StateStoreConf( /** Minimum versions a State Store implementation should retain to allow rollbacks */ val minVersionsToRetain: Int = sqlConf.minBatchesToRetain + /** + * Minimum number of stale snapshot version files that need to be present Review Comment: Not necessarily a stale snapshot version because changelog version can also be cleaned up. _Minimum number of stale checkpoint versions that need to be present in the DFS checkpoint directory for old state checkpoint version deletion to be invoked. This is to amortize the cost of discovering and deleting old checkpoint versions._ ########## sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala: ########## @@ -146,6 +146,9 @@ class RocksDBFileManager( private def codec = CompressionCodec.createCodec(sparkConf, codecName) + private var maxVersion: Option[Long] = None Review Comment: Can we rename to maxSeenVersion and minSeenVersion to indicates that it is only an approximate? Also change the method name accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
