chaoqin-li1123 commented on code in PR #41099:
URL: https://github.com/apache/spark/pull/41099#discussion_r1193343612
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -129,14 +140,18 @@ class RocksDB(
* Note that this will copy all the necessary file from DFS to local disk as
needed,
* and possibly restart the native RocksDB instance.
*/
- def load(version: Long): RocksDB = {
+ def load(version: Long, readOnly: Boolean = false): RocksDB = {
assert(version >= 0)
acquire()
logInfo(s"Loading $version")
try {
if (loadedVersion != version) {
closeDB()
- val metadata = fileManager.loadCheckpointFromDfs(version, workingDir)
+ val loadSnapshotVersion = if (enableChangelogCheckpointing) {
Review Comment:
We should support switching in both direction. I took care of this in the
initial implementation but accidentally removed the forward compatibility when
addressing the comment about not listing to get the latest snapshot when
changelog checkpointing is disabled. Add back the logic to fallback to
replaying changelog when changelog checkpointing is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
