WweiL commented on code in PR #48355:
URL: https://github.com/apache/spark/pull/48355#discussion_r1865967082
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -316,20 +346,45 @@ class RocksDBFileManager(
}
}
+ // Get latest snapshot version <= version
+ def getLatestSnapshotVersionAndUniqueId(
+ version: Long, checkpointUniqueId: Option[String] = None): Array[(Long,
Option[String])] = {
Review Comment:
This is from a similar way of how `getLatestSnapshotVersion` used in V1 is
checked. When `fm.exists(path)` returns false, that function returns 0. This is
because when first loading the path, the path does not exist. Here I use (0,
None) to represent because version 0 does not have checkpoint id.
Note that because of the code refactoring suggested in the comments, the
version = 0 part is specially handled in loadV2. But I feel as a utility
function it is better to still consider this situation in case in the future
people make any changes and are not aware of this fact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
