LuciferYang commented on code in PR #37648:
URL: https://github.com/apache/spark/pull/37648#discussion_r963302692
##########
resource-managers/yarn/src/test/scala/org/apache/spark/network/shuffle/ShuffleTestAccessor.scala:
##########
@@ -208,9 +208,12 @@ object ShuffleTestAccessor {
}
def reloadRegisteredExecutors(
- dbBackend: DBBackend,
- file: File): ConcurrentMap[ExternalShuffleBlockResolver.AppExecId,
ExecutorShuffleInfo] = {
- val db = DBProvider.initDB(dbBackend, file)
+ dbBackend: DBBackend,
+ file: File,
+ version: StoreVersion,
+ mapper: ObjectMapper)
+ : ConcurrentMap[ExternalShuffleBlockResolver.AppExecId,
ExecutorShuffleInfo] = {
+ val db = DBProvider.initDB(dbBackend, file, version, mapper)
Review Comment:
If the original case does not test the `execStateCopy exists but open
failed` scenario, it seems that there is no change.
For the `initDB(dbBackend, file) ` method:
https://github.com/apache/spark/blob/19b1780c4abed984b2223e6bbe7998b32a2ebad8/common/network-common/src/main/java/org/apache/spark/network/util/LevelDBProvider.java#L88-L94
- if `execStateCopy` exists, it will re-open it and load data from
`execStateCopy`
-if `execStateCopy` not exists, it will open a new db and return empty data
due to `options.createIfMissing(true)`, then assert failed
For the `initDB(dbBackend, dbFile, version, mapper) ` method:
- if `execStateCopy` exists, it will re-open it and load data from
`execStateCopy`
- else if `execStateCopy` not exists, it will open a new db and return
empty data, then assert failed
For `execStateCopy exists but open failed` scenario, use `initDB(dbBackend,
file)` will throw an unhandled exception,
use `initDB(dbBackend, dbFile, version, mapper) ` will assert failed. Is
this unacceptable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
