LuciferYang commented on code in PR #37826:
URL: https://github.com/apache/spark/pull/37826#discussion_r967386144
##########
common/network-common/src/main/java/org/apache/spark/network/util/LevelDBProvider.java:
##########
@@ -85,14 +84,6 @@ public static DB initLevelDB(File dbFile, StoreVersion
version, ObjectMapper map
return tmpDb;
}
- @VisibleForTesting
Review Comment:
When I wanted to create some failed cases to compare differences before and
after this pr, I found an interesting thing:
There are 3 cases are related to the code we discussed:
`YarnShuffleIntegrationWithLevelDBBackendSuite`,
`YarnShuffleAuthWithLevelDBBackendSuite`,
`YarnShuffleAlternateNameConfigWithLevelDBBackendSuite`.
But when I add `println(s"registeredExecFile = $registeredExecFile")` after
`val registeredExecFile =
YarnTestAccessor.getRegisteredExecutorFile(shuffleService)(line 73 as follow
code)`
https://github.com/apache/spark/blob/5a599dec507786139fb2ecb7ce1a44c83fd06b0d/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnShuffleIntegrationSuite.scala#L69-L86
I found all 3 case print `registeredExecFile = null`, so the code we
discussed was not actually executed due to registeredExecFile is always null
....
I also test these 3 cases use Spark 3.3, The problem also
exists(registeredExecFile = null), I haven't found out which version of this
code began to not execute, but this may be a very old bug.
@tgravescs @mridulm do you know when `YarnShuffleService` should call the
`setRecoveryPath` method? It seems that these tests did not call the
`setRecoveryPath` before serviceInit, so `registeredExecFile` is always null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
