zhouyejoe commented on code in PR #35906:
URL: https://github.com/apache/spark/pull/35906#discussion_r923945575
##########
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:
##########
@@ -316,22 +355,39 @@ public String[] getMergedBlockDirs(String appId) {
@Override
public void applicationRemoved(String appId, boolean cleanupLocalDirs) {
logger.info("Application {} removed, cleanupLocalDirs = {}", appId,
cleanupLocalDirs);
- AppShuffleInfo appShuffleInfo = appsShuffleInfo.remove(appId);
+ // Cleanup the DB within critical section to gain the consistency between
+ // DB and in-memory hashmap.
+ AtomicReference<AppShuffleInfo> ref = new AtomicReference<>(null);
+ appsShuffleInfo.compute(appId, (id, info) -> {
+ if (null != info) {
+ // Try cleaning up this application attempt local paths information
+ // and also the local paths information from former attempts in DB.
+ removeAppAttemptPathInfoFromDB(info.appId, info.attemptId);
+ if (info.attemptId != UNDEFINED_ATTEMPT_ID) {
+ for (int formerAttemptId = info.attemptId - 1; formerAttemptId >= 0;
formerAttemptId--) {
Review Comment:
It happens rarely, only when the original deletion fails during the new app
attempt registration. I added db.get(key) check in the
removeAppAttemptPathInfoFromDB method, and only trigger the delete while the
return of db.get is not null. The get(key) method docs is not that good, but it
will just return Null, but will not throw NoSuchElementException, code
reference
https://github.com/apache/spark/blob/master/common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java#L131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
