SteNicholas commented on code in PR #3845:
URL: https://github.com/apache/celeborn/pull/3845#discussion_r4258722471
##########
client/src/main/scala/org/apache/celeborn/client/LifecycleManager.scala:
##########
@@ -1279,6 +1289,12 @@ class LifecycleManager(val appUniqueId: String, val
conf: CelebornConf) extends
}
}
}
+ if (celebornShuffleIdToAppShuffleIdMap.containsKey(shuffleId)) {
+ val appShuffleId = celebornShuffleIdToAppShuffleIdMap.get(shuffleId)
+ if (shuffleIdMapping.containsKey(appShuffleId) &&
registeredShuffle.contains(shuffleId)) {
+ expiredShuffleIds.add(shuffleId)
Review Comment:
@shlomitubul, retire existing expired markers during final shuffle cleanup.
The new guards prevent final unregistration from adding a marker, but they do
not remove markers already added by an earlier failed-copy cleanup. There is no
`remove` or `clear` for `expiredShuffleIds`, including in
`unregisterAppShuffle`, so a long-lived SparkContext retains one entry per
released copy across completed jobs indefinitely. A probe that allocates and
registers a mapped shuffle, calls `unregisterShuffle`, then calls
`unregisterAppShuffle(appShuffleId, true)` still finds the expired marker.
Please tie marker retention to the app-shuffle lifecycle and retire those
existing entries when they are no longer needed, coordinating with the
failed-shuffle cleaner so delayed cleanup cannot re-add them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
