Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21390#discussion_r189808705
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -211,6 +211,26 @@ public void applicationRemoved(String appId, boolean
cleanupLocalDirs) {
}
}
+ /**
+ * Removes all the non-shuffle files in any local directories associated
with the finished
+ * executor.
+ */
+ public void executorRemoved(String executorId, String appId) {
+ logger.info("Clean up non-shuffle files associated with the finished
executor {}", executorId);
+ AppExecId fullId = new AppExecId(appId, executorId);
+ final ExecutorShuffleInfo executor = executors.get(fullId);
+ if (executor == null) {
+ // Executor not registered, skip clean up of the local directories.
+ logger.info("Executor is not registered (appId={}, execId={})",
appId, executorId);
+ } else {
+ logger.info("Cleaning up non-shuffle files in executor {}'s {} local
dirs", fullId,
+ executor.localDirs.length);
+
+ // Execute the actual deletion in a different thread, as it may take
some time.
--- End diff --
ð , thanks for doing this in a separate thread. This indeed could take a
_very_ long time under certain rare conditions.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
