Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21390#discussion_r189813156
--- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
---
@@ -732,6 +736,9 @@ private[deploy] class Worker(
trimFinishedExecutorsIfNecessary()
coresUsed -= executor.cores
memoryUsed -= executor.memory
+ if (CLEANUP_NON_SHUFFLE_FILES_ENABLED) {
+
shuffleService.executorRemoved(executorStateChanged.execId.toString, appId)
--- End diff --
Re-reading this code with fresh eyes made me realize that maybe the method
naming here is a bit confusing: from the name alone, it sounds like
`executorRemoved` is a notification-sending method, the kind of thing where
you'd always want to call it when the corresponding event (executor removal)
occurs and then leave it up to the method implementation to decide whether to
actually carry out specific cleanup logic.
On the other hand, we call `shuffleService.applicationRemoved()` in
`maybeCleanupApplication()` and there's a flag in that method controlling
whether we actually clean up, so we have a similar pattern there.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
