Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9084#discussion_r41826409
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala ---
@@ -73,15 +74,18 @@ class ShuffleMemoryManager protected (
// of active tasks, to let other tasks ramp down their memory in calls
to tryToAcquire
if (!taskMemory.contains(taskAttemptId)) {
taskMemory(taskAttemptId) = 0L
- notifyAll() // Will later cause waiting tasks to wake up and check
numTasks again
+ // This will later cause waiting tasks to wake up and check numTasks
again
+ memoryManager.notifyAll()
}
// Keep looping until we're either sure that we don't want to grant
this request (because this
// task would have more than 1 / numActiveTasks of the memory) or we
have enough free
// memory to give it (we always let each task get at least 1 / (2 *
numActiveTasks)).
+ // TODO: simplify this to limit each task to its own slot
--- End diff --
Can you elaborate on what changes this might require? Will we have to
effectively port the current cross-task accounting logic into the
StaticMemoryManager in order to preserve the existing behavior for legacy mode?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
