Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5725#discussion_r29301391
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -207,7 +210,23 @@ private[spark] class Executor(
// Run the actual task and measure its runtime.
taskStart = System.currentTimeMillis()
- val value = task.run(taskAttemptId = taskId, attemptNumber =
attemptNumber)
+ var succeeded: Boolean = false
+ val value = try {
+ val value = task.run(taskAttemptId = taskId, attemptNumber =
attemptNumber)
+ succeeded = true
+ value
+ } finally {
+ // Release managed memory used by this task
+ val freedMemory = taskMemoryManager.cleanUpAllAllocatedMemory()
+ if (succeeded && freedMemory > 0) {
--- End diff --
I originally considered putting this in Task, but unfortunately
`runLocally` doesn't go through `Task`. We could try to consolidate both of
the code paths by using Task in `runLocally`, but I think this will be a larger
change (we'd need changes higher up the DAGScheduler call stack to create tasks
of the appropriate types, etc., and this might be hard to review).
As a compromise, I'm going to leave the duplicated code in place but add
comments so that readers are aware of the duplication and remember to change
both copies when making changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
