Github user kayousterhout commented on a diff in the pull request:
https://github.com/apache/spark/pull/1566#discussion_r15330087
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -710,7 +710,6 @@ class DAGScheduler(
if (missing == Nil) {
logInfo("Submitting " + stage + " (" + stage.rdd + "), which has
no missing parents")
submitMissingTasks(stage, jobId.get)
- runningStages += stage
--- End diff --
So just to clarify what's going on here: prior to my change, we added a
stage to runningStages here, after calling submitMissingTasks (so *after* the
code I modified below gets executed). This could lead to a memory leak (if the
stage needed to be aborted in submitMissingTasks, due to a
NotSerializableException for example, because then it would never be removed
from runningStages). It also meant that the DAGScheduler sent a
SparkListenerStageSubmitted event to the UI, but never a
SparkListenerStageCompleted (because, on line 1072, we only send a
SparkListenerStageCompleted event if the stage is in runningStages).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
