Github user kayousterhout commented on the pull request:
https://github.com/apache/spark/pull/1516#issuecomment-49968180
I looked into this a bit more and this change doesn't quite fix the problem
in the right way, partially because of what @markhamstra pointed out (that we
shouldn't send a SparkListenerStageCompleted event before sending a
corresponding SparkListenerStageSubmitted event, which is I think what the
mysterious comment was getting at) and partially because the bigger underlying
problem here is that runningStages isn't updated at the right time (which also
leads to a memory leak). I submitted an alternate fix here:
https://github.com/apache/spark/pull/1566 -- let me know what you all think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
