Takuya Ueshin created SPARK-14269:
-------------------------------------
Summary: Eliminate unnecessary submitStage() call.
Key: SPARK-14269
URL: https://issues.apache.org/jira/browse/SPARK-14269
Project: Spark
Issue Type: Improvement
Components: Scheduler
Reporter: Takuya Ueshin
Currently a method {{submitStage()}} for waiting stages is called on every
iteration of the event loop in {{DAGScheduler}} to submit all waiting stages,
but most of them are not necessary because they are not related to Stage status.
The case we should try to submit waiting stages is only when their parent
stages are successfully completed.
This elimination can improve {{DAGScheduler}} performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
