[
https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takuya Ueshin updated SPARK-13902:
----------------------------------
Description:
{{DAGScheduler}} sometimes generate incorrect stage graph.
Some stages are generated for the same shuffleId twice or more and they are
referenced by the child stages because the building order of the graph is not
correct.
I added the sample RDD graph to show the illegal stage graph to
{{DAGSchedulerSuite}} and then fixed it.
was:
{{DAGScheduler}} sometimes generate correct stage graph.
Some stages are generated for the same shuffleId twice or more and they are
referenced by the child stages because the building order of the graph is not
correct.
I added the sample RDD graph to show the illegal stage graph to
{{DAGSchedulerSuite}} and then fixed it.
> Make DAGScheduler.getAncestorShuffleDependencies() return in topological
> order to ensure building ancestor stages first.
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-13902
> URL: https://issues.apache.org/jira/browse/SPARK-13902
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Reporter: Takuya Ueshin
>
> {{DAGScheduler}} sometimes generate incorrect stage graph.
> Some stages are generated for the same shuffleId twice or more and they are
> referenced by the child stages because the building order of the graph is not
> correct.
> I added the sample RDD graph to show the illegal stage graph to
> {{DAGSchedulerSuite}} and then fixed it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
