[
https://issues.apache.org/jira/browse/SPARK-24274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698370#comment-16698370
]
Huon Wilson commented on SPARK-24274:
-------------------------------------
I think part of the problem was that the order of the stages seemed to just be
random (based on [a hashset of the stages associated with a
job|https://github.com/apache/spark/blob/94145786a5b91a7f0bca44f27599a61c72f3a18f/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L143]).
However, the stages form a DAG themselves, and there's even a natural ordering
(the stage number). https://github.com/apache/spark/pull/21680 (which seems to
be in 2.3.2 and 2.4.0) changed the ordering from random to be sorted by the
stage number, which has simplified complicated stage graphs (similar to the one
attached above) that appear in my own code. There's much fewer crossing
inter-stage edges, and stages are better grouped so identifying inputs is
easier.
> Job UI should make stage dependencies clear in a complex DAG
> ------------------------------------------------------------
>
> Key: SPARK-24274
> URL: https://issues.apache.org/jira/browse/SPARK-24274
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.3.0
> Reporter: Imran Rashid
> Priority: Major
> Attachments: q95_job17.tgz
>
>
> When there is a complex dag, its really hard to tell in the UI the
> relationship between the stages in a job, even with the "DAG Visualization".
> I'll attach an example page from tpc-ds q95 -- its really hard to tell what
> the inputs are to stage 19.
> its actually extremely useful to know the relationship between the stage,
> especially when there is a failure. I want to know what the total shuffle
> input *would* have been to stage 19, even though it failed. If I could tell
> what the input stages were, then this would be easy.
> I dunno the best way to do this -- some ideas:
> * tooltips on the DAG
> * a list of parents with links on the stage page
> * stage tables on the job page could include a list of parents
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
