[
https://issues.apache.org/jira/browse/SPARK-26097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-26097:
------------------------------------
Assignee: Apache Spark
> Show partitioning details in DAG UI
> -----------------------------------
>
> Key: SPARK-26097
> URL: https://issues.apache.org/jira/browse/SPARK-26097
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2, 2.4.0
> Reporter: Idan Zalzberg
> Assignee: Apache Spark
> Priority: Major
> Attachments: image (8).png
>
>
> We run complex SQL queries using Spark SQL, often we have to tackle a join
> skew or incorrect partition count. The problem is that while the Spark UI
> shows the existence of the problem and what *stage* it is part of, it's hard
> to infer back to the original SQL query that was given (e.g. what is the
> specific join operation that is actually skewed).
> One way to resolve this is to relate the Exchange nodes in the DAG to the
> partitioning that they represent, this is actually a trivial change in code
> (less than one line) that we believe can greatly benefit the research of
> performance issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
