[
https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated SPARK-35639:
-----------------------------------
Description:
{{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number
of skewed partition splits".
It would be useful to also report "number of partitions to coalesce" and
"number of coalesced partitions" and include this in string rendering of the
SparkPlan node so that it looks like this
{code:java}
(12) CustomShuffleReader
Input [2]: [a#23, b#24]
Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
{code}
For accurate description see
https://github.com/apache/spark/pull/32776#issue-661274727
was:
{{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number
of skewed partition splits".
It would be useful to also report "number of partitions to coalesce" and
"number of coalesced partitions" and include this in string rendering of the
SparkPlan node so that it looks like this
{code:java}
(12) CustomShuffleReader
Input [2]: [a#23, b#24]
Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
{code}
> Add metrics about coalesced partitions to CustomShuffleReader in AQE
> --------------------------------------------------------------------
>
> Key: SPARK-35639
> URL: https://issues.apache.org/jira/browse/SPARK-35639
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Eugene Koifman
> Assignee: Eugene Koifman
> Priority: Major
> Fix For: 3.2.0
>
>
> {{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number
> of skewed partition splits".
> It would be useful to also report "number of partitions to coalesce" and
> "number of coalesced partitions" and include this in string rendering of the
> SparkPlan node so that it looks like this
> {code:java}
> (12) CustomShuffleReader
> Input [2]: [a#23, b#24]
> Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
> {code}
> For accurate description see
> https://github.com/apache/spark/pull/32776#issue-661274727
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
