Eugene Koifman created SPARK-35639:
--------------------------------------
Summary: Add metrics about coalesced partitions to
CustomShuffleReader in AQE
Key: SPARK-35639
URL: https://issues.apache.org/jira/browse/SPARK-35639
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Eugene Koifman
{{CustomShuffleReaderExec}} reports "number of skewed partitions" and "number
of skewed partition splits".
It would be useful to also report "number of partitions to coalesce" and
"number of coalesced partitions" and include this in string rendering of the
SparkPlan node so that it looks like this
{code:java}
(12) CustomShuffleReader
Input [2]: [a#23, b#24]
Arguments: coalesced 3 partitions into 1 and split 2 skewed partitions into 4
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
