[
https://issues.apache.org/jira/browse/SPARK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-34237:
------------------------------------
Assignee: Apache Spark
> Add more metrics (fallback, spill) for object hash aggregate
> ------------------------------------------------------------
>
> Key: SPARK-34237
> URL: https://issues.apache.org/jira/browse/SPARK-34237
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Cheng Su
> Assignee: Apache Spark
> Priority: Trivial
>
> As object hash aggregate fallback mechanism is special - it will fallback to
> sort-based aggregation based on number of keys seen so far [0]. This fallback
> logic sometimes is sub-optimal and leads to unnecessary sort, and performance
> degradation in run-time. The first step to help user/developer debug is to
> add more related metrics on UI, e.g. spill size, and number of fallback to
> sort-based aggregation. (spill size metrics was already added for hash
> aggregate [1])
>
> [0]:
> [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala#L161]
>
> [1]:
> [https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala#L68]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
