[
https://issues.apache.org/jira/browse/SPARK-31973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372286#comment-17372286
]
Apache Spark commented on SPARK-31973:
--------------------------------------
User 'shipra-a' has created a pull request for this issue:
https://github.com/apache/spark/pull/33161
> Add ability to disable Sort,Spill in Partial aggregation
> ---------------------------------------------------------
>
> Key: SPARK-31973
> URL: https://issues.apache.org/jira/browse/SPARK-31973
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Karuppayya
> Priority: Major
>
> In case of HashAggregation, a partial aggregation(update) is done followed by
> final aggregation(merge)
> During partial aggregation we sort and spill to disk everytime, when the fast
> Map(when enabled) and UnsafeFixedWidthAggregationMap gets exhausted
> *When the cardinality of grouping column is close to the total number of
> records being processed*, the sorting of data spilling to disk is not
> required, since it is kind of no-op and we can directly use in Final
> aggregation.
> When the user is aware of nature of data, currently he has no control over
> disabling this sort, spill operation.
> This is similar to following issue in Hive:
> https://issues.apache.org/jira/browse/HIVE-223
> https://issues.apache.org/jira/browse/HIVE-291
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
