[
https://issues.apache.org/jira/browse/SPARK-60561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-60561:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Judge the adaptive partial aggregation spill check by the recent rows, with a
> higher threshold
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-60561
> URL: https://issues.apache.org/jira/browse/SPARK-60561
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Wenchen Fan
> Priority: Major
>
> Adaptive partial aggregation (SPARK-58511) evaluates its compaction ratio
> periodically and right before its map would spill, against one threshold,
> {{minCompaction}} (default 1.05). Keeping the aggregation at a spill means
> aggregating the rest of the input through spills, which pays off only if that
> input keeps repeating keys. With 1.05, an input whose keys arrive together at
> a ratio of 1.5 keeps aggregating and spills every epoch, while the ratio of
> the whole epoch cannot tell such an input from one whose keys recur across
> the task.
> Judge the evaluation before a spill by the recent rows (those since the
> periodic evaluation before last), whose ratio climbs as the map fills when
> keys recur, and give it its own, higher threshold,
> {{spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill}}
> (default 2.0).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
