Wenchen Fan created SPARK-60561:
-----------------------------------
Summary: Judge the adaptive partial aggregation spill check by the
recent rows, with a higher threshold
Key: SPARK-60561
URL: https://issues.apache.org/jira/browse/SPARK-60561
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Wenchen Fan
Adaptive partial aggregation (SPARK-58511) evaluates its compaction ratio
periodically and right before its map would spill, against one threshold,
{{minCompaction}} (default 1.05). Keeping the aggregation at a spill means
aggregating the rest of the input through spills, which pays off only if that
input keeps repeating keys. With 1.05, an input whose keys arrive together at a
ratio of 1.5 keeps aggregating and spills every epoch, while the ratio of the
whole epoch cannot tell such an input from one whose keys recur across the task.
Judge the evaluation before a spill by the recent rows (those since the
periodic evaluation before last), whose ratio climbs as the map fills when keys
recur, and give it its own, higher threshold,
{{spark.sql.execution.aggregate.adaptivePartialAggregation.minCompactionToSpill}}
(default 2.0).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
