Rakesh Kumar created SPARK-61279:
------------------------------------
Summary: Keep GroupedPartitionCoalescer's groups out of tasks and
improve its location choice
Key: SPARK-61279
URL: https://issues.apache.org/jira/browse/SPARK-61279
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Rakesh Kumar
{{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
partitions for storage-partitioned operations, keeps its groups in a field that
is serialized into every task binary, because {{CoalescedRDD}} keeps the
coalescer. Only {{getPartitions}} on the driver reads the groups, so large
groupings add task serialization cost for nothing.
Its location choice can also be improved:
* A group of one partition names one of that partition's locations, so the
scheduler sees one host instead of all of them. It can name no location
instead, so the scheduler uses the partition's own locations.
* For larger groups, ties between equally preferred locations all go to the
same location, the first in a hash map's order, so several groups can prefer
one host. Ties can be spread across groups instead.
Over a scan, {{GroupPartitionsExec}} coalesces the scan's mapped RDD, which
reports no locations, so the location changes have no effect there: the
scheduler still gives each group the locations of its first input that has any.
Split out of the review of SPARK-60265
(https://github.com/apache/spark/pull/59522), which reuses this coalescer for
V2 scan partition packing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
