[
https://issues.apache.org/jira/browse/SPARK-61279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61279:
-----------------------------------
Labels: pull-request-available (was: )
> Keep GroupedPartitionCoalescer's groups out of tasks and improve its location
> choice
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61279
> URL: https://issues.apache.org/jira/browse/SPARK-61279
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> {{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
> partitions for storage-partitioned operations, keeps its groups in a field
> that is serialized into every task binary, because {{CoalescedRDD}} keeps the
> coalescer. Only {{getPartitions}} on the driver reads the groups, so large
> groupings add task serialization cost for nothing.
> Its location choice can also be improved:
> * A group of one partition names one of that partition's locations, so the
> scheduler sees one host instead of all of them. It can name no location
> instead, so the scheduler uses the partition's own locations.
> * For larger groups, ties between equally preferred locations all go to the
> same location, the first in a hash map's order, so several groups can prefer
> one host. Ties can be spread across groups instead.
> Over a scan, {{GroupPartitionsExec}} coalesces the scan's mapped RDD, which
> reports no locations, so the location changes have no effect there: the
> scheduler still gives each group the locations of its first input that has
> any.
> Split out of the review of SPARK-60265
> (https://github.com/apache/spark/pull/59522), which reuses this coalescer for
> V2 scan partition packing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
