[ 
https://issues.apache.org/jira/browse/SPARK-61279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Kumar updated SPARK-61279:
---------------------------------
    Description: 
{{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce 
partitions for storage-partitioned operations, keeps its groups in a field that 
is serialized into every task binary, because {{CoalescedRDD}}, and 
{{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only 
{{getPartitions}} on the driver reads the groups, so large groupings add task 
serialization cost for nothing.

Its location choice can also be improved. It reads only the parent's own 
{{preferredLocations}} and names the most common entry for every group, the 
first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}} 
coalesces the scan's mapped RDD, which reports no locations of its own, so no 
group names one, and the scheduler offers the locations of each group's first 
input that has any.

Proposed:
* A group of several partitions prefers a host only when more of its partitions 
prefer it than any other host. It finds each partition's locations through the 
parent's lineage with {{SparkContext.getPreferredLocs}}, as 
{{DefaultPartitionCoalescer}} does. When the only location that the group's 
partitions name on that host is an executor or an HDFS cache, the group keeps 
that location.
* Every other group names no location, so the scheduler finds the locations 
itself when it submits each stage, as it does today over a scan: a group of one 
partition, a group with no host ahead, groups that share partitions, as the 
slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered join, 
and groups whose parent's narrow lineage is persisted, since only the scheduler 
sees the cached blocks.

In a join that zips the grouped RDDs directly, a side whose group names no 
location adds none when the other side's group names one, so the task prefers 
the other side's location.

Split out of the review of SPARK-60265 
(https://github.com/apache/spark/pull/59522), which reuses this coalescer for 
V2 scan partition packing.

  was:
{{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce 
partitions for storage-partitioned operations, keeps its groups in a field that 
is serialized into every task binary, because {{CoalescedRDD}} keeps the 
coalescer. Only {{getPartitions}} on the driver reads the groups, so large 
groupings add task serialization cost for nothing.

Its location choice can also be improved:
* A group of one partition names one of that partition's locations, so the 
scheduler sees one host instead of all of them. It can name no location 
instead, so the scheduler uses the partition's own locations.
* For larger groups, ties between equally preferred locations all go to the 
same location, the first in a hash map's order, so several groups can prefer 
one host. Ties can be spread across groups instead.

Over a scan, {{GroupPartitionsExec}} coalesces the scan's mapped RDD, which 
reports no locations, so the location changes have no effect there: the 
scheduler still gives each group the locations of its first input that has any.

Split out of the review of SPARK-60265 
(https://github.com/apache/spark/pull/59522), which reuses this coalescer for 
V2 scan partition packing.


> Keep GroupedPartitionCoalescer's groups out of tasks and improve its location 
> choice
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-61279
>                 URL: https://issues.apache.org/jira/browse/SPARK-61279
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>
> {{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce 
> partitions for storage-partitioned operations, keeps its groups in a field 
> that is serialized into every task binary, because {{CoalescedRDD}}, and 
> {{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only 
> {{getPartitions}} on the driver reads the groups, so large groupings add task 
> serialization cost for nothing.
> Its location choice can also be improved. It reads only the parent's own 
> {{preferredLocations}} and names the most common entry for every group, the 
> first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}} 
> coalesces the scan's mapped RDD, which reports no locations of its own, so no 
> group names one, and the scheduler offers the locations of each group's first 
> input that has any.
> Proposed:
> * A group of several partitions prefers a host only when more of its 
> partitions prefer it than any other host. It finds each partition's locations 
> through the parent's lineage with {{SparkContext.getPreferredLocs}}, as 
> {{DefaultPartitionCoalescer}} does. When the only location that the group's 
> partitions name on that host is an executor or an HDFS cache, the group keeps 
> that location.
> * Every other group names no location, so the scheduler finds the locations 
> itself when it submits each stage, as it does today over a scan: a group of 
> one partition, a group with no host ahead, groups that share partitions, as 
> the slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered 
> join, and groups whose parent's narrow lineage is persisted, since only the 
> scheduler sees the cached blocks.
> In a join that zips the grouped RDDs directly, a side whose group names no 
> location adds none when the other side's group names one, so the task prefers 
> the other side's location.
> Split out of the review of SPARK-60265 
> (https://github.com/apache/spark/pull/59522), which reuses this coalescer for 
> V2 scan partition packing.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to