[ 
https://issues.apache.org/jira/browse/SPARK-61279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Toth resolved SPARK-61279.
--------------------------------
    Fix Version/s: 5.0.0
       Resolution: Fixed

Issue resolved by pull request 60256
[https://github.com/apache/spark/pull/60256]

> Keep GroupedPartitionCoalescer's groups out of tasks and improve its location 
> choice
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-61279
>                 URL: https://issues.apache.org/jira/browse/SPARK-61279
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Rakesh Kumar
>            Assignee: Rakesh Kumar
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 5.0.0
>
>
> {{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce 
> partitions for storage-partitioned operations, keeps its groups in a field 
> that is serialized into every task binary, because {{CoalescedRDD}}, and 
> {{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only 
> {{getPartitions}} on the driver reads the groups, so large groupings add task 
> serialization cost for nothing.
> Its location choice can also be improved. It reads only the parent's own 
> {{preferredLocations}} and names the most common entry for every group, the 
> first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}} 
> coalesces the scan's mapped RDD, which reports no locations of its own, so no 
> group names one, and the scheduler offers the locations of each group's first 
> input that has any.
> Proposed:
> * A group of several partitions prefers a host only when more of its 
> partitions prefer it than any other host. It finds each partition's locations 
> through the parent's lineage with {{SparkContext.getPreferredLocs}}, as 
> {{DefaultPartitionCoalescer}} does. When the only location that the group's 
> partitions name on that host is an executor or an HDFS cache, the group keeps 
> that location.
> * Every other group names no location, so the scheduler finds the locations 
> itself when it submits each stage, as it does today over a scan: a group of 
> one partition, a group with no host ahead, groups that share partitions, as 
> the slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered 
> join, and groups whose parent's narrow lineage is persisted, since only the 
> scheduler sees the cached blocks.
> In a join that zips the grouped RDDs directly, a side whose group names no 
> location adds none when the other side's group names one, so the task prefers 
> the other side's location.
> Split out of the review of SPARK-60265 
> (https://github.com/apache/spark/pull/59522), which reuses this coalescer for 
> V2 scan partition packing.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to