[
https://issues.apache.org/jira/browse/SPARK-61279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth reassigned SPARK-61279:
----------------------------------
Assignee: Rakesh Kumar
> Keep GroupedPartitionCoalescer's groups out of tasks and improve its location
> choice
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61279
> URL: https://issues.apache.org/jira/browse/SPARK-61279
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Assignee: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> {{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
> partitions for storage-partitioned operations, keeps its groups in a field
> that is serialized into every task binary, because {{CoalescedRDD}}, and
> {{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only
> {{getPartitions}} on the driver reads the groups, so large groupings add task
> serialization cost for nothing.
> Its location choice can also be improved. It reads only the parent's own
> {{preferredLocations}} and names the most common entry for every group, the
> first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}}
> coalesces the scan's mapped RDD, which reports no locations of its own, so no
> group names one, and the scheduler offers the locations of each group's first
> input that has any.
> Proposed:
> * A group of several partitions prefers a host only when more of its
> partitions prefer it than any other host. It finds each partition's locations
> through the parent's lineage with {{SparkContext.getPreferredLocs}}, as
> {{DefaultPartitionCoalescer}} does. When the only location that the group's
> partitions name on that host is an executor or an HDFS cache, the group keeps
> that location.
> * Every other group names no location, so the scheduler finds the locations
> itself when it submits each stage, as it does today over a scan: a group of
> one partition, a group with no host ahead, groups that share partitions, as
> the slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered
> join, and groups whose parent's narrow lineage is persisted, since only the
> scheduler sees the cached blocks.
> In a join that zips the grouped RDDs directly, a side whose group names no
> location adds none when the other side's group names one, so the task prefers
> the other side's location.
> Split out of the review of SPARK-60265
> (https://github.com/apache/spark/pull/59522), which reuses this coalescer for
> V2 scan partition packing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]