[
https://issues.apache.org/jira/browse/SPARK-60382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-60382:
--------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> CoalesceBucketsInJoin uses a broadcast join's build-side bucket count
> ---------------------------------------------------------------------
>
> Key: SPARK-60382
> URL: https://issues.apache.org/jira/browse/SPARK-60382
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Yuming Wang
> Priority: Critical
>
> When spark.sql.bucketing.coalesceBucketsInJoin.enabled is true, a sort-merge
> join whose child is a BuildLeft broadcast hash join, broadcast nested-loop
> join, or broadcast range join can coalesce the streamed bucketed scan using
> the build side's bucket count.
> ExtractJoinWithBuckets.getBucketSpec uses collectFirst, which walks the left
> child first. For BuildLeft the left child is the broadcast build side, so a
> bucketed scan there is reported as the join input's bucket count.
> updateNumCoalescedBucketsInScan then stamps every bucketed scan in that
> subtree, including the streamed scan whose partitioning the parent join
> actually uses.
> Bucket files are grouped by the original bucket id modulo the coalesced
> count, while the scan advertises HashPartitioning of that count. If the
> streamed table's bucket count is not divisible by the build side's count,
> hash(key) % streamedBuckets and hash(key) % coalescedBuckets land in
> different partitions. The other side of the sort-merge join is a real table
> of the coalesced bucket count, EnsureRequirements inserts no shuffle, and an
> inner join drops matching rows. An outer join null-pads them.
> The count that reaches the parent join is the streamed side's. The lookup
> should follow the same streamed-side walk as hasScanOperation (Filter,
> Project, then the streamed child of a broadcast hash, nested-loop, or range
> join), and only that scan should be rewritten.
> The config defaults to false, so this does not affect plans unless bucket
> coalescing is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
