ulysses-you opened a new pull request, #58522:
URL: https://github.com/apache/spark/pull/58522
### What changes were proposed in this pull request?
Keep the reported key-grouped partitioning when some of its keys are
column-pruned out of the scan
output, instead of dropping the whole partitioning, so storage-partitioned
join still applies. The
behavior is gated on
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled`:
- `V2ScanPartitioningAndOrdering` keeps the full partitioning when at least
one key remains in the
scan output.
- `DataSourceV2ScanExecBase.outputPartitioning` projects away the keys that
reference pruned columns,
so the physical partitioning only references output columns.
- `DataSourceV2ScanRelation.doCanonicalize` and `BatchScanExec`
(`doCanonicalize` / `equals` /
`hashCode`) ignore partitioning keys and sort orders that reference pruned
columns, so canonicalized
plan equality (subplan merging, exchange reuse) is unaffected.
### Why are the changes needed?
With `allowKeysSubsetOfPartitionKeys`, the operation keys may be a subset of
the partition keys, so an
extra partition key that is column-pruned out of the scan output should not
disable
storage-partitioned join.
Background: SPARK-40429 (#37886) introduced the drop-to-`None` guard to
avoid a "missing inputs"
plan-validation error when a partitioning key referenced a pruned column.
That failure mode no longer
applies: SPARK-40259 made `DataSourceV2ScanRelation.references` return empty
(partitioning, ordering
and pushed filters are scan metadata, not references to resolve), so a
dangling partitioning key no
longer trips plan validation. This change relies on that; in addition the
physical `outputPartitioning`
and the canonicalization project/drop the pruned keys so nothing dangling
reaches any consumer.
### Does this PR introduce _any_ user-facing change?
Yes. When
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` is true
and a
partition key is pruned from the scan output, Spark now applies
storage-partitioned join (reporting the
projected partitioning) instead of falling back to a shuffle. With the
config off (default), behavior is
unchanged.
### How was this patch tested?
New tests added:
- `KeyGroupedPartitioningSuite`: a join with a pruned partition key triggers
SPJ; all partition keys
pruned reports no partitioning; a self-join keeps plans canonicalizable; a
pruned partition key and a
pruned source-reported ordering do not defeat plan/exchange reuse.
- `MergeSubplansSuite`: identical scans whose reported ordering is on a
pruned column are deduplicated,
not fused. Verified this test fails without the canonicalization change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
