XiDuo You created SPARK-59272:
---------------------------------
Summary: Revalidate the storage-partitioned join pairing when a
regrouped side drops the keyed claim
Key: SPARK-59272
URL: https://issues.apache.org/jira/browse/SPARK-59272
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
## Problem
In a storage-partitioned join (SPJ), `GroupPartitionsExec` deliberately gives
up a marked layout's keyed claim - reporting `UnknownPartitioning` - when the
regrouping is not the identity: a marked side realigned onto
differently-ordered merged keys, or a grouping that applies a reducer or a
non-identity projection (`identityGrouping` now requires `keysRewritten` to be
false). This is the intended response to a claim the regrouping invalidates,
and results stay correct because the give-up forces downstream consumers to
stop trusting the layout.
The gap is that the give-up is evaluated inside
`GroupPartitionsExec.outputPartitioning`, which `checkKeyGroupCompatible`
reaches through `applyGroupPartitions` after it has decided the pairing is
compatible and before it commits it. `EnsureRequirements` never re-checks the
rebuilt children, so a committed join can carry a child that no longer
satisfies its required distribution. `ValidateRequirements` then rejects the
stage, vetoing AQE rules (partition coalescing, local read, skew join) on it.
The regrouping test on SPARK-59050 exercises this state today. No wrong results
are known on the current head; the cost is planning quality and lost AQE
optimizations.
## Why a plain recheck is wrong
The direct fix - re-check `newLeft.outputPartitioning.satisfies(distribution)`
after `applyGroupPartitions` - is not viable: partial clustering deliberately
produces ungrouped yet value-aligned children, both aligned onto one
`mergedPartitionKeys` by `alignToExpectedKeys`. A `KeyedPartitioning` only
satisfies a `ClusteredDistribution` when grouped, so a `satisfies` gate rejects
the whole partial-clustering family (measured: the partial-clustering suites
regress).
## Proposed direction
Add a pairwise co-location check at the pairing site: whether the two rebuilt
children still declare the same aligned partition key sequence (duplicates and
order included) - the guarantee `alignToExpectedKeys` constructs - and whether
either side gave up its keyed layout. This should be reconciled with
`ValidateRequirements`' semantics, which reason per side from `satisfies` and
cannot express the constructed twin spread.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
