[
https://issues.apache.org/jira/browse/SPARK-59256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59256:
-------------------------------
Description:
`ShuffleSpecCollection` answers two different kinds of question with one type.
* *Matching*: `isCompatibleWith` succeeds when **any** member matches. That is
a collection-level question and it is the reason the type exists.
* *Shuffle template*: `createPartitioning` and `numPartitions` need **one**
member, and the collection picks `specs.head` - whichever member the alias
cross-product enumerated first.
The second answer is a guess. Which member is right depends on what the *other*
side matched, and that is not knowable inside the collection.
SPARK-59080 fixed the consequence: `EnsureRequirements` now resolves the agreed
member itself and calls `createPartitioning` on that member, so the collection
is never asked. This ticket removes the ability to ask.
h3. What changes
Split the trait so that the shuffle-template contract is a separate type, and
`ShuffleSpecCollection` carries only `isCompatibleWith`.
* `createPartitioning` leaves the collection. After SPARK-59080 it has no
production caller, and the only thing standing between a future caller and a
wrong partitioning is a runtime `require`. The compiler should enforce that
instead.
* `numPartitions` leaves the collection. Its consumers get an answer that does
not read an arbitrary member:
** `EnsureRequirements`, `finalCandidateSpecs.values.maxBy(_.numPartitions)`,
which ranks the children to pick the reference layout. A head that understates
the count makes a child lose that ranking, so the join lands on a coarser
layout: an extra shuffle or less parallelism, never wrong results.
** `SinglePartitionShuffleSpec.isCompatibleWith`, which reads
`other.numPartitions == 1`. Reached from `ValidateRequirements`, where
`specs.tail.forall(_.isCompatibleWith(specs.head))` can put a collection on the
right-hand side. A collection whose head projects to one partition while
another member does not then answers on the head's authority.
Neither consumer is measured, and I could not build a query that reaches the
second one.
h3. Scope
Master only. No user-facing change and no known bug, so there is nothing to
backport. The behavioural part is limited to the ranking answer; everything
else is a type move.
> Remove createPartitioning and numPartitions from ShuffleSpecCollection
> ----------------------------------------------------------------------
>
> Key: SPARK-59256
> URL: https://issues.apache.org/jira/browse/SPARK-59256
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> `ShuffleSpecCollection` answers two different kinds of question with one type.
> * *Matching*: `isCompatibleWith` succeeds when **any** member matches. That
> is a collection-level question and it is the reason the type exists.
> * *Shuffle template*: `createPartitioning` and `numPartitions` need **one**
> member, and the collection picks `specs.head` - whichever member the alias
> cross-product enumerated first.
> The second answer is a guess. Which member is right depends on what the
> *other* side matched, and that is not knowable inside the collection.
> SPARK-59080 fixed the consequence: `EnsureRequirements` now resolves the
> agreed member itself and calls `createPartitioning` on that member, so the
> collection is never asked. This ticket removes the ability to ask.
> h3. What changes
> Split the trait so that the shuffle-template contract is a separate type, and
> `ShuffleSpecCollection` carries only `isCompatibleWith`.
> * `createPartitioning` leaves the collection. After SPARK-59080 it has no
> production caller, and the only thing standing between a future caller and a
> wrong partitioning is a runtime `require`. The compiler should enforce that
> instead.
> * `numPartitions` leaves the collection. Its consumers get an answer that
> does not read an arbitrary member:
> ** `EnsureRequirements`, `finalCandidateSpecs.values.maxBy(_.numPartitions)`,
> which ranks the children to pick the reference layout. A head that
> understates the count makes a child lose that ranking, so the join lands on a
> coarser layout: an extra shuffle or less parallelism, never wrong results.
> ** `SinglePartitionShuffleSpec.isCompatibleWith`, which reads
> `other.numPartitions == 1`. Reached from `ValidateRequirements`, where
> `specs.tail.forall(_.isCompatibleWith(specs.head))` can put a collection on
> the right-hand side. A collection whose head projects to one partition while
> another member does not then answers on the head's authority.
> Neither consumer is measured, and I could not build a query that reaches the
> second one.
> h3. Scope
> Master only. No user-facing change and no known bug, so there is nothing to
> backport. The behavioural part is limited to the ranking answer; everything
> else is a type move.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
