[
https://issues.apache.org/jira/browse/SPARK-59285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Toth updated SPARK-59285:
-------------------------------
Description:
{{KeyedPartitioning}} carries four things: the partition expressions, the
partition key rows, whether those rows are unique ({{isGrouped}}), and whether
one of them stands for several keys of a partitioning it was derived from
({{isCollapsed}}). Only the first is per instance. The members of a
{{PartitioningCollection}} name one physical layout with their own expressions,
so everything else is shared, and {{PartitioningCollection}} spends a clause
per shared field to say so:
{code:scala}
require(rep.expressions.length == first.expressions.length, ...)
require(rep.partitionKeys eq first.partitionKeys, ...)
require(rep.isCollapsed == first.isCollapsed, ...)
{code}
{{isGrouped}} is not in that list, so two members over one key list can
disagree about whether those keys are unique, and {{satisfies0}} is an
{{exists}} over the members, so it would answer from whichever one won.
h3. Proposal
{{KeyLayout(partitionKeys, dataTypes, isGrouped, isCollapsed)}}, shared by
reference, with {{KeyedPartitioning = (expressions, layout)}}:
* the collection's invariant becomes one {{eq}} on the layout plus the
expressions' arity, whatever fields the layout later grows;
* {{fromPartitionings}} merges one canonical layout instead of interning the
key reference and ORing a flag;
* {{KeyedShuffleSpec.createPartitioning}} has nothing to decide, since a copy
that only replaces the expressions keeps the layout;
* the key types move onto the layout, so {{keyDataTypes}} is one read rather
than a sample of the first key row with a fallback to the partition
expressions. A partitioning whose partitions were all pruned then still answers
for its key space, which is what SPARK-59176 needed an exception in
{{EnsureRequirements}} for. That exception goes.
h3. Sequencing
Stacked on SPARK-59187, which erases the naming from the types the key rows are
compared at. Without that the types are not shared: two sides of one join name
the same key space differently.
It should also wait for SPARK-59050, which adds
{{mayContainUnknownPartitionKeys}}, another field of the shared layout. Doing
this first means doing it twice.
> Hold a KeyedPartitioning's shared partition layout in one value
> ---------------------------------------------------------------
>
> Key: SPARK-59285
> URL: https://issues.apache.org/jira/browse/SPARK-59285
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> {{KeyedPartitioning}} carries four things: the partition expressions, the
> partition key rows, whether those rows are unique ({{isGrouped}}), and
> whether one of them stands for several keys of a partitioning it was derived
> from ({{isCollapsed}}). Only the first is per instance. The members of a
> {{PartitioningCollection}} name one physical layout with their own
> expressions, so everything else is shared, and {{PartitioningCollection}}
> spends a clause per shared field to say so:
> {code:scala}
> require(rep.expressions.length == first.expressions.length, ...)
> require(rep.partitionKeys eq first.partitionKeys, ...)
> require(rep.isCollapsed == first.isCollapsed, ...)
> {code}
> {{isGrouped}} is not in that list, so two members over one key list can
> disagree about whether those keys are unique, and {{satisfies0}} is an
> {{exists}} over the members, so it would answer from whichever one won.
> h3. Proposal
> {{KeyLayout(partitionKeys, dataTypes, isGrouped, isCollapsed)}}, shared by
> reference, with {{KeyedPartitioning = (expressions, layout)}}:
> * the collection's invariant becomes one {{eq}} on the layout plus the
> expressions' arity, whatever fields the layout later grows;
> * {{fromPartitionings}} merges one canonical layout instead of interning the
> key reference and ORing a flag;
> * {{KeyedShuffleSpec.createPartitioning}} has nothing to decide, since a copy
> that only replaces the expressions keeps the layout;
> * the key types move onto the layout, so {{keyDataTypes}} is one read rather
> than a sample of the first key row with a fallback to the partition
> expressions. A partitioning whose partitions were all pruned then still
> answers for its key space, which is what SPARK-59176 needed an exception in
> {{EnsureRequirements}} for. That exception goes.
> h3. Sequencing
> Stacked on SPARK-59187, which erases the naming from the types the key rows
> are compared at. Without that the types are not shared: two sides of one join
> name the same key space differently.
> It should also wait for SPARK-59050, which adds
> {{mayContainUnknownPartitionKeys}}, another field of the shared layout. Doing
> this first means doing it twice.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
