Peter Toth created SPARK-59187:
----------------------------------
Summary: Carry the partition key data types on KeyedPartitioning
Key: SPARK-59187
URL: https://issues.apache.org/jira/browse/SPARK-59187
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
{{KeyedPartitioning}} carries its partition keys as
{{InternalRowComparableWrapper}} rows and no types. Every reader that needs the
types samples them from the first key row, via {{keyDataTypes}}:
{code:scala}
@transient lazy val keyDataTypes: Seq[DataType] =
partitionKeys.headOption.map(_.dataTypes).getOrElse(expressionDataTypes)
{code}
Two problems follow from that.
* With no key row there is nothing to sample, so it falls back to the partition
expressions' own types. Those describe the keys only while the expressions do,
and after a join reduced both sides' keys they no longer do (SPARK-59121).
SPARK-59176 was a query that failed because one caller held that fallback
against a real answer.
* Nothing checks that the remaining key rows agree with the first one.
The types are known where the keys are built, and are already computed and
dropped there:
{code:scala}
def apply(expressions: Seq[Expression], partitionKeys: Seq[InternalRow]):
KeyedPartitioning = {
val dataTypes = expressions.map(_.dataType)
val comparableKeyWrapperFactory =
InternalRowComparableWrapper.getInternalRowComparableWrapperFactory(dataTypes)
...
{code}
The proposal is to carry them on the partitioning beside {{expressions}}, and
to have {{projectKeys}}, {{reduceKeys}},
{{KeyedShuffleSpec.createPartitioning}} and {{GroupPartitionsExec}} carry them
through the transformations they apply to the keys. Then no reader samples a
key row, and a partitioning with no key still reports what its keys would hold.
This subsumes SPARK-59176's fix, which leaves a keyless side out of one
comparison rather than giving it an answer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
