[
https://issues.apache.org/jira/browse/SPARK-61654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146921#comment-18146921
]
Anil K commented on SPARK-61654:
--------------------------------
Status after https://github.com/apache/spark/pull/60431 (merged): most of this
is covered there, by a marker ({{TransformExpression.storedKey}}) that
{{createPartitioning}} sets on the transform it rebuilds over a cogroup key of
another type than the layout's column, when {{keysStoredAlike}} accepts it. The
marked call compares datetime and interval types by storage, so a {{TIME}} or
{{TIMESTAMP_NTZ}} key onto a bucket over a {{TIMESTAMP}} column is placed by
its bytes, as on master. A planner matrix and real Pandas cogroups show the
same results as master.
What remains, also on master: a key stored as the column is, which the function
takes only through a cast the column itself would get, still fails while
planning. E.g. a {{DATE}} key onto an {{INT}} column whose function declares
{{DOUBLE}}: the {{INT}} column is widened to {{DOUBLE}}, but a {{DATE}} key has
no such cast, so {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}} is raised,
although the keys are stored alike and the hash shuffle would pair them.
SPARK-60782 will likely make this obsolete: its proposed fix widens or rejects
cogroup keys of different types at analysis, so a cogroup's keys reach planning
with equal types. I'll close this once SPARK-60782 lands, or keep it for the
case above if that fix changes.
> Place a cogroup key stored like the layout's column by the column's own
> transform in the one-side shuffle
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61654
> URL: https://issues.apache.org/jira/browse/SPARK-61654
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Anil K
> Priority: Major
>
> When a storage-partitioned layout has a cogroup side shuffled onto it
> ({{spark.sql.sources.v2.bucketing.shuffle.enabled=true}}),
> {{KeyedShuffleSpec.createPartitioning}} rebuilds the layout's transform over
> the other side's key and asks whether Spark can evaluate the function on that
> key's type. For a cogroup that does not coerce its keys (e.g. a Pandas or
> Arrow cogroup), that is a stricter question than placement needs: the cogroup
> pairs groups by comparing key bytes, so a shuffled row only has to land where
> the keyed side put rows with the same bytes, which is the column's own
> transform on those bytes.
> So since SPARK-60813 and SPARK-61650
> (https://github.com/apache/spark/pull/60431), a cogroup whose key is stored
> like the layout's column but is a datetime type the function would read
> differently, e.g. a {{TIME}} or {{TIMESTAMP_NTZ}} key onto a bucket over a
> {{TIMESTAMP}} column whose function declares {{TIMESTAMP}}, fails while
> planning with {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}}. The hash shuffle,
> with the flag off, runs the same query and pairs equal bytes.
> A fix: when the key passes the storage check
> ({{TransformExpression.keysStoredAlike}}) and is an attribute, rebuild the
> transform over that attribute retyped as the column's type. The shuffle then
> reads the key's bytes as the column's type, the call is the layout's own
> (which {{canEvaluatePartitioning}} already asserts), and the pairing equals
> the hash shuffle's for every key the storage check takes. A join's keys share
> the column's type, so joins do not change; other keys keep the current path.
> The error then fires only for keys stored otherwise, as for an {{INT}} key
> onto a {{BIGINT}} column.
> This touches the code SPARK-50593
> (https://github.com/apache/spark/pull/55885) changes: the rebuild has to
> replace only the transform's column slots ({{rewriteColumnSlots}}), not its
> literal parameters. So it should follow SPARK-50593. If SPARK-60782 instead
> rejects cogroup keys of different types at analysis, this becomes moot for
> those types.
> Found in the review of https://github.com/apache/spark/pull/60431.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]