[ 
https://issues.apache.org/jira/browse/SPARK-61654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146921#comment-18146921
 ] 

Anil K commented on SPARK-61654:
--------------------------------

Status after https://github.com/apache/spark/pull/60431 (merged): most of this 
is covered there, by a marker ({{TransformExpression.storedKey}}) that 
{{createPartitioning}} sets on the transform it rebuilds over a cogroup key of 
another type than the layout's column, when {{keysStoredAlike}} accepts it. The 
marked call compares datetime and interval types by storage, so a {{TIME}} or 
{{TIMESTAMP_NTZ}} key onto a bucket over a {{TIMESTAMP}} column is placed by 
its bytes, as on master. A planner matrix and real Pandas cogroups show the 
same results as master.

What remains, also on master: a key stored as the column is, which the function 
takes only through a cast the column itself would get, still fails while 
planning. E.g. a {{DATE}} key onto an {{INT}} column whose function declares 
{{DOUBLE}}: the {{INT}} column is widened to {{DOUBLE}}, but a {{DATE}} key has 
no such cast, so {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}} is raised, 
although the keys are stored alike and the hash shuffle would pair them.

SPARK-60782 will likely make this obsolete: its proposed fix widens or rejects 
cogroup keys of different types at analysis, so a cogroup's keys reach planning 
with equal types. I'll close this once SPARK-60782 lands, or keep it for the 
case above if that fix changes.


> Place a cogroup key stored like the layout's column by the column's own 
> transform in the one-side shuffle
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61654
>                 URL: https://issues.apache.org/jira/browse/SPARK-61654
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Anil K
>            Priority: Major
>
> When a storage-partitioned layout has a cogroup side shuffled onto it 
> ({{spark.sql.sources.v2.bucketing.shuffle.enabled=true}}), 
> {{KeyedShuffleSpec.createPartitioning}} rebuilds the layout's transform over 
> the other side's key and asks whether Spark can evaluate the function on that 
> key's type. For a cogroup that does not coerce its keys (e.g. a Pandas or 
> Arrow cogroup), that is a stricter question than placement needs: the cogroup 
> pairs groups by comparing key bytes, so a shuffled row only has to land where 
> the keyed side put rows with the same bytes, which is the column's own 
> transform on those bytes.
> So since SPARK-60813 and SPARK-61650 
> (https://github.com/apache/spark/pull/60431), a cogroup whose key is stored 
> like the layout's column but is a datetime type the function would read 
> differently, e.g. a {{TIME}} or {{TIMESTAMP_NTZ}} key onto a bucket over a 
> {{TIMESTAMP}} column whose function declares {{TIMESTAMP}}, fails while 
> planning with {{STORAGE_PARTITION_JOIN_KEY_TYPE_MISMATCH}}. The hash shuffle, 
> with the flag off, runs the same query and pairs equal bytes.
> A fix: when the key passes the storage check 
> ({{TransformExpression.keysStoredAlike}}) and is an attribute, rebuild the 
> transform over that attribute retyped as the column's type. The shuffle then 
> reads the key's bytes as the column's type, the call is the layout's own 
> (which {{canEvaluatePartitioning}} already asserts), and the pairing equals 
> the hash shuffle's for every key the storage check takes. A join's keys share 
> the column's type, so joins do not change; other keys keep the current path. 
> The error then fires only for keys stored otherwise, as for an {{INT}} key 
> onto a {{BIGINT}} column.
> This touches the code SPARK-50593 
> (https://github.com/apache/spark/pull/55885) changes: the rebuild has to 
> replace only the transform's column slots ({{rewriteColumnSlots}}), not its 
> literal parameters. So it should follow SPARK-50593. If SPARK-60782 instead 
> rejects cogroup keys of different types at analysis, this becomes moot for 
> those types.
> Found in the review of https://github.com/apache/spark/pull/60431.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to