[
https://issues.apache.org/jira/browse/SPARK-60813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anil K updated SPARK-60813:
---------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Storage-partitioned join evaluates a partition transform on a datetime or
> interval value of another unit
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60813
> URL: https://issues.apache.org/jira/browse/SPARK-60813
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Anil K
> Priority: Major
>
> A storage-partitioned join evaluates a connector's partition transform itself
> in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes
> an argument as it is when it is stored the way the function's declared input
> type is, by comparing their physical types. Two datetime or interval types
> can share a physical type while counting in different units:
> * {{TIME}} holds nanoseconds of the day in a {{long}}, where {{TIMESTAMP}},
> {{TIMESTAMP_NTZ}} and a {{DAY TO SECOND}} interval hold microseconds;
> * a {{YEAR TO MONTH}} interval holds months in an {{int}}, where a {{DATE}}
> holds days;
> * a {{DAY TO SECOND}} interval holds a duration, which a function declaring
> {{TIMESTAMP}} reads as an instant.
> So a connector that binds such a column to a function declaring the other
> type gets the raw value in the wrong unit, as master does for every type.
> That is the shape SPARK-60027 fixes for a {{DATE}} against a {{TIMESTAMP}},
> where the test catalog's {{years}} returned 1 of 3 rows. Measured by
> evaluating the transform on master (0aa4f9136bc):
> * a {{years}} that declares {{TIMESTAMP}} and reads micros, over {{TIME
> '10:00'}}: year 1, from the nanoseconds read as microseconds;
> * a {{months}} that declares {{DATE}} and reads days, over {{INTERVAL '30'
> YEAR}}: month 11, from the 360 months read as days.
> A fix could let two datetime or interval types go as they are only when they
> are equal. {{TIMESTAMP_NTZ}} against {{TIMESTAMP}} can stay, since both hold
> microseconds and the connector's keys do not depend on the session time zone.
> {{DATE}} against {{INT}} and {{TIMESTAMP}} against {{BIGINT}} are not
> datetime pairs, so they are not affected. Anything else is refused, so the
> join shuffles both sides.
> Raised in the review of SPARK-60027
> (https://github.com/apache/spark/pull/59760#discussion_r4435873390).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
