Anil K created SPARK-60813:
------------------------------
Summary: Storage-partitioned join evaluates a partition transform
on a datetime or interval value of another unit
Key: SPARK-60813
URL: https://issues.apache.org/jira/browse/SPARK-60813
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
A storage-partitioned join evaluates a connector's partition transform itself
in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes an
argument as it is when it is stored the way the function's declared input type
is, by comparing their physical types. Two datetime or interval types can share
a physical type while counting in different units:
* {{TIME}} holds nanoseconds of the day in a {{long}}, where {{TIMESTAMP}},
{{TIMESTAMP_NTZ}} and a {{DAY TO SECOND}} interval hold microseconds;
* a {{YEAR TO MONTH}} interval holds months in an {{int}}, where a {{DATE}}
holds days;
* a {{DAY TO SECOND}} interval holds a duration, which a function declaring
{{TIMESTAMP}} reads as an instant.
So a connector that binds such a column to a function declaring the other type
gets the raw value in the wrong unit, as master does for every type. That is
the shape SPARK-60027 fixes for a {{DATE}} against a {{TIMESTAMP}}, where the
test catalog's {{years}} returned 1 of 3 rows. Measured by evaluating the
transform on master (0aa4f9136bc):
* a {{years}} that declares {{TIMESTAMP}} and reads micros, over {{TIME
'10:00'}}: year 1, from the nanoseconds read as microseconds;
* a {{months}} that declares {{DATE}} and reads days, over {{INTERVAL '30'
YEAR}}: month 11, from the 360 months read as days.
A fix could let two datetime or interval types go as they are only when they
are equal. {{TIMESTAMP_NTZ}} against {{TIMESTAMP}} can stay, since both hold
microseconds and the connector's keys do not depend on the session time zone.
{{DATE}} against {{INT}} and {{TIMESTAMP}} against {{BIGINT}} are not datetime
pairs, so they are not affected. Anything else is refused, so the join shuffles
both sides.
Raised in the review of SPARK-60027
(https://github.com/apache/spark/pull/59760#discussion_r4435873390).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
