Anil K created SPARK-61650:
------------------------------
Summary: Storage-partitioned join passes a datetime or interval
value that its cast to the declared type would change by time zone or precision
Key: SPARK-61650
URL: https://issues.apache.org/jira/browse/SPARK-61650
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
A storage-partitioned join evaluates a connector's partition transform itself
in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes an
argument as it is when it is stored the way the function's declared input type
is, by comparing their physical types. SPARK-60813 refuses two datetime or
interval types that count in different units. Two more pairs share a physical
type and a unit, but the function still gets another value than Spark's cast to
the declared type gives, which is what the write path
({{DistributionAndOrderingUtils}}) passes:
* *another time zone reference*: {{TIMESTAMP_NTZ}} holds a local time, which
the cast to {{TIMESTAMP}} converts with the session time zone. In
{{America/Los_Angeles}}, the transform got 3600000000 (01:00) where the write
path passes 32400000000 (09:00 UTC);
* *a finer precision or end field*: the cast from {{TIME(9)}} to {{TIME(6)}}
truncates to microseconds, and one from a {{DAY TO SECOND}} interval to a {{DAY
TO HOUR}} one to hours, while the transform got the untruncated value, e.g.
3600123456789 where the write path passes 3600123456000 (with
{{spark.sql.ansi.enabled=false}}).
Rows of the shuffled side can then land in the wrong partitions and join
matches are silently lost.
A fix: let two datetime or interval types go as they are only where Spark's
cast between them leaves the value unchanged, i.e. the same class at no finer a
precision or end field, e.g. {{TIME(3)}} against {{TIME(6)}}, or a {{DAY TO
HOUR}} interval against a {{DAY TO SECOND}} one, and refuse the rest, so the
join shuffles both sides.
SPARK-60813 kept {{TIMESTAMP_NTZ}} against {{TIMESTAMP}}, on the grounds that
the connector computes its keys without the session time zone. Refusing it is
safe either way: if the connector's keys come from the raw value, the join only
shuffles both sides; if they come from the cast, as Spark's write path computes
them, it avoids wrong results.
Found in the review of SPARK-60813 (https://github.com/apache/spark/pull/60431).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]