Peter Toth created SPARK-60539:
----------------------------------
Summary: Storage-partitioned join places rows by a transform call
whose cast depends on the session time zone
Key: SPARK-60539
URL: https://issues.apache.org/jira/browse/SPARK-60539
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
SPARK-60027 makes {{TransformExpression.resolvedFunction}} cast a partition
transform's arguments to its function's {{inputTypes()}}, as the analyzer does
for a function call. A cast that needs a time zone, such as DATE to TIMESTAMP,
then takes the session time zone. The keys the connector reported do not depend
on it.
The storage-partitioned join paths that evaluate the call to place rows, the
one-side shuffle and the identity reduce, can therefore place a row away from
the partition the connector put its key in. Example with the test catalog's
{{years}}, which binds a DATE but declares a TIMESTAMP, while the in-memory
table computes the key from the DATE itself:
* table {{t}} with a DATE column {{day}}, partitioned by {{years(day)}};
* table {{u}} with a DATE column {{day}}, not partitioned;
* {{spark.sql.sources.v2.bucketing.shuffle.enabled=true}} and
{{spark.sql.session.timeZone=Asia/Tokyo}};
* {{SELECT /*+ MERGE(t, u) */ * FROM t JOIN u ON t.day = u.day}}.
The join drops the {{DATE'2021-01-01'}} row: the cast puts it at 2020-12-31
15:00 UTC, so {{years}} sends it to the 2020 partition. In UTC and
America/Los_Angeles the same join returns every row.
The one-side shuffle also builds the call in the task. With no explicit
{{spark.sql.session.timeZone}}, the cast there takes the executor JVM's default
zone, not the driver's.
One fix: the join paths refuse a call that holds a cast which needs a time
zone, through the {{canEvaluate}} check SPARK-60522 adds. The write path keeps
the cast, since there the call only clusters rows.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
