Yaoxuan Wu created SPARK-61944:
----------------------------------
Summary: SQL script results are rebuilt from external Rows: DATE
values in 1582-10-05..14 change to 1582-10-15 and ARRAY<TIME(p)> fails with
CANNOT_UP_CAST_DATATYPE
Key: SPARK-61944
URL: https://issues.apache.org/jira/browse/SPARK-61944
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0
Reporter: Yaoxuan Wu
The result of a SQL script is collected to external Rows and rebuilt from them.
With default settings this changes a DATE in the Julian-Gregorian gap to
1582-10-15, and it rejects a TIME(p) with p < 6 nested in an ARRAY, although
the same SELECT outside a script works.
{code:sql}
SELECT DATE'1582-10-10' AS d;
-- 1582-10-10
BEGIN SELECT DATE'1582-10-10' AS d; END;
-- expected: 1582-10-10
-- actual: 1582-10-15
SET spark.sql.timeType.enabled = true;
SELECT array(CAST(TIME'12:00:01' AS TIME(0))) AS x;
-- [12:00:01]
BEGIN SELECT array(CAST(TIME'12:00:01' AS TIME(0))) AS x; END;
-- [CANNOT_UP_CAST_DATATYPE] Cannot up cast array element from "TIME(0)" to
"TIME(6)".
BEGIN SELECT CAST(TIME'12:00:01' AS TIME(0)) AS x; END;
-- 12:00:01 (top level is fine)
{code}
* Same on 4.1.1 and 4.2.0, in any JVM time zone.
* With {{spark.sql.datetime.java8API.enabled = true}} the DATE is right.
* TIMESTAMPs are affected the same way where the Gregorian-to-Julian rebase is
not injective (e.g. JVM zone America/Los_Angeles, an instant in the LMT-to-PST
overlap of 1883-11-18).
* Every value a script returns goes through this, e.g. a variable read back
with {{SELECT v}}.
Cause (master d3f4a73a, SqlScriptingExecution.scala, {{executeSqlScript}}): the
result of the last statement is collected with {{df.collect()}} (external
{{Row}}s) and returned as {{LocalRelation.fromExternalRows(schema, result)}}.
1. With the default {{java8API.enabled = false}}, a DATE becomes a
{{java.sql.Date}} ({{DateTimeUtils.toJavaDate}}), which rebases from the
Proleptic Gregorian to the hybrid Julian calendar. The hybrid calendar has no
1582-10-05..14, so those dates come back as 1582-10-15.
2. The external Row deserializer for {{ARRAY<TIME(p)>}} uses
{{LocalTimeEncoder}}, whose type is {{TimeType()}} = TIME(6), and the analyzer
refuses the {{UpCast}} of the array element from TIME(0) to TIME(6).
The results could stay internal:
{{df.queryExecution.executedPlan.executeCollect()}} copied into
{{LocalRelation(output, rows)}}. The recursive CTE executor has the same
pattern (UnionLoopExec, filed separately).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]