Yaoxuan Wu created SPARK-61944:
----------------------------------

             Summary: SQL script results are rebuilt from external Rows: DATE 
values in 1582-10-05..14 change to 1582-10-15 and ARRAY<TIME(p)> fails with 
CANNOT_UP_CAST_DATATYPE
                 Key: SPARK-61944
                 URL: https://issues.apache.org/jira/browse/SPARK-61944
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.1, 4.2.0
            Reporter: Yaoxuan Wu


The result of a SQL script is collected to external Rows and rebuilt from them. 
With default settings this changes a DATE in the Julian-Gregorian gap to 
1582-10-15, and it rejects a TIME(p) with p < 6 nested in an ARRAY, although 
the same SELECT outside a script works.

{code:sql}
SELECT DATE'1582-10-10' AS d;
-- 1582-10-10

BEGIN SELECT DATE'1582-10-10' AS d; END;
-- expected: 1582-10-10
-- actual:   1582-10-15

SET spark.sql.timeType.enabled = true;
SELECT array(CAST(TIME'12:00:01' AS TIME(0))) AS x;
-- [12:00:01]

BEGIN SELECT array(CAST(TIME'12:00:01' AS TIME(0))) AS x; END;
-- [CANNOT_UP_CAST_DATATYPE] Cannot up cast array element from "TIME(0)" to 
"TIME(6)".

BEGIN SELECT CAST(TIME'12:00:01' AS TIME(0)) AS x; END;
-- 12:00:01 (top level is fine)
{code}

* Same on 4.1.1 and 4.2.0, in any JVM time zone.
* With {{spark.sql.datetime.java8API.enabled = true}} the DATE is right.
* TIMESTAMPs are affected the same way where the Gregorian-to-Julian rebase is 
not injective (e.g. JVM zone America/Los_Angeles, an instant in the LMT-to-PST 
overlap of 1883-11-18).
* Every value a script returns goes through this, e.g. a variable read back 
with {{SELECT v}}.

Cause (master d3f4a73a, SqlScriptingExecution.scala, {{executeSqlScript}}): the 
result of the last statement is collected with {{df.collect()}} (external 
{{Row}}s) and returned as {{LocalRelation.fromExternalRows(schema, result)}}.
1. With the default {{java8API.enabled = false}}, a DATE becomes a 
{{java.sql.Date}} ({{DateTimeUtils.toJavaDate}}), which rebases from the 
Proleptic Gregorian to the hybrid Julian calendar. The hybrid calendar has no 
1582-10-05..14, so those dates come back as 1582-10-15.
2. The external Row deserializer for {{ARRAY<TIME(p)>}} uses 
{{LocalTimeEncoder}}, whose type is {{TimeType()}} = TIME(6), and the analyzer 
refuses the {{UpCast}} of the array element from TIME(0) to TIME(6).

The results could stay internal: 
{{df.queryExecution.executedPlan.executeCollect()}} copied into 
{{LocalRelation(output, rows)}}. The recursive CTE executor has the same 
pattern (UnionLoopExec, filed separately).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to