Anshul Baliga created SPARK-61132:
-------------------------------------
Summary: Raise a clear error for TIMESTAMP_NTZ columns in the Hive
ORC data source
Key: SPARK-61132
URL: https://issues.apache.org/jira/browse/SPARK-61132
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Anshul Baliga
The Hive ORC data source (\{{spark.sql.orc.impl=hive}}) has no mapping for
\{{TIMESTAMP_NTZ}}. \{{OrcSerializer.toHiveCompatibleDataType}} rewrites only
the nanosecond timestamp types and collated strings, so a \{{TIMESTAMP_NTZ}}
column reaches Hive's type parser as the string \{{timestamp_ntz}} and the
write fails with a Hive internal error that does not say which column is wrong
or why.
SPARK-60545 fixed the same problem for Hive serde tables by rejecting
\{{TIMESTAMP_NTZ}} in \{{HiveFileFormat.supportDataType}} with
\{{UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE}}, but the sibling
\{{OrcFileFormat.supportDataType}} in \{{sql/hive}} was not updated. The three
earlier rejections in that method (\{{TIME}} in SPARK-51590, the nanosecond
timestamps in SPARK-57166, the geospatial types in SPARK-60519) were applied to
both methods.
*Steps to reproduce (verified on master):*
{code:sql}
SET spark.sql.orc.impl=hive;
CREATE TABLE t (ts TIMESTAMP_NTZ) USING ORC; -- succeeds
INSERT INTO t VALUES (TIMESTAMP_NTZ'1970-01-01 00:00:01'); -- fails inside Hive
{code}
*Expected:* \{{CREATE TABLE}} fails with
\{{[UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support
the column `ts` of the type "TIMESTAMP_NTZ".}}, like the Hive serde format and
like the other unsupported types in Hive ORC.
*Proposed fix:* add \{{case _: TimestampNTZType => false}} to
\{{org.apache.spark.sql.hive.orc.OrcFileFormat.supportDataType}}, with a test
in \{{HiveOrcQuerySuite}} covering the write path (top-level and nested), the
read path with a user-specified schema, and \{{CREATE TABLE ... USING ORC}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
