anshulbaliga7 opened a new pull request, #60211: URL: https://github.com/apache/spark/pull/60211
### What changes were proposed in this pull request? This PR makes the Hive ORC data source (`spark.sql.orc.impl=hive`) reject `TIMESTAMP_NTZ` columns in `supportDataType`, so that the write path, the read path with a user-specified schema and `CREATE TABLE ... USING ORC` fail with the existing `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` error, including nested `STRUCT`, `ARRAY` and `MAP` columns. SPARK-60545 did the same for Hive serde tables in `HiveFileFormat.supportDataType`. The three earlier rejections in the Hive ORC `OrcFileFormat.supportDataType` (`TIME` in SPARK-51590, the geospatial types in SPARK-60519, and `VARIANT`) use the same pattern. ### Why are the changes needed? Hive has no `TIMESTAMP_NTZ` type. `OrcSerializer.toHiveCompatibleDataType` rewrites only the nanosecond timestamp types and collated strings, so a `TIMESTAMP_NTZ` column reaches Hive's type parser as the string `timestamp_ntz` and the write job fails inside a task with a Hive internal error that doesn't tell the user which column is wrong or why: ```sql SET spark.sql.orc.impl=hive; CREATE TABLE t (ts TIMESTAMP_NTZ) USING ORC; -- succeeds INSERT INTO t VALUES (TIMESTAMP_NTZ'1970-01-01 00:00:01'); -- fails ``` ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: ... java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'timestamp_ntz' but 'timestamp_ntz' is found. ``` With `spark.sql.timestampType=TIMESTAMP_NTZ`, the plain `TIMESTAMP` keyword hits this too. ### Does this PR introduce _any_ user-facing change? Yes, only the error changes. These statements already failed before this PR. For the example above, `CREATE TABLE` now fails at analysis time with: ``` [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The ORC datasource doesn't support the column `ts` of the type "TIMESTAMP_NTZ". SQLSTATE: 0A000 ``` The native ORC implementation (the default) is not affected and keeps supporting `TIMESTAMP_NTZ`. ### How was this patch tested? New test case in `HiveOrcQuerySuite` covering the write path (top-level and nested), the read path with a user-specified schema and `CREATE TABLE ... USING ORC`. Without the fix it fails with the `SparkException` wrapping the Hive `IllegalArgumentException` above. Also ran all `*HiveOrc*` suites locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude claude-opus-4-8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
