dongjoon-hyun opened a new pull request, #59783: URL: https://github.com/apache/spark/pull/59783
### What changes were proposed in this pull request? This PR aims to raise a clear error for `TIMESTAMP_NTZ` columns in Hive serde tables. Hive has no `TIMESTAMP_NTZ` type, so Hive serde tables cannot store `TIMESTAMP_NTZ` columns. However, Spark passes `timestamp_ntz` to Hive as it is and fails with a Hive internal error. `DDLUtils.checkTableColumns` already rejects the ANSI interval columns of Hive tables (SPARK-36949). This PR adds a check for `TIMESTAMP_NTZ` there, including the nested types (`STRUCT`, `ARRAY` and `MAP`). Like the `TIME` type (SPARK-57556), the nanosecond timestamp types (SPARK-57815) and the geospatial types (SPARK-60519) in the Hive serde write path, it raises the existing `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE` error with the `Hive` format. The check covers the following statements: - `CREATE TABLE` and `CREATE TABLE AS SELECT`, including `DataFrameWriter.saveAsTable` with the `hive` format and `Catalog.createTable` with the `hive` source - `ALTER TABLE ADD COLUMNS` - `INSERT OVERWRITE DIRECTORY ... STORED AS` when it is not converted to a data source write ### Why are the changes needed? With `spark.sql.timestampType=TIMESTAMP_NTZ`, the `TIMESTAMP` keyword in DDL statements is `TIMESTAMP_NTZ` by design. So, the simple DDL statement below fails with a Hive internal error, which doesn't tell the users which column is wrong or why. ```sql SET spark.sql.timestampType=TIMESTAMP_NTZ; CREATE TABLE t (ts TIMESTAMP) STORED AS PARQUET; ``` ``` org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: Error: type expected at the position 0 of 'timestamp_ntz' but 'timestamp_ntz' is found. ``` This issue was found during the review of #59674. ### Does this PR introduce _any_ user-facing change? Yes, only the error changes. These statements already failed before this PR. For the above example, the error becomes the following. ``` [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't support the column `ts` of the type "TIMESTAMP_NTZ". SQLSTATE: 0A000 ``` ### How was this patch tested? Pass the CIs with the newly added test cases in `HiveDDLSuite`, `AlterTableAddColumnsSuite` and `InsertSuite`. The test cases fail without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
