dongjoon-hyun opened a new pull request, #59734: URL: https://github.com/apache/spark/pull/59734
### What changes were proposed in this pull request? This PR aims to disallow Hive incompatible partition column types in Hive serde tables with a clear error, `UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE`. The affected types are `TIMESTAMP_NTZ`, ANSI intervals, the nanosecond timestamp types (`TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)`), and `TIME`. Since SPARK-51418 (#50182), `HiveClientImpl.toHiveTable` stores Hive incompatible partition column types with a placeholder type, `<derived from deserializer>`, in Hive metastore. This works for data source tables because Spark restores their schema from the table properties. However, it applies to Hive serde tables too. `HiveClientImpl` drops the placeholder columns when it reads a table, so `HiveExternalCatalog.restoreHiveSerdeTable` sees a schema mismatch with the table properties and falls back to the Hive metastore schema, which has no partition column. This PR adds a check to the Hive serde table branch of `HiveExternalCatalog.createTable`. It uses `isHiveCompatibleDataType`, the same condition as the placeholder. Since all table creation paths go through it, the check covers the following: - `CREATE TABLE ... STORED AS` and `CREATE TABLE ... USING HIVE` - `CREATE TABLE ... AS SELECT` - `CREATE TABLE ... LIKE` with `STORED AS` or `USING HIVE` - `DataFrameWriter.saveAsTable` and `DataFrameWriterV2.create` with the `hive` provider Data source tables are not affected. ### Why are the changes needed? Since Apache Spark 4.0.0, the following statement succeeds, but it creates a broken table without the partition column `p`. ```sql CREATE TABLE t (i INT) PARTITIONED BY (p TIMESTAMP_NTZ) STORED AS PARQUET; ``` ``` spark-sql> INSERT INTO t PARTITION (p = TIMESTAMP_NTZ'2026-01-01 00:00:00') VALUES (1); [PARTITIONS_NOT_FOUND] The partition(s) `p` cannot be found in table `spark_catalog`.`default`.`t`. ... spark-sql> SHOW PARTITIONS t; [INVALID_PARTITION_OPERATION.PARTITION_SCHEMA_IS_EMPTY] The partition command is invalid. Table `spark_catalog`.`default`.`t` is not partitioned. SQLSTATE: 42601 ``` `CREATE TABLE AS SELECT` creates the table without the partition column, so the values of `p` are silently lost. ```sql CREATE TABLE t PARTITIONED BY (p) STORED AS PARQUET AS SELECT 1 AS i, TIMESTAMP_NTZ'2026-01-01 00:00:00' AS p; SELECT * FROM t; -- returns only `i` ``` With `spark.sql.timestampType=TIMESTAMP_NTZ`, `PARTITIONED BY (p TIMESTAMP)` has the same issue. `TIME` partition columns have it too since SPARK-60482 (#59720), which is not released yet. Moreover, Hive cannot read these tables (`Unknown type <derived from deserializer>`), and `ALTER TABLE ADD COLUMNS` corrupts the table metadata so that even `DROP TABLE` fails. ``` The metadata is corrupted. Unable to find the partition column names from the schema. schema: struct<i:int,j:int>. Partition columns: [p] ``` Before SPARK-51418, Hive metastore rejected these tables at the table creation, e.g., `InvalidObjectException(message:Invalid partition column type: timestamp_ntz)`. ### Does this PR introduce _any_ user-facing change? Yes. Creating a Hive serde table with these partition column types fails with the following error instead of creating a broken table. Data source tables are not affected. ``` [UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE] The Hive datasource doesn't support the column `p` of the type "TIMESTAMP_NTZ". SQLSTATE: 0A000 ``` Previously, some CTAS statements with ANSI interval partition columns, e.g., `STORED AS PARQUET`, failed with `UNSUPPORTED_FEATURE.HIVE_WITH_ANSI_INTERVALS` after creating and dropping the table. Now, they fail with the above error before creating the table. This PR doesn't change the existing tables. ### How was this patch tested? Pass the CIs with the newly added test case in `HiveDDLSuite`. The test case fails without this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
