dongjoon-hyun opened a new pull request, #60006: URL: https://github.com/apache/spark/pull/60006
### What changes were proposed in this pull request? This PR aims to fix a correctness issue of Hive table scans. `HadoopTableReader.fillPartitionKeys` now returns `NULL` for the default partition name, `__HIVE_DEFAULT_PARTITION__`, instead of casting it like a regular value. `CatalogTablePartition.toRow` and `HiveShim` already treat the name as `NULL`. This PR also changes the expected answer of `SPARK-33904: null and empty string as partition values` in the Hive `ShowPartitionsSuite` to `NULL`, which is what its data source counterpart expects, and adds a migration guide entry. ### Why are the changes needed? This is a correctness issue. A query on a Hive table returns a wrong result without any error, with the default configurations and regardless of the ANSI mode. The default partition name stands for a `NULL` partition value. Partition pruning already treats it as `NULL`, but `fillPartitionKeys` casts it like a regular value. This causes two issues. 1. A string partition column keeps the name as a non-null value, so a predicate gives different answers depending on where it is evaluated. A predicate only on the partition column is evaluated by partition pruning and sees `NULL`. A predicate on rows sees the name, so `IS NULL` does not match and the partition is silently dropped. 2. A non-string partition column fails the query in ANSI mode when the query selects the column. ```sql SET hive.exec.dynamic.partition.mode=nonstrict; CREATE TABLE t1 (a INT) PARTITIONED BY (p STRING) STORED AS TEXTFILE; INSERT INTO t1 PARTITION (p) VALUES (10, NULL), (11, 'x'); CREATE TABLE t2 (a INT) PARTITIONED BY (p INT) STORED AS TEXTFILE; INSERT INTO t2 PARTITION (p) VALUES (10, NULL), (11, 5); -- Issue 1: No row has `a = 12`, so it should return the same row as `WHERE p IS NULL`, (10, NULL). -- However, it returns no row. SELECT a, p FROM t1 WHERE a = 12 OR p IS NULL; -- Issue 1: It should return (10, NULL), but it returns (10, __HIVE_DEFAULT_PARTITION__). SELECT a, p FROM t1 WHERE p IS NULL; -- Issue 2: It should return (10, NULL) and (11, 5), but it fails. SELECT a, p FROM t2; -- [CAST_INVALID_INPUT] The value '__HIVE_DEFAULT_PARTITION__' of the type "STRING" cannot be cast to "INT" ... ``` Both issues happen on every Hive table that Spark reads through its Hive SerDe, e.g. `STORED AS TEXTFILE`, `SEQUENCEFILE`, `RCFILE` and `AVRO`, or Parquet and ORC when the conversion is disabled. Data source tables and the Hive Parquet and ORC tables that Spark converts by default already return the correct results, so the result of the same Hive table currently depends on `spark.sql.hive.convertMetastoreParquet`. This is related to #60002, which fixes the same cast in `HiveTableScanExec.prunePartitions`. ### Does this PR introduce _any_ user-facing change? Yes. The above queries return the correct results after this PR. Both issues are reproduced with Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0. A string partition column now returns `NULL` instead of the name. As a result, a predicate on rows that compares the column with the name, e.g. `a = 12 OR p = '__HIVE_DEFAULT_PARTITION__'`, no longer matches the partition. A predicate only on the partition column, `p = '__HIVE_DEFAULT_PARTITION__'`, already returned no row. This PR adds a migration guide entry and no legacy configuration. Apache Hive itself returns the name for a string partition column, as the golden file `load_dyn_part14-7-7c931249465f330d51ef0610f214429e` shows, but `HiveCompatibilitySuite` excludes that test. Spark already differs from Hive there because of partition pruning: `WHERE p <> 'a'` drops the partition in Spark and returns it in Hive. ### How was this patch tested? Pass the CIs with the newly added test case and the changed test case. ```bash build/sbt -Phive 'hive/testOnly *HiveTableScanSuite -- -z "SPARK-60801"' ``` I also compared the results before and after this PR with a temporary suite: the partition column types BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DECIMAL, DATE, TIMESTAMP, STRING, VARCHAR and CHAR in both ANSI modes, and the formats TEXTFILE, SEQUENCEFILE, RCFILE, AVRO, and PARQUET and ORC with and without the conversion. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
