dongjoon-hyun opened a new pull request, #60002: URL: https://github.com/apache/spark/pull/60002
### What changes were proposed in this pull request? This PR aims to fix `HiveTableScanExec.prunePartitions` to treat the default partition name, `__HIVE_DEFAULT_PARTITION__`, as `NULL`, like `CatalogTablePartition.toRow` and `HiveShim` already do. `prunePartitions` casts the raw value of every partition column, including the columns which the pruning predicates do not reference. It is used in the following cases. - The relation is already pruned by `PruneHiveTablePartitions` and the predicates have an exec subquery, e.g. a dynamic pruning filter reusing a broadcast, a runtime bloom filter, or a scalar subquery. - The relation is not pruned yet and `spark.sql.hive.metastorePartitionPruning` is `false`. ### Why are the changes needed? The default partition name stands for a `NULL` partition value, but `prunePartitions` casts it like a regular value. This causes two issues. 1. A non-string partition column fails the query in ANSI mode, even when the predicates do not reference the column. 2. A string partition column keeps the name as a non-null value. `IS NULL` does not match it, so the partition is silently dropped and the query returns a wrong result regardless of the ANSI mode. ```sql SET hive.exec.dynamic.partition.mode=nonstrict; CREATE TABLE dim (k INT, x INT) STORED AS TEXTFILE; INSERT INTO dim VALUES (1, 1); CREATE TABLE t1 (a INT) PARTITIONED BY (p1 INT, p2 INT) STORED AS TEXTFILE; INSERT INTO t1 PARTITION (p1 = 1, p2) VALUES (10, NULL), (11, 5); CREATE TABLE t2 (a INT) PARTITIONED BY (p1 INT, p2 STRING) STORED AS TEXTFILE; INSERT INTO t2 PARTITION (p1 = 1, p2) VALUES (10, NULL), (11, 'x'); -- Issue 1: It should return 10 and 11, but it fails. SELECT /*+ BROADCAST(d) */ t1.a FROM t1 JOIN (SELECT k FROM dim WHERE x = 1) d ON t1.p1 = d.k; -- [CAST_INVALID_INPUT] The value '__HIVE_DEFAULT_PARTITION__' of the type "STRING" cannot be cast to "INT" ... -- Issue 2: It should return 10, but it returns no row. -- It returns 10 if `spark.sql.optimizer.dynamicPartitionPruning.enabled` is `false`. SELECT /*+ BROADCAST(d) */ t2.a FROM t2 JOIN (SELECT k FROM dim WHERE x = 1) d ON t2.p1 = d.k WHERE t2.p2 IS NULL; ``` The hint is not required for Issue 2. The same happens with the default configurations when the dimension table is the broadcast side at planning time, e.g. after `ANALYZE TABLE` on a dimension table and a larger fact table. ### Does this PR introduce _any_ user-facing change? Yes. The above queries return the correct results after this PR. Both issues are reproduced with Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0. This PR does not change how `HadoopTableReader.fillPartitionKeys` fills the partition columns of the scan output. For example, `SELECT p2 FROM t1` still fails in ANSI mode, and `SELECT p2 FROM t2` still returns the name instead of `NULL`. ### How was this patch tested? Pass the CIs with the newly added test case. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
