This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f9060a328b03 [SPARK-60801][SQL] Fix wrong results by reading the Hive 
default partition as NULL
f9060a328b03 is described below

commit f9060a328b03e5082b67dfa96aa1ec7cbfe4c010
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 07:51:47 2026 -0800

    [SPARK-60801][SQL] Fix wrong results by reading the Hive default partition 
as NULL
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix a correctness issue of Hive table scans. 
`HadoopTableReader.fillPartitionKeys` now returns `NULL` for the default 
partition name, `__HIVE_DEFAULT_PARTITION__`, instead of casting it like a 
regular value. `CatalogTablePartition.toRow` and `HiveShim` already treat the 
name as `NULL`.
    
    This PR also changes the expected answer of `SPARK-33904: null and empty 
string as partition values` in the Hive `ShowPartitionsSuite` to `NULL`, which 
is what its data source counterpart expects, and adds a migration guide entry.
    
    ### Why are the changes needed?
    
    This is a correctness issue. A query on a Hive table returns a wrong result 
without any error, with the default configurations and regardless of the ANSI 
mode.
    
    The default partition name stands for a `NULL` partition value. Partition 
pruning already treats it as `NULL`, but `fillPartitionKeys` casts it like a 
regular value. This causes two issues.
    
    1. A string partition column keeps the name as a non-null value, so a 
predicate gives different answers depending on where it is evaluated. A 
predicate only on the partition column is evaluated by partition pruning and 
sees `NULL`. A predicate on rows sees the name, so `IS NULL` does not match and 
the partition is silently dropped.
    2. A non-string partition column fails the query in ANSI mode when the 
query selects the column.
    
    ```sql
    SET hive.exec.dynamic.partition.mode=nonstrict;
    CREATE TABLE t1 (a INT) PARTITIONED BY (p STRING) STORED AS TEXTFILE;
    INSERT INTO t1 PARTITION (p) VALUES (10, NULL), (11, 'x');
    CREATE TABLE t2 (a INT) PARTITIONED BY (p INT) STORED AS TEXTFILE;
    INSERT INTO t2 PARTITION (p) VALUES (10, NULL), (11, 5);
    
    -- Issue 1: No row has `a = 12`, so it should return the same row as `WHERE 
p IS NULL`, (10, NULL).
    -- However, it returns no row.
    SELECT a, p FROM t1 WHERE a = 12 OR p IS NULL;
    
    -- Issue 1: It should return (10, NULL), but it returns (10, 
__HIVE_DEFAULT_PARTITION__).
    SELECT a, p FROM t1 WHERE p IS NULL;
    
    -- Issue 2: It should return (10, NULL) and (11, 5), but it fails.
    SELECT a, p FROM t2;
    -- [CAST_INVALID_INPUT] The value '__HIVE_DEFAULT_PARTITION__' of the type 
"STRING" cannot be cast to "INT" ...
    ```
    
    Both issues happen on every Hive table that Spark reads through its Hive 
SerDe, e.g. `STORED AS TEXTFILE`, `SEQUENCEFILE`, `RCFILE` and `AVRO`, or 
Parquet and ORC when the conversion is disabled. Data source tables and the 
Hive Parquet and ORC tables that Spark converts by default already return the 
correct results, so the result of the same Hive table currently depends on 
`spark.sql.hive.convertMetastoreParquet`.
    
    This is related to #60002, which fixes the same cast in 
`HiveTableScanExec.prunePartitions`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The above queries return the correct results after this PR. Both 
issues are reproduced with Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0.
    
    A string partition column now returns `NULL` instead of the name. As a 
result, a predicate on rows that compares the column with the name, e.g. `a = 
12 OR p = '__HIVE_DEFAULT_PARTITION__'`, no longer matches the partition. A 
predicate only on the partition column, `p = '__HIVE_DEFAULT_PARTITION__'`, 
already returned no row. This PR adds a migration guide entry and no legacy 
configuration.
    
    Apache Hive itself returns the name for a string partition column, as the 
golden file `load_dyn_part14-7-7c931249465f330d51ef0610f214429e` shows, but 
`HiveCompatibilitySuite` excludes that test. Spark already differs from Hive 
there because of partition pruning: `WHERE p <> 'a'` drops the partition in 
Spark and returns it in Hive.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case and the changed test case.
    
    ```bash
    build/sbt -Phive 'hive/testOnly *HiveTableScanSuite -- -z "SPARK-60801"'
    ```
    
    I also compared the results before and after this PR with a temporary 
suite: the partition column types BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, 
FLOAT, DOUBLE, DECIMAL, DATE, TIMESTAMP, STRING, VARCHAR and CHAR in both ANSI 
modes, and the formats TEXTFILE, SEQUENCEFILE, RCFILE, AVRO, and PARQUET and 
ORC with and without the conversion.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60006 from dongjoon-hyun/SPARK-60801.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                            |  1 +
 .../scala/org/apache/spark/sql/hive/TableReader.scala  |  6 +++++-
 .../spark/sql/hive/execution/HiveTableScanSuite.scala  | 18 ++++++++++++++++++
 .../hive/execution/command/ShowPartitionsSuite.scala   |  4 +---
 4 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index cdd0bb2a0dbd..f4b81561a00d 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -50,6 +50,7 @@ license: |
 - Since Spark 4.4, the Oracle JDBC datasource maps Oracle DATE and TIMESTAMP 
to TimestampNTZType by default, ignoring the JDBC read option 
`preferTimestampNTZ` (DATE only when the driver default 
`oracle.jdbc.mapDateToTimestamp=true` surfaces it as TIMESTAMP; otherwise DATE 
stays DateType). In 4.3 and earlier these were read as TimestampType, or as 
TimestampNTZType only when `preferTimestampNTZ=true`. This faithfully 
represents these zoneless Oracle types. Unaffected: Oracle DATE read wit [...]
 - Since Spark 4.4, a nondeterministic common expression that a conditional 
branch reads more than once is evaluated once, and every read sees that one 
value. `BETWEEN` and `NULLIF` read their input twice, so an expression such as 
`CASE WHEN c THEN monotonically_increasing_id() BETWEEN 3 AND 5 END` previously 
gave each read its own value and could return a different result. Queries whose 
results depended on the earlier behaviour change, and an optimized or physical 
plan may now show a `wi [...]
 - Since Spark 4.4, `format_string` and `printf` pass a decimal argument to 
`java.util.Formatter` as a `java.math.BigDecimal`, so the floating-point 
conversions `%f`, `%e`, and `%g` format decimals instead of failing with 
`IllegalFormatConversionException`. `%a` still rejects a decimal, because 
`Formatter` supports it only for `float` and `double`. As a side effect, `%h` 
on a decimal now hashes the `java.math.BigDecimal` (unscaled value and scale) 
rather than Spark's internal `Decimal` (w [...]
+- Since Spark 4.4, a Hive table that is read through its Hive SerDe, e.g. 
`STORED AS TEXTFILE` or Parquet with `spark.sql.hive.convertMetastoreParquet` 
set to `false`, returns `NULL` for the partition column of the default 
partition `__HIVE_DEFAULT_PARTITION__`. This matches data source tables and the 
Hive Parquet and ORC tables that Spark converts by default. Previously, a 
string partition column returned the name itself, so that a predicate on rows 
such as `a = 1 OR p IS NULL` skipped  [...]
 - Since Spark 4.4, for storage-partitioned joins, 
`spark.sql.requireAllClusterKeysForCoPartition` requires every join key to be 
covered by some partition key instead of matching the partition keys 
positionally. As a result, a join-key column partitioned by more than one 
transform no longer prevents shuffle elimination, and 
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` no 
longer additionally requires `spark.sql.requireAllClusterKeysForCoPartition` to 
be `false` w [...]
 - Since Spark 4.4, the one-side shuffle that 
`spark.sql.sources.v2.bucketing.shuffle.enabled` turns on follows the new 
`spark.sql.sources.v2.bucketing.requireAllClusterKeysForCoPartition.enabled`, 
as the storage-partitioned join does. It defaults to 
`spark.sql.requireAllClusterKeysForCoPartition`, which is `true`. A join runs 
on one side's partition keys only if they cover every join key. Previously, a 
join could run on partition keys that covered only some of the join keys. Now 
it runs  [...]
 - Since Spark 4.4, the CSV `extension` option must be non-empty and contain 
only letters. In Spark 4.3 and earlier, due to a bug in the extension 
validation check, empty strings and three-character suffixes containing 
non-letters (for example `ab1`) were accepted. This affects both writes and 
reads of CSV tables whose persisted `OPTIONS` contain such values; update the 
option to a non-empty letters-only suffix to restore access.
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
index ae9c49ea7ed8..c3a764746025 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
@@ -42,6 +42,7 @@ import org.apache.spark.rdd.{EmptyRDD, HadoopRDD, 
NewHadoopRDD, RDD, UnionRDD}
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
 import org.apache.spark.sql.catalyst.analysis.CastSupport
+import org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils
 import org.apache.spark.sql.catalyst.expressions._
 import org.apache.spark.sql.catalyst.util.DateTimeUtils
 import org.apache.spark.sql.internal.SQLConf
@@ -195,7 +196,10 @@ class HadoopTableReader(
       def fillPartitionKeys(rawPartValues: Array[String], row: InternalRow): 
Unit = {
         partitionKeyAttrs.foreach { case (attr, ordinal) =>
           val partOrdinal = partitionKeys.indexOf(attr)
-          row(ordinal) = cast(Literal(rawPartValues(partOrdinal)), 
attr.dataType).eval(null)
+          val rawPartValue = rawPartValues(partOrdinal)
+          val partValue =
+            if (rawPartValue == ExternalCatalogUtils.DEFAULT_PARTITION_NAME) 
null else rawPartValue
+          row(ordinal) = cast(Literal(partValue), attr.dataType).eval(null)
         }
       }
 
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
index 50de9246fac4..5b90c00cf5d0 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveTableScanSuite.scala
@@ -254,6 +254,24 @@ class HiveTableScanSuite extends HiveComparisonTest with 
QueryTest with TestHive
     }
   }
 
+  test("SPARK-60801: Read the default partition name as null") {
+    withSQLConf(
+      SQLConf.ANSI_ENABLED.key -> "true",
+      "hive.exec.dynamic.partition.mode" -> "nonstrict") {
+      withTable("t1", "t2") {
+        // The name fails the ANSI cast to a non-string partition column.
+        sql("CREATE TABLE t1 (a INT) USING hive PARTITIONED BY (p INT)")
+        sql("INSERT INTO t1 PARTITION (p) VALUES (10, NULL), (11, 5)")
+        checkAnswer(sql("SELECT a, p FROM t1"), Seq(Row(10, null), Row(11, 5)))
+
+        // The name survives the cast to a string partition column.
+        sql("CREATE TABLE t2 (a INT) USING hive PARTITIONED BY (p STRING)")
+        sql("INSERT INTO t2 PARTITION (p) VALUES (10, NULL), (11, 'x')")
+        checkAnswer(sql("SELECT a, p FROM t2"), Seq(Row(10, null), Row(11, 
"x")))
+      }
+    }
+  }
+
   test("SPARK-32069: Improve error message on reading unexpected directory") {
     withTable("t") {
       withTempDir { f =>
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowPartitionsSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowPartitionsSuite.scala
index 73d1918886ed..51a77c804b4c 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowPartitionsSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowPartitionsSuite.scala
@@ -32,9 +32,7 @@ class ShowPartitionsSuite extends v1.ShowPartitionsSuiteBase 
with CommandSuiteBa
         runShowPartitionsSql(
           s"SHOW PARTITIONS $t",
           Row("part=__HIVE_DEFAULT_PARTITION__") :: Nil)
-        checkAnswer(spark.table(t),
-          Row(0, "__HIVE_DEFAULT_PARTITION__") ::
-          Row(1, "__HIVE_DEFAULT_PARTITION__") :: Nil)
+        checkAnswer(spark.table(t), Row(0, null) :: Row(1, null) :: Nil)
       }
     }
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to