This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 7d0a478c1d52 [SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT 
partition values when they are out of range
7d0a478c1d52 is described below

commit 7d0a478c1d52bd1ddeb81379ee99a69075aefc7e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 17:43:21 2026 -0800

    [SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT partition values when 
they are out of range
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix a correctness issue of file source partition values. 
`PartitioningUtils.castPartValueToDesiredType` now parses a `TINYINT` or 
`SMALLINT` partition value with `java.lang.Byte.parseByte` and 
`java.lang.Short.parseShort` instead of narrowing the `Int` that 
`Integer.parseInt` returns.
    
    ```scala
    -    case ByteType => Integer.parseInt(value).toByte
    -    case ShortType => Integer.parseInt(value).toShort
    +    case ByteType => JByte.parseByte(value)
    +    case ShortType => JShort.parseShort(value)
    ```
    
    Both methods accept the same strings as `Integer.parseInt` and throw 
`NumberFormatException` for a value out of the range of the type, as 
`Integer.parseInt` does for `INT`. The two callers already handle it.
    - `parsePartitions` fails with `INVALID_PARTITION_VALUE`, or returns `NULL` 
when `spark.sql.sources.validatePartitionColumns` is `false`.
    - `removeLeadingZerosFromNumberTypePartition`, which normalizes a partition 
value for its path on write, throws the exception, or keeps the value as it is 
when `spark.sql.sources.validatePartitionColumns` is `false`.
    
    This is related to #60003 (SPARK-60797), which fixed the `DECIMAL` case of 
the same function.
    
    ### Why are the changes needed?
    
    This is a correctness issue. When a file source reads a partition directory 
with a user-specified `TINYINT` or `SMALLINT` type, it returns a wrong value 
without any error if the value is out of the range of the type, regardless of 
the ANSI mode. A value out of the range of `INT` already fails.
    
    ```scala
    spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t1/b=300")
    spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t2/s=70000")
    spark.range(1).selectExpr("id AS 
v").write.parquet(s"$base/t3/i=99999999999")
    
    // It should fail with INVALID_PARTITION_VALUE, but it returns [44].
    spark.read.schema("v LONG, b 
TINYINT").parquet(s"$base/t1").select("b").collect()
    
    // It should fail with INVALID_PARTITION_VALUE, but it returns [4464].
    spark.read.schema("v LONG, s 
SMALLINT").parquet(s"$base/t2").select("s").collect()
    
    // An out-of-range INT value already fails.
    spark.read.schema("v LONG, i 
INT").parquet(s"$base/t3").select("i").collect()
    // [INVALID_PARTITION_VALUE] Failed to cast value '99999999999' to data 
type "INT" for partition column `i`. ...
    ```
    
    `castPartValueToDesiredType` parses the value as an `Int` and narrows it 
with `toByte` or `toShort`, which keeps only the low-order bits. This affects 
queries as follows.
    - Partition pruning compares the wrapped value, so `WHERE b = 44` keeps the 
directory `b=300`.
    - A catalog table that does not track its partitions 
(`spark.sql.hive.manageFilesourcePartitions=false`) reads the directories in 
the same way and returns `44`.
    - On write, the path of a static partition is computed from the wrapped 
value. With `spark.sql.legacy.skipTypeValidationOnAlterPartition=true` and 
`spark.sql.storeAssignmentPolicy=LEGACY`, `INSERT INTO t PARTITION (b=300)` of 
no rows adds the partition `b=44` to the catalog and creates its directory.
    
    Reading partitions tracked in the catalog is not affected, because 
`CatalogTablePartition.toRow` uses `Cast`.
    
    This issue has existed since SPARK-40212 in 
[v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07), 
which added `toByte` and `toShort` to fix a `ClassCastException` on every 
`TINYINT` and `SMALLINT` partition value. I reproduced the wrong values with 
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12), 
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11), 
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (202 [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These are the results of reading each directory with the 
user-specified type. The results before this PR are the same in both ANSI modes.
    
    | Directory | Type | Before | After |
    |---|---|---|---|
    | `b=300` | `TINYINT` | `44` | `INVALID_PARTITION_VALUE` |
    | `b=-129` | `TINYINT` | `127` | `INVALID_PARTITION_VALUE` |
    | `s=70000` | `SMALLINT` | `4464` | `INVALID_PARTITION_VALUE` |
    | `b=127`, `b=007`, `b=+5` | `TINYINT` | `127`, `7`, `5` | `127`, `7`, `5` |
    
    A value out of range now fails regardless of `spark.sql.ansi.enabled`, and 
it becomes `NULL` when `spark.sql.sources.validatePartitionColumns` is `false`. 
This is how an `INT` value already behaves, and the SQL migration guide has 
stated it since Spark 3.0: "An exception is thrown if the validation fails".
    
    On write, an out-of-range value now throws `NumberFormatException` when 
Spark computes the path of the partition, or it is kept as it is when 
`spark.sql.sources.validatePartitionColumns` is `false`, as an out-of-range 
`INT` value is. So the empty insert above now fails, or adds the partition 
`b=300`. This also changes a table whose catalog tracks such a partition, e.g. 
one that `MSCK REPAIR TABLE` added from the directory `b=300`. An insert that 
does not exclude that partition with a  [...]
    
    The SQL migration guide describes this change.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases. Both of them fail without the 
fix.
    - `FileIndexSuite` reads `a=300` as `TINYINT` and `a=70000` as `SMALLINT`, 
with `spark.sql.sources.validatePartitionColumns` on and off.
    - `PartitioningUtilsSuite` checks 
`removeLeadingZerosFromNumberTypePartition` with the same values.
    
    ```
    $ build/sbt "sql/testOnly *FileIndexSuite *PartitioningUtilsSuite -- -z 
SPARK-60818"
    ```
    
    I also ran a temporary suite, which is not part of this PR. It compares the 
results before and after the fix for in-range and out-of-range values, in both 
ANSI modes, with `spark.sql.sources.validatePartitionColumns` on and off, and 
through both V1 and V2. Only the out-of-range `TINYINT` and `SMALLINT` values 
changed. It also covers catalog partitions added by `MSCK REPAIR TABLE` and 
`ALTER TABLE ADD PARTITION`, and static partition inserts.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60037 from dongjoon-hyun/SPARK-60818.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                        |  1 +
 .../execution/datasources/PartitioningUtils.scala  |  6 ++---
 .../sql/execution/datasources/FileIndexSuite.scala | 31 +++++++++++++++++++++-
 .../datasources/PartitioningUtilsSuite.scala       | 11 ++++++++
 4 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index fdde6419834a..38341b7c6fc4 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -29,6 +29,7 @@ license: |
 ## Upgrading from Spark SQL 4.3 to 4.4
 
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `DECIMAL` type from directory names, it converts each value to 
the precision and scale of that type, rounding half up as `CAST` does. 
Previously, the vectorized Parquet and ORC readers applied the declared scale 
to the digits in the directory name without an error, for example `d=1.5` read 
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results 
may need to be recomputed. A value that [...]
+- Since Spark 4.4, when a file source reads a partition column with a 
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out 
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of 
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped 
around without an error, for example `b=300` read as `TINYINT` returned `44`. 
Data derived from affected query results may need to be recomputed. To get 
`NULL` for such a value, set `spark. [...]
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 570c6dd2b0dc..ae2dbcb867b5 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -17,7 +17,7 @@
 
 package org.apache.spark.sql.execution.datasources
 
-import java.lang.{Double => JDouble, Long => JLong}
+import java.lang.{Byte => JByte, Double => JDouble, Long => JLong, Short => 
JShort}
 import java.math.{BigDecimal => JBigDecimal}
 import java.time.ZoneId
 import java.util.Locale
@@ -582,8 +582,8 @@ object PartitioningUtils extends SQLConfHelper {
     case _ if value == DEFAULT_PARTITION_NAME => null
     case NullType => null
     case _: StringType => UTF8String.fromString(unescapePathName(value))
-    case ByteType => Integer.parseInt(value).toByte
-    case ShortType => Integer.parseInt(value).toShort
+    case ByteType => JByte.parseByte(value)
+    case ShortType => JShort.parseShort(value)
     case IntegerType => Integer.parseInt(value)
     case LongType => JLong.parseLong(value)
     case FloatType => JDouble.parseDouble(value).toFloat
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index f8b425285530..c0032d7588ab 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.util._
 import org.apache.spark.sql.functions.col
 import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{DecimalType, IntegerType, StringType, 
StructField, StructType}
+import org.apache.spark.sql.types.{ByteType, DecimalType, IntegerType, 
ShortType, StringType, StructField, StructType}
 import org.apache.spark.util.{HadoopFSUtils, KnownSizeEstimation}
 
 class FileIndexSuite extends SharedSparkSession {
@@ -186,6 +186,35 @@ class FileIndexSuite extends SharedSparkSession {
     }
   }
 
+  test("SPARK-60818: validate partition values out of the range of TINYINT and 
SMALLINT") {
+    for ((dataType, value) <- Seq(ByteType -> "300", ShortType -> "70000")) {
+      withTempDir { dir =>
+        val partitionDirectory = new File(dir, s"a=$value")
+        partitionDirectory.mkdir()
+        stringToFile(new File(partitionDirectory, "text.txt"), "text")
+        val path = new Path(dir.getCanonicalPath)
+        val schema = StructType(Seq(StructField("a", dataType, false)))
+        withSQLConf(SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "true") {
+          val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty, 
Some(schema))
+          checkError(
+            exception = intercept[SparkRuntimeException] {
+              fileIndex.partitionSpec()
+            },
+            condition = "INVALID_PARTITION_VALUE",
+            parameters = Map(
+              "value" -> s"'$value'",
+              "dataType" -> s"\"${dataType.sql}\"",
+              "columnName" -> "`a`"))
+        }
+        withSQLConf(SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "false") {
+          val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty, 
Some(schema))
+          val partitionValues = 
fileIndex.partitionSpec().partitions.map(_.values)
+          assert(partitionValues.length == 1 && partitionValues(0).isNullAt(0))
+        }
+      }
+    }
+  }
+
   test("InMemoryFileIndex: input paths are converted to qualified paths") {
     withTempDir { dir =>
       val file = new File(dir, "text.txt")
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
index 64c0a5fe217b..1beb2d2fecb4 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
@@ -101,4 +101,15 @@ class PartitioningUtilsSuite extends SparkFunSuite {
         null, StringType, validatePartitionColumns = false) === null
     )
   }
+
+  test("SPARK-60818: removeLeadingZerosFromNumberTypePartition with an 
out-of-range value") {
+    for ((dataType, value) <- Seq(ByteType -> "300", ShortType -> "70000")) {
+      intercept[NumberFormatException] {
+        PartitioningUtils.removeLeadingZerosFromNumberTypePartition(
+          value, dataType, validatePartitionColumns = true)
+      }
+      assert(PartitioningUtils.removeLeadingZerosFromNumberTypePartition(
+        value, dataType, validatePartitionColumns = false) === value)
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to