This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 2adb6ec82487 [SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT
partition values when they are out of range
2adb6ec82487 is described below
commit 2adb6ec824871298cf7c9738ea07e277f1f3c70c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 17:43:21 2026 -0800
[SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT partition values when
they are out of range
### What changes were proposed in this pull request?
This PR aims to fix a correctness issue of file source partition values.
`PartitioningUtils.castPartValueToDesiredType` now parses a `TINYINT` or
`SMALLINT` partition value with `java.lang.Byte.parseByte` and
`java.lang.Short.parseShort` instead of narrowing the `Int` that
`Integer.parseInt` returns.
```scala
- case ByteType => Integer.parseInt(value).toByte
- case ShortType => Integer.parseInt(value).toShort
+ case ByteType => JByte.parseByte(value)
+ case ShortType => JShort.parseShort(value)
```
Both methods accept the same strings as `Integer.parseInt` and throw
`NumberFormatException` for a value out of the range of the type, as
`Integer.parseInt` does for `INT`. The two callers already handle it.
- `parsePartitions` fails with `INVALID_PARTITION_VALUE`, or returns `NULL`
when `spark.sql.sources.validatePartitionColumns` is `false`.
- `removeLeadingZerosFromNumberTypePartition`, which normalizes a partition
value for its path on write, throws the exception, or keeps the value as it is
when `spark.sql.sources.validatePartitionColumns` is `false`.
This is related to #60003 (SPARK-60797), which fixed the `DECIMAL` case of
the same function.
### Why are the changes needed?
This is a correctness issue. When a file source reads a partition directory
with a user-specified `TINYINT` or `SMALLINT` type, it returns a wrong value
without any error if the value is out of the range of the type, regardless of
the ANSI mode. A value out of the range of `INT` already fails.
```scala
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t1/b=300")
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t2/s=70000")
spark.range(1).selectExpr("id AS
v").write.parquet(s"$base/t3/i=99999999999")
// It should fail with INVALID_PARTITION_VALUE, but it returns [44].
spark.read.schema("v LONG, b
TINYINT").parquet(s"$base/t1").select("b").collect()
// It should fail with INVALID_PARTITION_VALUE, but it returns [4464].
spark.read.schema("v LONG, s
SMALLINT").parquet(s"$base/t2").select("s").collect()
// An out-of-range INT value already fails.
spark.read.schema("v LONG, i
INT").parquet(s"$base/t3").select("i").collect()
// [INVALID_PARTITION_VALUE] Failed to cast value '99999999999' to data
type "INT" for partition column `i`. ...
```
`castPartValueToDesiredType` parses the value as an `Int` and narrows it
with `toByte` or `toShort`, which keeps only the low-order bits. This affects
queries as follows.
- Partition pruning compares the wrapped value, so `WHERE b = 44` keeps the
directory `b=300`.
- A catalog table that does not track its partitions
(`spark.sql.hive.manageFilesourcePartitions=false`) reads the directories in
the same way and returns `44`.
- On write, the path of a static partition is computed from the wrapped
value. With `spark.sql.legacy.skipTypeValidationOnAlterPartition=true` and
`spark.sql.storeAssignmentPolicy=LEGACY`, `INSERT INTO t PARTITION (b=300)` of
no rows adds the partition `b=44` to the catalog and creates its directory.
Reading partitions tracked in the catalog is not affected, because
`CatalogTablePartition.toRow` uses `Cast`.
This issue has existed since SPARK-40212 in
[v3.4.0](https://github.com/apache/spark/releases/tag/v3.4.0) (2023-04-07),
which added `toByte` and `toShort` to fix a `ClassCastException` on every
`TINYINT` and `SMALLINT` partition value. I reproduced the wrong values with
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12),
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11),
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (202 [...]
### Does this PR introduce _any_ user-facing change?
Yes. These are the results of reading each directory with the
user-specified type. The results before this PR are the same in both ANSI modes.
| Directory | Type | Before | After |
|---|---|---|---|
| `b=300` | `TINYINT` | `44` | `INVALID_PARTITION_VALUE` |
| `b=-129` | `TINYINT` | `127` | `INVALID_PARTITION_VALUE` |
| `s=70000` | `SMALLINT` | `4464` | `INVALID_PARTITION_VALUE` |
| `b=127`, `b=007`, `b=+5` | `TINYINT` | `127`, `7`, `5` | `127`, `7`, `5` |
A value out of range now fails regardless of `spark.sql.ansi.enabled`, and
it becomes `NULL` when `spark.sql.sources.validatePartitionColumns` is `false`.
This is how an `INT` value already behaves, and the SQL migration guide has
stated it since Spark 3.0: "An exception is thrown if the validation fails".
On write, an out-of-range value now throws `NumberFormatException` when
Spark computes the path of the partition, or it is kept as it is when
`spark.sql.sources.validatePartitionColumns` is `false`, as an out-of-range
`INT` value is. So the empty insert above now fails, or adds the partition
`b=300`. This also changes a table whose catalog tracks such a partition, e.g.
one that `MSCK REPAIR TABLE` added from the directory `b=300`. An insert that
does not exclude that partition with a [...]
The SQL migration guide describes this change.
### How was this patch tested?
Pass the CIs with the newly added test cases. Both of them fail without the
fix.
- `FileIndexSuite` reads `a=300` as `TINYINT` and `a=70000` as `SMALLINT`,
with `spark.sql.sources.validatePartitionColumns` on and off.
- `PartitioningUtilsSuite` checks
`removeLeadingZerosFromNumberTypePartition` with the same values.
```
$ build/sbt "sql/testOnly *FileIndexSuite *PartitioningUtilsSuite -- -z
SPARK-60818"
```
I also ran a temporary suite, which is not part of this PR. It compares the
results before and after the fix for in-range and out-of-range values, in both
ANSI modes, with `spark.sql.sources.validatePartitionColumns` on and off, and
through both V1 and V2. Only the out-of-range `TINYINT` and `SMALLINT` values
changed. It also covers catalog partitions added by `MSCK REPAIR TABLE` and
`ALTER TABLE ADD PARTITION`, and static partition inserts.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60037 from dongjoon-hyun/SPARK-60818.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 7d0a478c1d52bd1ddeb81379ee99a69075aefc7e)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../execution/datasources/PartitioningUtils.scala | 6 ++---
.../sql/execution/datasources/FileIndexSuite.scala | 31 +++++++++++++++++++++-
.../datasources/PartitioningUtilsSuite.scala | 11 ++++++++
4 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 4d22edbe604f..30bbaf7bc163 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -25,6 +25,7 @@ license: |
## Upgrading from Spark SQL 4.3 to 4.4
- Since Spark 4.4, when a file source reads a partition column with a
user-specified `DECIMAL` type from directory names, it converts each value to
the precision and scale of that type, rounding half up as `CAST` does.
Previously, the vectorized Parquet and ORC readers applied the declared scale
to the digits in the directory name without an error, for example `d=1.5` read
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results
may need to be recomputed. A value that [...]
+- Since Spark 4.4, when a file source reads a partition column with a
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped
around without an error, for example `b=300` read as `TINYINT` returned `44`.
Data derived from affected query results may need to be recomputed. To get
`NULL` for such a value, set `spark. [...]
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index 570c6dd2b0dc..ae2dbcb867b5 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.datasources
-import java.lang.{Double => JDouble, Long => JLong}
+import java.lang.{Byte => JByte, Double => JDouble, Long => JLong, Short =>
JShort}
import java.math.{BigDecimal => JBigDecimal}
import java.time.ZoneId
import java.util.Locale
@@ -582,8 +582,8 @@ object PartitioningUtils extends SQLConfHelper {
case _ if value == DEFAULT_PARTITION_NAME => null
case NullType => null
case _: StringType => UTF8String.fromString(unescapePathName(value))
- case ByteType => Integer.parseInt(value).toByte
- case ShortType => Integer.parseInt(value).toShort
+ case ByteType => JByte.parseByte(value)
+ case ShortType => JShort.parseShort(value)
case IntegerType => Integer.parseInt(value)
case LongType => JLong.parseLong(value)
case FloatType => JDouble.parseDouble(value).toFloat
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index f8b425285530..c0032d7588ab 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{DecimalType, IntegerType, StringType,
StructField, StructType}
+import org.apache.spark.sql.types.{ByteType, DecimalType, IntegerType,
ShortType, StringType, StructField, StructType}
import org.apache.spark.util.{HadoopFSUtils, KnownSizeEstimation}
class FileIndexSuite extends SharedSparkSession {
@@ -186,6 +186,35 @@ class FileIndexSuite extends SharedSparkSession {
}
}
+ test("SPARK-60818: validate partition values out of the range of TINYINT and
SMALLINT") {
+ for ((dataType, value) <- Seq(ByteType -> "300", ShortType -> "70000")) {
+ withTempDir { dir =>
+ val partitionDirectory = new File(dir, s"a=$value")
+ partitionDirectory.mkdir()
+ stringToFile(new File(partitionDirectory, "text.txt"), "text")
+ val path = new Path(dir.getCanonicalPath)
+ val schema = StructType(Seq(StructField("a", dataType, false)))
+ withSQLConf(SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "true") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ fileIndex.partitionSpec()
+ },
+ condition = "INVALID_PARTITION_VALUE",
+ parameters = Map(
+ "value" -> s"'$value'",
+ "dataType" -> s"\"${dataType.sql}\"",
+ "columnName" -> "`a`"))
+ }
+ withSQLConf(SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "false") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ val partitionValues =
fileIndex.partitionSpec().partitions.map(_.values)
+ assert(partitionValues.length == 1 && partitionValues(0).isNullAt(0))
+ }
+ }
+ }
+ }
+
test("InMemoryFileIndex: input paths are converted to qualified paths") {
withTempDir { dir =>
val file = new File(dir, "text.txt")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
index 64c0a5fe217b..1beb2d2fecb4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PartitioningUtilsSuite.scala
@@ -101,4 +101,15 @@ class PartitioningUtilsSuite extends SparkFunSuite {
null, StringType, validatePartitionColumns = false) === null
)
}
+
+ test("SPARK-60818: removeLeadingZerosFromNumberTypePartition with an
out-of-range value") {
+ for ((dataType, value) <- Seq(ByteType -> "300", ShortType -> "70000")) {
+ intercept[NumberFormatException] {
+ PartitioningUtils.removeLeadingZerosFromNumberTypePartition(
+ value, dataType, validatePartitionColumns = true)
+ }
+ assert(PartitioningUtils.removeLeadingZerosFromNumberTypePartition(
+ value, dataType, validatePartitionColumns = false) === value)
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]