This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ab5bb6c7b09a [SPARK-60797][SQL] Fix wrong decimal partition values
when the scale differs from the schema
ab5bb6c7b09a is described below
commit ab5bb6c7b09adeba5a5796b1e70c0d369402eb98
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 08:08:56 2026 -0800
[SPARK-60797][SQL] Fix wrong decimal partition values when the scale
differs from the schema
### What changes were proposed in this pull request?
This PR fixes `PartitioningUtils.castPartValueToDesiredType` to convert a
decimal partition value to the user-specified `DecimalType`.
```scala
- case _: DecimalType => Literal(new JBigDecimal(value)).value
+ case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled =
true).eval()
```
`Cast` gives the value the declared precision and scale, rounding half up.
It is in ANSI mode so that a value that the type cannot hold fails here in any
session. `parsePartitions` already handles that failure according to
`spark.sql.sources.validatePartitionColumns`.
Alternatives that were not taken:
- `Cast` in the mode of the session, like the `DATE` and `TIME` cases next
to it: with `spark.sql.ansi.enabled=false`, a value that overflows or is not a
number would become `NULL` even when
`spark.sql.sources.validatePartitionColumns` is `true`. `d=abc` fails today, so
this would be a regression.
- Rescale in `ColumnVectorUtils.populate`: this fixes the value for the
vectorized readers, but a value that overflows is still not validated.
### Why are the changes needed?
When a file source reads a partition directory with a user-specified
decimal type, it returns a wrong value without an error if the scale in the
directory name differs from the declared scale.
```scala
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t/d=1.5")
spark.read.schema("v LONG, d
DECIMAL(10,2)").parquet(s"$base/t").select("d").collect()
// Expected: [1.50], Actual: [0.15]
```
`castPartValueToDesiredType` keeps the precision and scale of the directory
name, so the partition row holds a `Decimal` with the unscaled value `15` and
scale `1` for a `DECIMAL(10,2)` column. `ColumnVectorUtils.populate` copies
that unscaled value into a `ConstantColumnVector`, which then applies the
declared scale when it is read. The other readers are correct only because
`UnsafeRowWriter` changes the precision and scale of the value when it writes
the row.
This affects queries as follows.
- Partition pruning compares the numeric values, so `WHERE d = 1.5` keeps
the partition and returns `0.15`.
- A join with a table that has `d = 1.50` returns no rows, with or without
a runtime bloom filter.
- A directory that Spark wrote also returns a wrong value if the reader
declares another scale. For example, `d=1.50` read as `DECIMAL(12,4)` returns
`0.0150`.
- A catalog table that does not track its partitions
(`spark.sql.hive.manageFilesourcePartitions=false`) reads the directories in
the same way and returns `0.15`.
Partitions tracked in the catalog are not affected, because
`CatalogTablePartition.toRow` already uses `Cast`.
This is a regression from SPARK-34314 in
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06). By
code reading, the value went through `Cast` before that, and a `null` result
failed the validation. I reproduced the wrong values with
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12),
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11),
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11) and
[v4 [...]
### Does this PR introduce _any_ user-facing change?
Yes. These are the results of reading each directory as `DECIMAL(10,2)`.
| Directory | Before, vectorized Parquet and ORC | Before, other readers |
After, all readers |
|---|---|---|---|
| `d=1.5` | `0.15` | `1.50` | `1.50` |
| `d=2` | `0.02` | `2.00` | `2.00` |
| `d=1.555` | `15.55` | `1.56` | `1.56` |
| `d=12345678901.5` | `1234567890.15` | `NULL` | `INVALID_PARTITION_VALUE` |
A value that does not fit the declared precision now fails regardless of
`spark.sql.ansi.enabled`, and it becomes `NULL` when
`spark.sql.sources.validatePartitionColumns` is `false`. The reasons are:
- The SQL migration guide has stated this since Spark 3.0: "An exception is
thrown if the validation fails".
- The `INT` and `BOOLEAN` partition values that do not parse already behave
this way in both ANSI modes, for example `i=99999999999` read as `INT`.
- It is the behavior before SPARK-34314.
Because the conversion now follows `Cast`, it also trims surrounding
whitespace in the directory name, and it rounds a value with more fractional
digits than `DECIMAL(38)` can hold instead of failing.
The SQL migration guide describes this change.
### How was this patch tested?
Pass the CIs with the newly added test cases. Both of them fail without the
fix.
- `ParquetPartitionDiscoverySuite` (V1 and V2) reads `d=1.5`, `d=2` and
`d=1.555` as `DECIMAL(10,2)`. Without the fix, it returns `0.15`, `0.02` and
`15.55`.
- `FileIndexSuite` checks a value that overflows and a value that is not a
number, in both ANSI modes, with `spark.sql.sources.validatePartitionColumns`
on and off.
```
$ build/sbt "sql/testOnly *ParquetV1PartitionDiscoverySuite
*ParquetV2PartitionDiscoverySuite *FileIndexSuite -- -z SPARK-60797"
```
I also ran a temporary suite, which is not part of this PR. It reads
Parquet and ORC with the vectorized reader on and off, and JSON and CSV,
through both V1 and V2, as `DECIMAL(5,2)`, `DECIMAL(10,2)` and `DECIMAL(20,2)`.
All 12 combinations return the same value after the fix. It also shows that
catalog partitions added by `MSCK REPAIR TABLE` return the same results before
and after the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60003 from dongjoon-hyun/SPARK-60797.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 532495d8bf28a95d8b2d9e5fb9ee640081bc108b)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../execution/datasources/PartitioningUtils.scala | 4 ++-
.../sql/execution/datasources/FileIndexSuite.scala | 36 +++++++++++++++++++++-
.../parquet/ParquetPartitionDiscoverySuite.scala | 15 +++++++++
4 files changed, 54 insertions(+), 2 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index c2fd1f5cea57..f7c89757a8bc 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -24,6 +24,7 @@ license: |
## Upgrading from Spark SQL 4.3 to 4.4
+- Since Spark 4.4, when a file source reads a partition column with a
user-specified `DECIMAL` type from directory names, it converts each value to
the precision and scale of that type, rounding half up as `CAST` does.
Previously, the vectorized Parquet and ORC readers applied the declared scale
to the digits in the directory name without an error, for example `d=1.5` read
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results
may need to be recomputed. A value that [...]
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index ef45e49021ed..570c6dd2b0dc 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -588,7 +588,9 @@ object PartitioningUtils extends SQLConfHelper {
case LongType => JLong.parseLong(value)
case FloatType => JDouble.parseDouble(value).toFloat
case DoubleType => JDouble.parseDouble(value)
- case _: DecimalType => Literal(new JBigDecimal(value)).value
+ // Cast in ANSI mode to fail on a value that the type cannot hold in any
session, so that
+ // `spark.sql.sources.validatePartitionColumns` alone decides between an
error and null.
+ case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled = true).eval()
case DateType =>
Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index f6ccf1be62c5..f8b425285530 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{IntegerType, StringType, StructField,
StructType}
+import org.apache.spark.sql.types.{DecimalType, IntegerType, StringType,
StructField, StructType}
import org.apache.spark.util.{HadoopFSUtils, KnownSizeEstimation}
class FileIndexSuite extends SharedSparkSession {
@@ -152,6 +152,40 @@ class FileIndexSuite extends SharedSparkSession {
}
}
+ test("SPARK-60797: validate decimal partition values regardless of ANSI
mode") {
+ // These values overflow the declared precision and are not a number,
respectively.
+ for (value <- Seq("1234.5", "foo"); ansiEnabled <- Seq("true", "false")) {
+ withTempDir { dir =>
+ val partitionDirectory = new File(dir, s"a=$value")
+ partitionDirectory.mkdir()
+ stringToFile(new File(partitionDirectory, "text.txt"), "text")
+ val path = new Path(dir.getCanonicalPath)
+ val schema = StructType(Seq(StructField("a", DecimalType(5, 2),
false)))
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+ SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "true") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ fileIndex.partitionSpec()
+ },
+ condition = "INVALID_PARTITION_VALUE",
+ parameters = Map(
+ "value" -> s"'$value'",
+ "dataType" -> "\"DECIMAL(5,2)\"",
+ "columnName" -> "`a`"))
+ }
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+ SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "false") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ val partitionValues =
fileIndex.partitionSpec().partitions.map(_.values)
+ assert(partitionValues.length == 1 && partitionValues(0).isNullAt(0))
+ }
+ }
+ }
+ }
+
test("InMemoryFileIndex: input paths are converted to qualified paths") {
withTempDir { dir =>
val file = new File(dir, "text.txt")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala
index 7c9ac3bfd035..afaf92f184ec 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala
@@ -1212,6 +1212,21 @@ abstract class ParquetPartitionDiscoverySuite
}
}
+ test("SPARK-60797: read decimal partition values with the user-specified
scale") {
+ withTempDir { dir =>
+ Seq("1.5", "2", "1.555").foreach { value =>
+ spark.range(1).write.parquet(new File(dir,
s"d=$value").getCanonicalPath)
+ }
+ val res = spark.read
+ .schema("id LONG, d DECIMAL(10, 2)")
+ .parquet(dir.getCanonicalPath)
+ checkAnswer(res, Seq(
+ Row(0L, BigDecimal("1.50")),
+ Row(0L, BigDecimal("2.00")),
+ Row(0L, BigDecimal("1.56"))))
+ }
+ }
+
test("SPARK-49163: Attempt to create table based on broken parquet partition
data") {
withTempDir { dir =>
val data = Seq[(String, String, String)](("a", "b", "c"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]