dongjoon-hyun opened a new pull request, #60003:
URL: https://github.com/apache/spark/pull/60003
[SPARK-XXXXX][SQL] Cast decimal partition values to the user-specified type
### What changes were proposed in this pull request?
This PR fixes `PartitioningUtils.castPartValueToDesiredType` to convert a
decimal partition value to the user-specified `DecimalType`.
```scala
- case _: DecimalType => Literal(new JBigDecimal(value)).value
+ case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled =
true).eval()
```
`Cast` gives the value the declared precision and scale, rounding half up.
It is in ANSI mode so that a value that the type cannot hold fails here in any
session. `parsePartitions` already handles that failure according to
`spark.sql.sources.validatePartitionColumns`.
Alternatives that were not taken:
- `Cast` in the mode of the session, like the `DATE` and `TIME` cases next
to it: with `spark.sql.ansi.enabled=false`, a value that overflows or is not a
number would become `NULL` even when
`spark.sql.sources.validatePartitionColumns` is `true`. `d=abc` fails today, so
this would be a regression.
- Rescale in `ColumnVectorUtils.populate`: this fixes the value for the
vectorized readers, but a value that overflows is still not validated.
### Why are the changes needed?
When a file source reads a partition directory with a user-specified decimal
type, it returns a wrong value without an error if the scale in the directory
name differs from the declared scale.
```scala
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/t/d=1.5")
spark.read.schema("v LONG, d
DECIMAL(10,2)").parquet(s"$base/t").select("d").collect()
// Expected: [1.50], Actual: [0.15]
```
`castPartValueToDesiredType` keeps the precision and scale of the directory
name, so the partition row holds a `Decimal` with the unscaled value `15` and
scale `1` for a `DECIMAL(10,2)` column. `ColumnVectorUtils.populate` copies
that unscaled value into a `ConstantColumnVector`, which then applies the
declared scale when it is read. The other readers are correct only because
`UnsafeRowWriter` changes the precision and scale of the value when it writes
the row.
This affects queries as follows.
- Partition pruning compares the numeric values, so `WHERE d = 1.5` keeps
the partition and returns `0.15`.
- A join with a table that has `d = 1.50` returns no rows, with or without a
runtime bloom filter.
- A directory that Spark wrote also returns a wrong value if the reader
declares another scale. For example, `d=1.50` read as `DECIMAL(12,4)` returns
`0.0150`.
- A catalog table that does not track its partitions
(`spark.sql.hive.manageFilesourcePartitions=false`) reads the directories in
the same way and returns `0.15`.
Partitions tracked in the catalog are not affected, because
`CatalogTablePartition.toRow` already uses `Cast`.
This is a regression from SPARK-34314 in
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06). By
code reading, the value went through `Cast` before that, and a `null` result
failed the validation. I reproduced the wrong values with
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12),
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11),
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11) and
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
### Does this PR introduce _any_ user-facing change?
Yes. These are the results of reading each directory as `DECIMAL(10,2)`.
| Directory | Before, vectorized Parquet and ORC | Before, other readers |
After, all readers |
|---|---|---|---|
| `d=1.5` | `0.15` | `1.50` | `1.50` |
| `d=2` | `0.02` | `2.00` | `2.00` |
| `d=1.555` | `15.55` | `1.56` | `1.56` |
| `d=12345678901.5` | `1234567890.15` | `NULL` | `INVALID_PARTITION_VALUE` |
A value that does not fit the declared precision now fails regardless of
`spark.sql.ansi.enabled`, and it becomes `NULL` when
`spark.sql.sources.validatePartitionColumns` is `false`. The reasons are:
- The SQL migration guide has stated this since Spark 3.0: "An exception is
thrown if the validation fails".
- The `INT` and `BOOLEAN` partition values that do not parse already behave
this way in both ANSI modes, for example `i=99999999999` read as `INT`.
- It is the behavior before SPARK-34314.
Because the conversion now follows `Cast`, it also trims surrounding
whitespace in the directory name, and it rounds a value with more fractional
digits than `DECIMAL(38)` can hold instead of failing.
The SQL migration guide describes this change.
### How was this patch tested?
Pass the CIs with the newly added test cases. Both of them fail without the
fix.
- `ParquetPartitionDiscoverySuite` (V1 and V2) reads `d=1.5`, `d=2` and
`d=1.555` as `DECIMAL(10,2)`. Without the fix, it returns `0.15`, `0.02` and
`15.55`.
- `FileIndexSuite` checks a value that overflows and a value that is not a
number, in both ANSI modes, with `spark.sql.sources.validatePartitionColumns`
on and off.
```
$ build/sbt "sql/testOnly *ParquetV1PartitionDiscoverySuite
*ParquetV2PartitionDiscoverySuite *FileIndexSuite -- -z SPARK-XXXXX"
```
I also ran a temporary suite, which is not part of this PR. It reads Parquet
and ORC with the vectorized reader on and off, and JSON and CSV, through both
V1 and V2, as `DECIMAL(5,2)`, `DECIMAL(10,2)` and `DECIMAL(20,2)`. All 12
combinations return the same value after the fix. It also shows that catalog
partitions added by `MSCK REPAIR TABLE` return the same results before and
after the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
