dongjoon-hyun opened a new pull request, #60043:
URL: https://github.com/apache/spark/pull/60043
### What changes were proposed in this pull request?
This PR fixes `PartitioningUtils.castPartValueToDesiredType` to validate a
`DATE`, `TIME` or timestamp partition value regardless of
`spark.sql.ansi.enabled`, as SPARK-60797 did for `DECIMAL`.
```scala
case DateType =>
- Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
- case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
+ Cast(Literal(value), DateType, Some(zoneId.getId), ansiEnabled =
true).eval()
+ case tt: TimeType => Cast(Literal(unescapePathName(value)), tt,
ansiEnabled = true).eval()
// Timestamp types
case dt if AnyTimestampType.acceptsType(dt) ||
AnyTimestampNanoType.acceptsType(dt) =>
- Try {
- Cast(Literal(unescapePathName(value)), dt,
Some(zoneId.getId)).eval()
- }.getOrElse {
- Cast(Cast(Literal(value), DateType, Some(zoneId.getId)), dt).eval()
- }
+ Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId),
ansiEnabled = true).eval()
```
`Cast` is in ANSI mode so that a value that cannot be cast fails in any
session. `parsePartitions` already handles that failure according to
`spark.sql.sources.validatePartitionColumns`. For a valid value, the ANSI cast
returns the same value as before, because each ANSI string parser is the
non-ANSI one that throws an error instead of returning `NULL`.
The timestamp case no longer falls back to a cast to `DATE`. A cast from a
string to a timestamp type already accepts a date without a time, such as
`2020-01-01`, so the fallback mattered only for a value with a valid date and
an invalid time. For `TIMESTAMP` and `TIMESTAMP_LTZ(p)`, it failed anyway,
because the outer cast from `DATE` has no time zone. For `TIMESTAMP_NTZ` and
`TIMESTAMP_NTZ(p)`, it returned the start of the day, so a value such as
`2020-01-01 25:00:00` passed the validation in an ANSI session.
### Why are the changes needed?
When a file source reads a partition directory with a user-specified `DATE`,
`TIME` or timestamp type in a session with `spark.sql.ansi.enabled=false`, a
value that cannot be cast to the type becomes `NULL`, even when
`spark.sql.sources.validatePartitionColumns` is `true`, the default. In such a
session, `Cast` returns `NULL` instead of failing, so `parsePartitions` never
sees an error.
```scala
spark.conf.set("spark.sql.ansi.enabled", false)
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/a=abc")
spark.read.schema("v LONG, a DATE").parquet(base).select("a").collect()
// Expected: INVALID_PARTITION_VALUE, Actual: [null]
```
The SQL migration guide has stated since Spark 3.0 that "An exception is
thrown if the validation fails". `INT`, `BOOLEAN`, `DECIMAL` and interval
partition values already fail in both modes.
This is a regression from SPARK-34314 in
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06),
which removed the `null` check after the cast. I reproduced it for `DATE`,
`TIMESTAMP` and `TIMESTAMP_NTZ` with
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12),
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11),
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11) and
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
### Does this PR introduce _any_ user-facing change?
Yes. These are the results of reading each directory with
`spark.sql.sources.validatePartitionColumns=true`.
| Directory | Type | Before, ANSI on | Before, ANSI off | After, both modes |
|---|---|---|---|---|
| `a=abc` | `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ` |
`INVALID_PARTITION_VALUE` | `NULL` | `INVALID_PARTITION_VALUE` |
| `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP` | `INVALID_PARTITION_VALUE` |
`NULL` | `INVALID_PARTITION_VALUE` |
| `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP_NTZ` | `2020-01-01 00:00:00` |
`NULL` | `INVALID_PARTITION_VALUE` |
With `spark.sql.sources.validatePartitionColumns=false`, all of them are
`NULL` after this PR. Valid values, including a date without a time read as a
timestamp type, are read as before in both modes. Partitions tracked in the
catalog are not affected, because `CatalogTablePartition.toRow` casts their
values separately.
The SQL migration guide describes this change.
### How was this patch tested?
Pass the CIs with the newly added test case, which fails without the fix.
`FileIndexSuite` reads `a=foo` as `DATE` and `TIME(6)`, and `a=2020-01-01
25%3A00%3A00` as `TIMESTAMP_NTZ`, in both ANSI modes, with
`spark.sql.sources.validatePartitionColumns` on and off.
```
$ build/sbt "sql/testOnly *FileIndexSuite -- -z SPARK-60825"
```
I also ran a temporary suite, which is not part of this PR. It reads 19
directory names as `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ`,
`TIMESTAMP_LTZ(9)`, `TIMESTAMP_NTZ(9)`, two interval types, `INT`, `BOOLEAN`
and `DECIMAL(5,2)`, in both ANSI modes, with the validation on and off.
- After this PR, the two ANSI modes return the same result for all 418
combinations of directory, type and validation. Before this PR, they differed
only for the datetime types.
- With ANSI mode on, the only results that changed are those of
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)` for the four directories with a valid
date and an invalid time, such as `a=2020-01-01 xyz`.
- With ANSI mode off, the only results that changed are `NULL` values that
became `INVALID_PARTITION_VALUE` with the validation on.
It also casts 32 values to each timestamp type, with the cast in this PR and
with the removed fallback alone. The fallback returned a value only for
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)`. The cast in this PR accepts each of
those values, such as `2020`, `2020-1-1` or `2020-01-01 12%3A34`, except the
values whose time part does not parse, such as `2020-01-01T` or `2020-01-01
UTC`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
