This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 116f05ca38e2 [SPARK-60825][SQL] Fix missing validation of datetime
partition values when ANSI mode is off
116f05ca38e2 is described below
commit 116f05ca38e2748b6ce18f116d0241361522dabe
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 12:25:42 2026 -0800
[SPARK-60825][SQL] Fix missing validation of datetime partition values when
ANSI mode is off
### What changes were proposed in this pull request?
This PR fixes `PartitioningUtils.castPartValueToDesiredType` to validate a
`DATE`, `TIME` or timestamp partition value regardless of
`spark.sql.ansi.enabled`, as SPARK-60797 did for `DECIMAL`.
```scala
case DateType =>
- Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
- case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
+ Cast(Literal(value), DateType, Some(zoneId.getId), ansiEnabled =
true).eval()
+ case tt: TimeType => Cast(Literal(unescapePathName(value)), tt,
ansiEnabled = true).eval()
// Timestamp types
case dt if AnyTimestampType.acceptsType(dt) ||
AnyTimestampNanoType.acceptsType(dt) =>
- Try {
- Cast(Literal(unescapePathName(value)), dt,
Some(zoneId.getId)).eval()
- }.getOrElse {
- Cast(Cast(Literal(value), DateType, Some(zoneId.getId)), dt).eval()
- }
+ Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId),
ansiEnabled = true).eval()
case it: AnsiIntervalType =>
- Cast(Literal(unescapePathName(value)), it).eval()
+ Cast(Literal(unescapePathName(value)), it, ansiEnabled = true).eval()
```
`Cast` is in ANSI mode so that a value that cannot be cast fails in any
session. `parsePartitions` already handles that failure according to
`spark.sql.sources.validatePartitionColumns`. For a valid value, the ANSI cast
returns the same value as before, because each ANSI string parser is the
non-ANSI one that throws an error instead of returning `NULL`. The interval
case is in ANSI mode too, for consistency. A cast from a string to an interval
type already fails in both modes, so it d [...]
The timestamp case no longer falls back to a cast to `DATE`. A cast from a
string to a timestamp type already accepts a date without a time, such as
`2020-01-01`, so the fallback mattered only for a value with a valid date and
an invalid time. For `TIMESTAMP` and `TIMESTAMP_LTZ(p)`, it failed anyway,
because the outer cast from `DATE` has no time zone. For `TIMESTAMP_NTZ` and
`TIMESTAMP_NTZ(p)`, it returned the start of the day, so a value such as
`2020-01-01 25:00:00` passed the vali [...]
### Why are the changes needed?
When a file source reads a partition directory with a user-specified
`DATE`, `TIME` or timestamp type in a session with
`spark.sql.ansi.enabled=false`, a value that cannot be cast to the type becomes
`NULL`, even when `spark.sql.sources.validatePartitionColumns` is `true`, the
default. In such a session, `Cast` returns `NULL` instead of failing, so
`parsePartitions` never sees an error.
```scala
spark.conf.set("spark.sql.ansi.enabled", false)
spark.range(1).selectExpr("id AS v").write.parquet(s"$base/a=abc")
spark.read.schema("v LONG, a DATE").parquet(base).select("a").collect()
// Expected: INVALID_PARTITION_VALUE, Actual: [null]
```
The SQL migration guide has stated since Spark 3.0 that "An exception is
thrown if the validation fails". `INT`, `BOOLEAN`, `DECIMAL` and interval
partition values already fail in both modes.
This is a regression from SPARK-34314 in
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06),
which removed the `null` check after the cast. I reproduced it for `DATE`,
`TIMESTAMP` and `TIMESTAMP_NTZ` with
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12),
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11),
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11) and
[v4.3.0](https://github.com/a [...]
### Does this PR introduce _any_ user-facing change?
Yes. These are the results of reading each directory with
`spark.sql.sources.validatePartitionColumns=true`.
| Directory | Type | Before, ANSI on | Before, ANSI off | After, both modes
|
|---|---|---|---|---|
| `a=abc` | `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ` |
`INVALID_PARTITION_VALUE` | `NULL` | `INVALID_PARTITION_VALUE` |
| `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP` | `INVALID_PARTITION_VALUE` |
`NULL` | `INVALID_PARTITION_VALUE` |
| `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP_NTZ` | `2020-01-01 00:00:00` |
`NULL` | `INVALID_PARTITION_VALUE` |
With `spark.sql.sources.validatePartitionColumns=false`, all of them are
`NULL` after this PR. Valid values, including a date without a time read as a
timestamp type, are read as before in both modes. Partitions tracked in the
catalog are not affected, because `CatalogTablePartition.toRow` casts their
values separately.
The SQL migration guide describes this change.
### How was this patch tested?
Pass the CIs with the newly added test case, which fails without the fix.
`FileIndexSuite` reads `a=foo` as `DATE`, `TIME(6)` and `TIMESTAMP`, and
`a=2020-01-01 25%3A00%3A00` as `TIMESTAMP_NTZ`, in both ANSI modes, with
`spark.sql.sources.validatePartitionColumns` on and off. It also reads
`a=2020-01-01` as `TIMESTAMP_NTZ` in both ANSI modes and checks that the value
is the start of the day.
```
$ build/sbt "sql/testOnly *FileIndexSuite -- -z SPARK-60825"
```
I also ran a temporary suite, which is not part of this PR. It reads 19
directory names as `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ`,
`TIMESTAMP_LTZ(9)`, `TIMESTAMP_NTZ(9)`, two interval types, `INT`, `BOOLEAN`
and `DECIMAL(5,2)`, in both ANSI modes, with the validation on and off.
- After this PR, the two ANSI modes return the same result for all 418
combinations of directory, type and validation. Before this PR, they differed
only for the datetime types.
- With ANSI mode on, the only results that changed are those of
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)` for the four directories with a valid
date and an invalid time, such as `a=2020-01-01 xyz`.
- With ANSI mode off, the only results that changed are `NULL` values that
became `INVALID_PARTITION_VALUE` with the validation on.
It also casts 32 values to each timestamp type, with the cast in this PR
and with the removed fallback alone. The fallback returned a value only for
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)`. The cast in this PR accepts each of
those values, such as `2020`, `2020-1-1` or `2020-01-01 12%3A34`, except the
values whose time part does not parse, such as `2020-01-01T` or `2020-01-01
UTC`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60043 from dongjoon-hyun/SPARK-60825.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../execution/datasources/PartitioningUtils.scala | 17 +++---
.../sql/execution/datasources/FileIndexSuite.scala | 60 +++++++++++++++++++++-
3 files changed, 67 insertions(+), 11 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 4c2dc2e427a5..ba9ec5dbea04 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -31,6 +31,7 @@ license: |
- Since Spark 4.4, when a file source reads a partition column with a
user-specified `DECIMAL` type from directory names, it converts each value to
the precision and scale of that type, rounding half up as `CAST` does.
Previously, the vectorized Parquet and ORC readers applied the declared scale
to the digits in the directory name without an error, for example `d=1.5` read
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results
may need to be recomputed. A value that [...]
- Since Spark 4.4, when a file source reads a partition column with a
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped
around without an error, for example `b=300` read as `TINYINT` returned `44`.
Data derived from affected query results may need to be recomputed. To get
`NULL` for such a value, set `spark. [...]
- Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic
Python UDF filter predicates that reference only partition columns. Previously,
these predicates could be dropped, returning rows that did not satisfy the
filter (SPARK-60486). Data derived from affected query results may need to be
recomputed.
+- Since Spark 4.4, when a file source reads a partition column with a
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory
names, a value that cannot be cast to that type fails with
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously,
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such
as `2020-01-01 25:00:00`, became the start of that da [...]
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index ae2dbcb867b5..0e18212d9e98 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -588,21 +588,18 @@ object PartitioningUtils extends SQLConfHelper {
case LongType => JLong.parseLong(value)
case FloatType => JDouble.parseDouble(value).toFloat
case DoubleType => JDouble.parseDouble(value)
- // Cast in ANSI mode to fail on a value that the type cannot hold in any
session, so that
- // `spark.sql.sources.validatePartitionColumns` alone decides between an
error and null.
+ // Cast in ANSI mode, here and in the datetime and interval cases below,
so that a value that
+ // the type cannot hold fails in any session and
`spark.sql.sources.validatePartitionColumns`
+ // alone decides between an error and null.
case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled = true).eval()
case DateType =>
- Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
- case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
+ Cast(Literal(value), DateType, Some(zoneId.getId), ansiEnabled =
true).eval()
+ case tt: TimeType => Cast(Literal(unescapePathName(value)), tt,
ansiEnabled = true).eval()
// Timestamp types
case dt if AnyTimestampType.acceptsType(dt) ||
AnyTimestampNanoType.acceptsType(dt) =>
- Try {
- Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId)).eval()
- }.getOrElse {
- Cast(Cast(Literal(value), DateType, Some(zoneId.getId)), dt).eval()
- }
+ Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId),
ansiEnabled = true).eval()
case it: AnsiIntervalType =>
- Cast(Literal(unescapePathName(value)), it).eval()
+ Cast(Literal(unescapePathName(value)), it, ansiEnabled = true).eval()
case BinaryType => value.getBytes()
case BooleanType => value.toBoolean
case dt => throw SparkException.internalError(s"Unsupported partition
type: $dt")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index c0032d7588ab..f044c086dbe9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.datasources
import java.io.{File, FileNotFoundException}
import java.net.URI
+import java.time.LocalDateTime
import scala.collection.mutable
import scala.concurrent.duration._
@@ -37,7 +38,7 @@ import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{ByteType, DecimalType, IntegerType,
ShortType, StringType, StructField, StructType}
+import org.apache.spark.sql.types.{ByteType, DateType, DecimalType,
IntegerType, ShortType, StringType, StructField, StructType, TimestampNTZType,
TimestampType, TimeType}
import org.apache.spark.util.{HadoopFSUtils, KnownSizeEstimation}
class FileIndexSuite extends SharedSparkSession {
@@ -215,6 +216,63 @@ class FileIndexSuite extends SharedSparkSession {
}
}
+ test("SPARK-60825: validate datetime partition values regardless of ANSI
mode") {
+ // A TIMESTAMP_NTZ value with a valid date and an invalid time must not
become the start of
+ // that day.
+ val cases = Seq(
+ DateType -> "foo",
+ TimeType(6) -> "foo",
+ TimestampType -> "foo",
+ TimestampNTZType -> "2020-01-01 25%3A00%3A00")
+ for ((dataType, value) <- cases; ansiEnabled <- Seq("true", "false")) {
+ withTempDir { dir =>
+ val partitionDirectory = new File(dir, s"a=$value")
+ partitionDirectory.mkdir()
+ stringToFile(new File(partitionDirectory, "text.txt"), "text")
+ val path = new Path(dir.getCanonicalPath)
+ val schema = StructType(Seq(StructField("a", dataType, false)))
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+ SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "true") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ checkError(
+ exception = intercept[SparkRuntimeException] {
+ fileIndex.partitionSpec()
+ },
+ condition = "INVALID_PARTITION_VALUE",
+ parameters = Map(
+ "value" -> s"'$value'",
+ "dataType" -> s"\"${dataType.sql}\"",
+ "columnName" -> "`a`"))
+ }
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+ SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "false") {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ val partitionValues =
fileIndex.partitionSpec().partitions.map(_.values)
+ assert(partitionValues.length == 1 && partitionValues(0).isNullAt(0))
+ }
+ }
+ }
+
+ // A date without a time is a valid timestamp.
+ withTempDir { dir =>
+ val partitionDirectory = new File(dir, "a=2020-01-01")
+ partitionDirectory.mkdir()
+ stringToFile(new File(partitionDirectory, "text.txt"), "text")
+ val path = new Path(dir.getCanonicalPath)
+ val schema = StructType(Seq(StructField("a", TimestampNTZType, false)))
+ val midnight =
DateTimeUtils.localDateTimeToMicros(LocalDateTime.of(2020, 1, 1, 0, 0))
+ for (ansiEnabled <- Seq("true", "false")) {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled) {
+ val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty,
Some(schema))
+ val partitionValues =
fileIndex.partitionSpec().partitions.map(_.values)
+ assert(partitionValues.length == 1 && partitionValues(0).getLong(0)
== midnight)
+ }
+ }
+ }
+ }
+
test("InMemoryFileIndex: input paths are converted to qualified paths") {
withTempDir { dir =>
val file = new File(dir, "text.txt")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]