This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9b8cc3f99957 [SPARK-60825][SQL] Fix missing validation of datetime 
partition values when ANSI mode is off
9b8cc3f99957 is described below

commit 9b8cc3f99957d42474ce9ee43470e318282e9887
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Nov 3 12:25:42 2026 -0800

    [SPARK-60825][SQL] Fix missing validation of datetime partition values when 
ANSI mode is off
    
    ### What changes were proposed in this pull request?
    
    This PR fixes `PartitioningUtils.castPartValueToDesiredType` to validate a 
`DATE`, `TIME` or timestamp partition value regardless of 
`spark.sql.ansi.enabled`, as SPARK-60797 did for `DECIMAL`.
    
    ```scala
         case DateType =>
    -      Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
    -    case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
    +      Cast(Literal(value), DateType, Some(zoneId.getId), ansiEnabled = 
true).eval()
    +    case tt: TimeType => Cast(Literal(unescapePathName(value)), tt, 
ansiEnabled = true).eval()
         // Timestamp types
         case dt if AnyTimestampType.acceptsType(dt) || 
AnyTimestampNanoType.acceptsType(dt) =>
    -      Try {
    -        Cast(Literal(unescapePathName(value)), dt, 
Some(zoneId.getId)).eval()
    -      }.getOrElse {
    -        Cast(Cast(Literal(value), DateType, Some(zoneId.getId)), dt).eval()
    -      }
    +      Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId), 
ansiEnabled = true).eval()
         case it: AnsiIntervalType =>
    -      Cast(Literal(unescapePathName(value)), it).eval()
    +      Cast(Literal(unescapePathName(value)), it, ansiEnabled = true).eval()
    ```
    
    `Cast` is in ANSI mode so that a value that cannot be cast fails in any 
session. `parsePartitions` already handles that failure according to 
`spark.sql.sources.validatePartitionColumns`. For a valid value, the ANSI cast 
returns the same value as before, because each ANSI string parser is the 
non-ANSI one that throws an error instead of returning `NULL`. The interval 
case is in ANSI mode too, for consistency. A cast from a string to an interval 
type already fails in both modes, so it d [...]
    
    The timestamp case no longer falls back to a cast to `DATE`. A cast from a 
string to a timestamp type already accepts a date without a time, such as 
`2020-01-01`, so the fallback mattered only for a value with a valid date and 
an invalid time. For `TIMESTAMP` and `TIMESTAMP_LTZ(p)`, it failed anyway, 
because the outer cast from `DATE` has no time zone. For `TIMESTAMP_NTZ` and 
`TIMESTAMP_NTZ(p)`, it returned the start of the day, so a value such as 
`2020-01-01 25:00:00` passed the vali [...]
    
    ### Why are the changes needed?
    
    When a file source reads a partition directory with a user-specified 
`DATE`, `TIME` or timestamp type in a session with 
`spark.sql.ansi.enabled=false`, a value that cannot be cast to the type becomes 
`NULL`, even when `spark.sql.sources.validatePartitionColumns` is `true`, the 
default. In such a session, `Cast` returns `NULL` instead of failing, so 
`parsePartitions` never sees an error.
    
    ```scala
    spark.conf.set("spark.sql.ansi.enabled", false)
    spark.range(1).selectExpr("id AS v").write.parquet(s"$base/a=abc")
    spark.read.schema("v LONG, a DATE").parquet(base).select("a").collect()
    // Expected: INVALID_PARTITION_VALUE, Actual: [null]
    ```
    
    The SQL migration guide has stated since Spark 3.0 that "An exception is 
thrown if the validation fails". `INT`, `BOOLEAN`, `DECIMAL` and interval 
partition values already fail in both modes.
    
    This is a regression from SPARK-34314 in 
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06), 
which removed the `null` check after the cast. I reproduced it for `DATE`, 
`TIMESTAMP` and `TIMESTAMP_NTZ` with 
[v4.0.4](https://github.com/apache/spark/releases/tag/v4.0.4) (2026-07-12), 
[v4.1.3](https://github.com/apache/spark/releases/tag/v4.1.3) (2026-07-11), 
[v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) (2026-07-11) and 
[v4.3.0](https://github.com/a [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These are the results of reading each directory with 
`spark.sql.sources.validatePartitionColumns=true`.
    
    | Directory | Type | Before, ANSI on | Before, ANSI off | After, both modes 
|
    |---|---|---|---|---|
    | `a=abc` | `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ` | 
`INVALID_PARTITION_VALUE` | `NULL` | `INVALID_PARTITION_VALUE` |
    | `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP` | `INVALID_PARTITION_VALUE` | 
`NULL` | `INVALID_PARTITION_VALUE` |
    | `a=2020-01-01 25%3A00%3A00` | `TIMESTAMP_NTZ` | `2020-01-01 00:00:00` | 
`NULL` | `INVALID_PARTITION_VALUE` |
    
    With `spark.sql.sources.validatePartitionColumns=false`, all of them are 
`NULL` after this PR. Valid values, including a date without a time read as a 
timestamp type, are read as before in both modes. Partitions tracked in the 
catalog are not affected, because `CatalogTablePartition.toRow` casts their 
values separately.
    
    The SQL migration guide describes this change.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case, which fails without the fix. 
`FileIndexSuite` reads `a=foo` as `DATE`, `TIME(6)` and `TIMESTAMP`, and 
`a=2020-01-01 25%3A00%3A00` as `TIMESTAMP_NTZ`, in both ANSI modes, with 
`spark.sql.sources.validatePartitionColumns` on and off. It also reads 
`a=2020-01-01` as `TIMESTAMP_NTZ` in both ANSI modes and checks that the value 
is the start of the day.
    
    ```
    $ build/sbt "sql/testOnly *FileIndexSuite -- -z SPARK-60825"
    ```
    
    I also ran a temporary suite, which is not part of this PR. It reads 19 
directory names as `DATE`, `TIME(6)`, `TIMESTAMP`, `TIMESTAMP_NTZ`, 
`TIMESTAMP_LTZ(9)`, `TIMESTAMP_NTZ(9)`, two interval types, `INT`, `BOOLEAN` 
and `DECIMAL(5,2)`, in both ANSI modes, with the validation on and off.
    - After this PR, the two ANSI modes return the same result for all 418 
combinations of directory, type and validation. Before this PR, they differed 
only for the datetime types.
    - With ANSI mode on, the only results that changed are those of 
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)` for the four directories with a valid 
date and an invalid time, such as `a=2020-01-01 xyz`.
    - With ANSI mode off, the only results that changed are `NULL` values that 
became `INVALID_PARTITION_VALUE` with the validation on.
    
    It also casts 32 values to each timestamp type, with the cast in this PR 
and with the removed fallback alone. The fallback returned a value only for 
`TIMESTAMP_NTZ` and `TIMESTAMP_NTZ(9)`. The cast in this PR accepts each of 
those values, such as `2020`, `2020-1-1` or `2020-01-01 12%3A34`, except the 
values whose time part does not parse, such as `2020-01-01T` or `2020-01-01 
UTC`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60043 from dongjoon-hyun/SPARK-60825.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 116f05ca38e2748b6ce18f116d0241361522dabe)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 docs/sql-migration-guide.md                        |  1 +
 .../execution/datasources/PartitioningUtils.scala  | 17 +++---
 .../sql/execution/datasources/FileIndexSuite.scala | 60 +++++++++++++++++++++-
 3 files changed, 67 insertions(+), 11 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 73cdc3cc6c5f..0c46f88db266 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -27,6 +27,7 @@ license: |
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `DECIMAL` type from directory names, it converts each value to 
the precision and scale of that type, rounding half up as `CAST` does. 
Previously, the vectorized Parquet and ORC readers applied the declared scale 
to the digits in the directory name without an error, for example `d=1.5` read 
as `DECIMAL(10,2)` returned `0.15`. Data derived from affected query results 
may need to be recomputed. A value that [...]
 - Since Spark 4.4, when a file source reads a partition column with a 
user-specified `TINYINT` or `SMALLINT` type from directory names, a value out 
of the range of that type fails with `INVALID_PARTITION_VALUE` regardless of 
`spark.sql.ansi.enabled`, as it does for `INT`. Previously, the value wrapped 
around without an error, for example `b=300` read as `TINYINT` returned `44`. 
Data derived from affected query results may need to be recomputed. To get 
`NULL` for such a value, set `spark. [...]
 - Since Spark 4.4, Data Source V2 file scans correctly evaluate deterministic 
Python UDF filter predicates that reference only partition columns. Previously, 
these predicates could be dropped, returning rows that did not satisfy the 
filter (SPARK-60486). Data derived from affected query results may need to be 
recomputed.
+- Since Spark 4.4, when a file source reads a partition column with a 
user-specified datetime type such as `DATE` or `TIMESTAMP` from directory 
names, a value that cannot be cast to that type fails with 
`INVALID_PARTITION_VALUE` regardless of `spark.sql.ansi.enabled`. Previously, 
such a value became `NULL` when `spark.sql.ansi.enabled` was `false`. When it 
was `true`, a `TIMESTAMP_NTZ` value with a valid date and an invalid time, such 
as `2020-01-01 25:00:00`, became the start of that da [...]
 - Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`, 
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of 
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so 
existing paths can match different keys or become invalid. For example, use 
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes 
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set 
`spark.sql.legacy.variantPathB [...]
 
 - Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a field of another column, such as `s.f`, from the value stored in that 
column. Previously, it read the value of the query before the store assignment 
cast, so it could silently store a value that does not match the stored column, 
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION` 
(SPARK-60803). Rows that are already stored keep their values, and data derived 
from them may need to be re [...]
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
index ae2dbcb867b5..0e18212d9e98 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
@@ -588,21 +588,18 @@ object PartitioningUtils extends SQLConfHelper {
     case LongType => JLong.parseLong(value)
     case FloatType => JDouble.parseDouble(value).toFloat
     case DoubleType => JDouble.parseDouble(value)
-    // Cast in ANSI mode to fail on a value that the type cannot hold in any 
session, so that
-    // `spark.sql.sources.validatePartitionColumns` alone decides between an 
error and null.
+    // Cast in ANSI mode, here and in the datetime and interval cases below, 
so that a value that
+    // the type cannot hold fails in any session and 
`spark.sql.sources.validatePartitionColumns`
+    // alone decides between an error and null.
     case dt: DecimalType => Cast(Literal(value), dt, ansiEnabled = true).eval()
     case DateType =>
-      Cast(Literal(value), DateType, Some(zoneId.getId)).eval()
-    case tt: TimeType => Cast(Literal(unescapePathName(value)), tt).eval()
+      Cast(Literal(value), DateType, Some(zoneId.getId), ansiEnabled = 
true).eval()
+    case tt: TimeType => Cast(Literal(unescapePathName(value)), tt, 
ansiEnabled = true).eval()
     // Timestamp types
     case dt if AnyTimestampType.acceptsType(dt) || 
AnyTimestampNanoType.acceptsType(dt) =>
-      Try {
-        Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId)).eval()
-      }.getOrElse {
-        Cast(Cast(Literal(value), DateType, Some(zoneId.getId)), dt).eval()
-      }
+      Cast(Literal(unescapePathName(value)), dt, Some(zoneId.getId), 
ansiEnabled = true).eval()
     case it: AnsiIntervalType =>
-      Cast(Literal(unescapePathName(value)), it).eval()
+      Cast(Literal(unescapePathName(value)), it, ansiEnabled = true).eval()
     case BinaryType => value.getBytes()
     case BooleanType => value.toBoolean
     case dt => throw SparkException.internalError(s"Unsupported partition 
type: $dt")
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
index c0032d7588ab..f044c086dbe9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.execution.datasources
 
 import java.io.{File, FileNotFoundException}
 import java.net.URI
+import java.time.LocalDateTime
 
 import scala.collection.mutable
 import scala.concurrent.duration._
@@ -37,7 +38,7 @@ import org.apache.spark.sql.catalyst.util._
 import org.apache.spark.sql.functions.col
 import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
 import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{ByteType, DecimalType, IntegerType, 
ShortType, StringType, StructField, StructType}
+import org.apache.spark.sql.types.{ByteType, DateType, DecimalType, 
IntegerType, ShortType, StringType, StructField, StructType, TimestampNTZType, 
TimestampType, TimeType}
 import org.apache.spark.util.{HadoopFSUtils, KnownSizeEstimation}
 
 class FileIndexSuite extends SharedSparkSession {
@@ -215,6 +216,63 @@ class FileIndexSuite extends SharedSparkSession {
     }
   }
 
+  test("SPARK-60825: validate datetime partition values regardless of ANSI 
mode") {
+    // A TIMESTAMP_NTZ value with a valid date and an invalid time must not 
become the start of
+    // that day.
+    val cases = Seq(
+      DateType -> "foo",
+      TimeType(6) -> "foo",
+      TimestampType -> "foo",
+      TimestampNTZType -> "2020-01-01 25%3A00%3A00")
+    for ((dataType, value) <- cases; ansiEnabled <- Seq("true", "false")) {
+      withTempDir { dir =>
+        val partitionDirectory = new File(dir, s"a=$value")
+        partitionDirectory.mkdir()
+        stringToFile(new File(partitionDirectory, "text.txt"), "text")
+        val path = new Path(dir.getCanonicalPath)
+        val schema = StructType(Seq(StructField("a", dataType, false)))
+        withSQLConf(
+          SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+          SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "true") {
+          val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty, 
Some(schema))
+          checkError(
+            exception = intercept[SparkRuntimeException] {
+              fileIndex.partitionSpec()
+            },
+            condition = "INVALID_PARTITION_VALUE",
+            parameters = Map(
+              "value" -> s"'$value'",
+              "dataType" -> s"\"${dataType.sql}\"",
+              "columnName" -> "`a`"))
+        }
+        withSQLConf(
+          SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+          SQLConf.VALIDATE_PARTITION_COLUMNS.key -> "false") {
+          val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty, 
Some(schema))
+          val partitionValues = 
fileIndex.partitionSpec().partitions.map(_.values)
+          assert(partitionValues.length == 1 && partitionValues(0).isNullAt(0))
+        }
+      }
+    }
+
+    // A date without a time is a valid timestamp.
+    withTempDir { dir =>
+      val partitionDirectory = new File(dir, "a=2020-01-01")
+      partitionDirectory.mkdir()
+      stringToFile(new File(partitionDirectory, "text.txt"), "text")
+      val path = new Path(dir.getCanonicalPath)
+      val schema = StructType(Seq(StructField("a", TimestampNTZType, false)))
+      val midnight = 
DateTimeUtils.localDateTimeToMicros(LocalDateTime.of(2020, 1, 1, 0, 0))
+      for (ansiEnabled <- Seq("true", "false")) {
+        withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled) {
+          val fileIndex = new InMemoryFileIndex(spark, Seq(path), Map.empty, 
Some(schema))
+          val partitionValues = 
fileIndex.partitionSpec().partitions.map(_.values)
+          assert(partitionValues.length == 1 && partitionValues(0).getLong(0) 
== midnight)
+        }
+      }
+    }
+  }
+
   test("InMemoryFileIndex: input paths are converted to qualified paths") {
     withTempDir { dir =>
       val file = new File(dir, "text.txt")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to