viirya commented on code in PR #60043:
URL: https://github.com/apache/spark/pull/60043#discussion_r4459777535
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala:
##########
@@ -588,19 +588,16 @@ object PartitioningUtils extends SQLConfHelper {
case LongType => JLong.parseLong(value)
case FloatType => JDouble.parseDouble(value).toFloat
case DoubleType => JDouble.parseDouble(value)
- // Cast in ANSI mode to fail on a value that the type cannot hold in any
session, so that
- // `spark.sql.sources.validatePartitionColumns` alone decides between an
error and null.
+ // Cast in ANSI mode, here and in the datetime cases below, to fail on a
value that the type
Review Comment:
nit: the `AnsiIntervalType` case below still uses the default cast mode.
That's fine, because the string-to-interval cast throws in either mode. But
after "here and in the datetime cases below", a reader may wonder why intervals
are left out. Maybe pass `ansiEnabled = true` there as well, or note that the
interval cast already fails in both modes.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala:
##########
@@ -215,6 +215,45 @@ class FileIndexSuite extends SharedSparkSession {
}
}
+ test("SPARK-60825: validate datetime partition values regardless of ANSI
mode") {
+ // A TIMESTAMP_NTZ value with a valid date and an invalid time must not
become the start of
+ // that day.
+ val cases = Seq(
+ DateType -> "foo",
+ TimeType(6) -> "foo",
+ TimestampNTZType -> "2020-01-01 25%3A00%3A00")
Review Comment:
Could we add `TimestampType -> "foo"` here? `TIMESTAMP` is the most common
case and goes through a different parser than NTZ. It would also help to add
one positive check, such as `a=2020-01-01` read as `TIMESTAMP_NTZ` giving
midnight. That is the case the removed fallback would have covered, and I
couldn't find an existing test that reads a date-only directory with a
user-specified timestamp type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
