Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20621#discussion_r168680871
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -407,6 +407,29 @@ object PartitioningUtils {
Literal(bigDecimal)
}
+ val dateTry = Try {
+ // try and parse the date, if no exception occurs this is a
candidate to be resolved as
+ // DateType
+ DateTimeUtils.getThreadLocalDateFormat.parse(raw)
+ // SPARK-23436: Casting the string to date may still return null if
a bad Date is provided.
+ // We need to check that we can cast the raw string since we later
can use Cast to get
+ // the partition values with the right DataType (see
+ //
org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.inferPartitioning)
+ val dateOption = Option(Cast(Literal(raw), DateType).eval())
--- End diff --
Can we add `require(dateOption.isDefine)` with some comments explicitly?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
