cloud-fan commented on code in PR #37147:
URL: https://github.com/apache/spark/pull/37147#discussion_r921807880
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala:
##########
@@ -356,6 +356,16 @@ class UnivocityParserSuite extends SparkFunSuite with
SQLHelper {
val optionsWithPattern = new CSVOptions(
Map("timestampFormat" -> "invalid", "dateFormat" -> "invalid"), false,
"UTC")
- check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+
+ // With legacy parser enabled, we are still able to parse dates and
timestamps.
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern) {
+ override val isLegacyParserPolicy: Boolean = true
+ })
+
+ // With legacy parser disabled, parsing results in error.
+ val err = intercept[IllegalArgumentException] {
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+ }
+ assert(err.getMessage.contains("Illegal pattern character: n"))
Review Comment:
I think the safest option is to copy-paste the old code of `stringToDate`
before https://github.com/apache/spark/pull/32959 and use it here, but that's
really ugly and hard to maintain.
I'd like to understand more about the invalid pattern behavior. Will we
trigger the fallback for every input row? That sounds like a big perf problem...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
