Jonathancui123 commented on code in PR #37147:
URL: https://github.com/apache/spark/pull/37147#discussion_r921696654
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala:
##########
@@ -356,6 +356,16 @@ class UnivocityParserSuite extends SparkFunSuite with
SQLHelper {
val optionsWithPattern = new CSVOptions(
Map("timestampFormat" -> "invalid", "dateFormat" -> "invalid"), false,
"UTC")
- check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+
+ // With legacy parser enabled, we are still able to parse dates and
timestamps.
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern) {
+ override val isLegacyParserPolicy: Boolean = true
+ })
+
+ // With legacy parser disabled, parsing results in error.
+ val err = intercept[IllegalArgumentException] {
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+ }
+ assert(err.getMessage.contains("Illegal pattern character: n"))
Review Comment:
> I can also add a feature flag to control this behaviour in JSON and CSV
connectors so users can always opt in to use legacy behaviour.
I think this should work. It feels weird that users have to opt-in to the
correct behavior but hopefully this is a small percentage of users. Maybe
@kamcheungting-db or @cloud-fan can weigh in.
> There could an alternative fix, maybe we can look into updating
DateTimeUtils.stringToDate but I am not sure
I personally wouldn't be confident updating `DateTimeUtils.stringToDate`
because there are so many usages elsewhere. But if you are familiar with the
other use cases of `DateTimeUtils.stringToDate` then this could work.
I'll loop back if I think of an alternative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
