sadikovi commented on code in PR #37147:
URL: https://github.com/apache/spark/pull/37147#discussion_r921642735
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala:
##########
@@ -356,6 +356,16 @@ class UnivocityParserSuite extends SparkFunSuite with
SQLHelper {
val optionsWithPattern = new CSVOptions(
Map("timestampFormat" -> "invalid", "dateFormat" -> "invalid"), false,
"UTC")
- check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+
+ // With legacy parser enabled, we are still able to parse dates and
timestamps.
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern) {
+ override val isLegacyParserPolicy: Boolean = true
+ })
+
+ // With legacy parser disabled, parsing results in error.
+ val err = intercept[IllegalArgumentException] {
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+ }
+ assert(err.getMessage.contains("Illegal pattern character: n"))
Review Comment:
This is a good point. It would be a breaking change for users if they were
relying on the compatibility fallback.
There could an alternative fix, maybe we can look into updating
`DateTimeUtils.stringToDate` but I am not sure.
I can also add a feature flag to control this behaviour in JSON and CSV
connectors so users can always opt in to use legacy behaviour. For example, I
can a data source option "useLegacyParsing" or something similar. The option
could be disabled by default, the exception would contain a message saying that
you can enable the option to maintain the previous behaviour. Maybe this could
be a good solution.
Let me know if something like that could work, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
