Jonathancui123 commented on code in PR #37147:
URL: https://github.com/apache/spark/pull/37147#discussion_r921635019
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/csv/UnivocityParserSuite.scala:
##########
@@ -356,6 +356,16 @@ class UnivocityParserSuite extends SparkFunSuite with
SQLHelper {
val optionsWithPattern = new CSVOptions(
Map("timestampFormat" -> "invalid", "dateFormat" -> "invalid"), false,
"UTC")
- check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+
+ // With legacy parser enabled, we are still able to parse dates and
timestamps.
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern) {
+ override val isLegacyParserPolicy: Boolean = true
+ })
+
+ // With legacy parser disabled, parsing results in error.
+ val err = intercept[IllegalArgumentException] {
+ check(new UnivocityParser(StructType(Seq.empty), optionsWithPattern))
+ }
+ assert(err.getMessage.contains("Illegal pattern character: n"))
Review Comment:
Is this technically a breaking change for users who could previously specify
an invalid pattern without LEGACY mode?
Before -- ignore the pattern and parse with `DateTimeUtils.stringToTimestamp`
Now -- it throws an error
We don't support invalid patterns but as a user I would be unhappy to see my
code break. I'm unsure if this is actually considered a breaking change because
this is such an edge case and the user is already doing something invalid. I'm
curious to hear your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
