xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389060387
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,35 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as
well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API
classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also
breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This
function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in
SPARK-31030.
+ *
+ * @param pattern - the input pattern.
+ * @return the pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ patternPart
+ // The meaning of 'u' was day number of week in Java 7, it changed
to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the
string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the
legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and
ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise,
return NULL as what Spark
+ // 2.4 does.
+ .replace("u", "e")
Review comment:
Actually the quoted text has been considered, let me add comments to
emphasize.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
