This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new fcdffbd [SPARK-31771][SQL][3.0] Disable Narrow TextStyle for datetime
pattern 'G/M/L/E/u/Q/q'
fcdffbd is described below
commit fcdffbd5c626de371754fae6a32e2cfdccd9153c
Author: Kent Yao <[email protected]>
AuthorDate: Tue May 26 11:30:14 2020 +0000
[SPARK-31771][SQL][3.0] Disable Narrow TextStyle for datetime pattern
'G/M/L/E/u/Q/q'
THIS PR BACKPORTS
https://github.com/apache/spark/commit/695cb617d42507eded9c7e50bc7cd5333bbe6f83
TO BRANCH-3.0
### What changes were proposed in this pull request?
Five continuous pattern characters with 'G/M/L/E/u/Q/q' means Narrow-Text
Style while we turn to use `java.time.DateTimeFormatterBuilder` since 3.0.0,
which output the leading single letter of the value, e.g. `December` would be
`D`. In Spark 2.4 they mean Full-Text Style.
In this PR, we explicitly disable Narrow-Text Style for these pattern
characters.
### Why are the changes needed?
Without this change, there will be a silent data change.
### Does this PR introduce _any_ user-facing change?
Yes, queries with datetime operations using datetime patterns, e.g.
`G/M/L/E/u` will fail if the pattern length is 5 and other patterns, e,g. 'k',
'm' also can accept a certain number of letters.
1. datetime patterns that are not supported by the new parser but the
legacy will get SparkUpgradeException, e.g. "GGGGG", "MMMMM", "LLLLL", "EEEEE",
"uuuuu", "aa", "aaa". 2 options are given to end-users, one is to use legacy
mode, and the other is to follow the new online doc for correct datetime
patterns
2, datetime patterns that are not supported by both the new parser and the
legacy, e.g. "QQQQQ", "qqqqq", will get IllegalArgumentException which is
captured by Spark internally and results NULL to end-users.
### How was this patch tested?
add unit tests
Closes #28637 from yaooqinn/SPARK-31771-30.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-ref-datetime-pattern.md | 50 +++---
.../spark/sql/catalyst/csv/UnivocityParser.scala | 4 +-
.../catalyst/expressions/datetimeExpressions.scala | 4 +
.../spark/sql/catalyst/json/JacksonParser.scala | 4 +-
.../spark/sql/catalyst/util/DateFormatter.scala | 16 +-
.../catalyst/util/DateTimeFormatterHelper.scala | 34 +++-
.../sql/catalyst/util/TimestampFormatter.scala | 15 +-
.../expressions/DateExpressionsSuite.scala | 2 +-
.../util/DateTimeFormatterHelperSuite.scala | 11 ++
.../spark/sql/util/TimestampFormatterSuite.scala | 11 ++
.../resources/sql-tests/inputs/datetime-legacy.sql | 2 +
.../test/resources/sql-tests/inputs/datetime.sql | 20 +++
.../sql-tests/results/ansi/datetime.sql.out | 163 +++++++++++++++++-
.../{datetime.sql.out => datetime-legacy.sql.out} | 188 ++++++++++++++++++---
.../resources/sql-tests/results/datetime.sql.out | 163 +++++++++++++++++-
.../native/stringCastAndExpressions.sql.out | 15 +-
.../org/apache/spark/sql/DateFunctionsSuite.scala | 20 ++-
17 files changed, 650 insertions(+), 72 deletions(-)
diff --git a/docs/sql-ref-datetime-pattern.md b/docs/sql-ref-datetime-pattern.md
index df19b9c..0e00e7b 100644
--- a/docs/sql-ref-datetime-pattern.md
+++ b/docs/sql-ref-datetime-pattern.md
@@ -30,25 +30,25 @@ Spark uses pattern letters in the following table for date
and timestamp parsing
|Symbol|Meaning|Presentation|Examples|
|------|-------|------------|--------|
-|**G**|era|text|AD; Anno Domini; A|
+|**G**|era|text|AD; Anno Domini|
|**y**|year|year|2020; 20|
-|**D**|day-of-year|number|189|
-|**M/L**|month-of-year|number/text|7; 07; Jul; July; J|
-|**d**|day-of-month|number|28|
+|**D**|day-of-year|number(3)|189|
+|**M/L**|month-of-year|month|7; 07; Jul; July|
+|**d**|day-of-month|number(3)|28|
|**Q/q**|quarter-of-year|number/text|3; 03; Q3; 3rd quarter|
|**Y**|week-based-year|year|1996; 96|
-|**w**|week-of-week-based-year|number|27|
-|**W**|week-of-month|number|4|
-|**E**|day-of-week|text|Tue; Tuesday; T|
-|**u**|localized day-of-week|number/text|2; 02; Tue; Tuesday; T|
-|**F**|week-of-month|number|3|
-|**a**|am-pm-of-day|text|PM|
-|**h**|clock-hour-of-am-pm (1-12)|number|12|
-|**K**|hour-of-am-pm (0-11)|number|0|
-|**k**|clock-hour-of-day (1-24)|number|0|
-|**H**|hour-of-day (0-23)|number|0|
-|**m**|minute-of-hour|number|30|
-|**s**|second-of-minute|number|55|
+|**w**|week-of-week-based-year|number(2)|27|
+|**W**|week-of-month|number(1)|4|
+|**E**|day-of-week|text|Tue; Tuesday|
+|**u**|localized day-of-week|number/text|2; 02; Tue; Tuesday|
+|**F**|week-of-month|number(1)|3|
+|**a**|am-pm-of-day|am-pm|PM|
+|**h**|clock-hour-of-am-pm (1-12)|number(2)|12|
+|**K**|hour-of-am-pm (0-11)|number(2)|0|
+|**k**|clock-hour-of-day (1-24)|number(2)|0|
+|**H**|hour-of-day (0-23)|number(2)|0|
+|**m**|minute-of-hour|number(2)|30|
+|**s**|second-of-minute|number(2)|55|
|**S**|fraction-of-second|fraction|978|
|**V**|time-zone ID|zone-id|America/Los_Angeles; Z; -08:30|
|**z**|time-zone name|zone-name|Pacific Standard Time; PST|
@@ -63,9 +63,9 @@ Spark uses pattern letters in the following table for date
and timestamp parsing
The count of pattern letters determines the format.
-- Text: The text style is determined based on the number of pattern letters
used. Less than 4 pattern letters will use the short form. Exactly 4 pattern
letters will use the full form. Exactly 5 pattern letters will use the narrow
form. Six or more letters will fail.
+- Text: The text style is determined based on the number of pattern letters
used. Less than 4 pattern letters will use the short form. Exactly 4 pattern
letters will use the full form. Exactly 5 pattern letters will use the narrow
form. 5 or more letters will fail.
-- Number: If the count of letters is one, then the value is output using the
minimum number of digits and without padding. Otherwise, the count of digits is
used as the width of the output field, with the value zero-padded as necessary.
The following pattern letters have constraints on the count of letters. Only
one letter 'F' can be specified. Up to two letters of 'd', 'H', 'h', 'K', 'k',
'm', and 's' can be specified. Up to three letters of 'D' can be specified.
+- Number(n): The n here represents the maximum count of letters this type of
datetime pattern can be used. If the count of letters is one, then the value is
output using the minimum number of digits and without padding. Otherwise, the
count of digits is used as the width of the output field, with the value
zero-padded as necessary.
- Number/Text: If the count of pattern letters is 3 or greater, use the Text
rules above. Otherwise use the Number rules above.
@@ -76,8 +76,7 @@ The count of pattern letters determines the format.
- Year: The count of letters determines the minimum field width below which
padding is used. If the count of letters is two, then a reduced two digit form
is used. For printing, this outputs the rightmost two digits. For parsing, this
will parse using the base value of 2000, resulting in a year within the range
2000 to 2099 inclusive. If the count of letters is less than four (but not
two), then the sign is only output for negative years. Otherwise, the sign is
output if the pad width is [...]
-- Month: If the number of pattern letters is 3 or more, the month is
interpreted as text; otherwise, it is interpreted as a number. The text form is
depend on letters - 'M' denotes the 'standard' form, and 'L' is for
'stand-alone' form. The difference between the 'standard' and 'stand-alone'
forms is trickier to describe as there is no difference in English. However, in
other languages there is a difference in the word used when the text is used
alone, as opposed to in a complete date. F [...]
- - `'M'` or `'L'`: Month number in a year starting from 1. There is no
difference between 'M' and 'L'. Month from 1 to 9 are printed without padding.
+- Month: If the number of pattern letters is 3 or more, the month is
interpreted as text; otherwise, it is interpreted as a number. The text form is
depend on letters - 'M' denotes the 'standard' form, and 'L' is for
'stand-alone' form. The difference between the 'standard' and 'stand-alone'
forms is trickier to describe as there is no difference in English. However, in
other languages there is a difference in the word used when the text is used
alone, as opposed to in a complete date. F [...]
```sql
spark-sql> select date_format(date '1970-01-01', "M");
1
@@ -119,13 +118,8 @@ The count of pattern letters determines the format.
spark-sql> select to_csv(named_struct('date', date '1970-01-01'),
map('dateFormat', 'LLLL', 'locale', 'RU'));
январь
```
- - `'LLLLL'` or `'MMMMM'`: Narrow textual representation of standard or
stand-alone forms. Typically it is a single letter.
- ```sql
- spark-sql> select date_format(date '1970-07-01', "LLLLL");
- J
- spark-sql> select date_format(date '1970-01-01', "MMMMM");
- J
- ```
+
+- am-pm: This outputs the am-pm-of-day. Pattern letter count must be 1.
- Zone ID(V): This outputs the display the time-zone ID. Pattern letter count
must be 2.
@@ -147,5 +141,3 @@ More details for the text style:
- Short Form: Short text, typically an abbreviation. For example, day-of-week
Monday might output "Mon".
- Full Form: Full text, typically the full description. For example,
day-of-week Monday might output "Monday".
-
-- Narrow Form: Narrow text, typically a single letter. For example,
day-of-week Monday might output "M".
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
index 8e87a827..f2bb7db 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala
@@ -85,13 +85,13 @@ class UnivocityParser(
// We preallocate it avoid unnecessary allocations.
private val noRows = None
- private val timestampFormatter = TimestampFormatter(
+ private lazy val timestampFormatter = TimestampFormatter(
options.timestampFormat,
options.zoneId,
options.locale,
legacyFormat = FAST_DATE_FORMAT,
needVarLengthSecondFraction = true)
- private val dateFormatter = DateFormatter(
+ private lazy val dateFormatter = DateFormatter(
options.dateFormat,
options.zoneId,
options.locale,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 6e95f64..a0cf446 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -801,6 +801,7 @@ abstract class ToTimestamp
legacyFormat = SIMPLE_DATE_FORMAT,
needVarLengthSecondFraction = true)
} catch {
+ case e: SparkUpgradeException => throw e
case NonFatal(_) => null
}
@@ -982,6 +983,7 @@ case class FromUnixTime(sec: Expression, format:
Expression, timeZoneId: Option[
legacyFormat = SIMPLE_DATE_FORMAT,
needVarLengthSecondFraction = false)
} catch {
+ case e: SparkUpgradeException => throw e
case NonFatal(_) => null
}
@@ -997,6 +999,7 @@ case class FromUnixTime(sec: Expression, format:
Expression, timeZoneId: Option[
try {
UTF8String.fromString(formatter.format(time.asInstanceOf[Long] *
MICROS_PER_SECOND))
} catch {
+ case e: SparkUpgradeException => throw e
case NonFatal(_) => null
}
}
@@ -1014,6 +1017,7 @@ case class FromUnixTime(sec: Expression, format:
Expression, timeZoneId: Option[
needVarLengthSecondFraction = false)
.format(time.asInstanceOf[Long] * MICROS_PER_SECOND))
} catch {
+ case e: SparkUpgradeException => throw e
case NonFatal(_) => null
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
index ef98793..c4f6121 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
@@ -56,13 +56,13 @@ class JacksonParser(
private val factory = options.buildJsonFactory()
- private val timestampFormatter = TimestampFormatter(
+ private lazy val timestampFormatter = TimestampFormatter(
options.timestampFormat,
options.zoneId,
options.locale,
legacyFormat = FAST_DATE_FORMAT,
needVarLengthSecondFraction = true)
- private val dateFormatter = DateFormatter(
+ private lazy val dateFormatter = DateFormatter(
options.dateFormat,
options.zoneId,
options.locale,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
index 8261f57..c4cedf2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
@@ -33,6 +33,8 @@ sealed trait DateFormatter extends Serializable {
def format(days: Int): String
def format(date: Date): String
def format(localDate: LocalDate): String
+
+ def validatePatternString(): Unit
}
class Iso8601DateFormatter(
@@ -70,6 +72,12 @@ class Iso8601DateFormatter(
override def format(date: Date): String = {
legacyFormatter.format(date)
}
+
+ override def validatePatternString(): Unit = {
+ try {
+ formatter
+ } catch checkLegacyFormatter(pattern,
legacyFormatter.validatePatternString)
+ }
}
trait LegacyDateFormatter extends DateFormatter {
@@ -93,6 +101,8 @@ class LegacyFastDateFormatter(pattern: String, locale:
Locale) extends LegacyDat
private lazy val fdf = FastDateFormat.getInstance(pattern, locale)
override def parseToDate(s: String): Date = fdf.parse(s)
override def format(d: Date): String = fdf.format(d)
+
+ override def validatePatternString(): Unit = fdf
}
class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends
LegacyDateFormatter {
@@ -100,6 +110,8 @@ class LegacySimpleDateFormatter(pattern: String, locale:
Locale) extends LegacyD
private lazy val sdf = new SimpleDateFormat(pattern, locale)
override def parseToDate(s: String): Date = sdf.parse(s)
override def format(d: Date): String = sdf.format(d)
+
+ override def validatePatternString(): Unit = sdf
}
object DateFormatter {
@@ -118,7 +130,9 @@ object DateFormatter {
if (SQLConf.get.legacyTimeParserPolicy == LEGACY) {
getLegacyFormatter(pattern, zoneId, locale, legacyFormat)
} else {
- new Iso8601DateFormatter(pattern, zoneId, locale, legacyFormat)
+ val df = new Iso8601DateFormatter(pattern, zoneId, locale, legacyFormat)
+ df.validatePatternString()
+ df
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
index 35f95db..8289568 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelper.scala
@@ -117,6 +117,34 @@ trait DateTimeFormatterHelper {
s"set ${SQLConf.LEGACY_TIME_PARSER_POLICY.key} to LEGACY to restore
the behavior " +
s"before Spark 3.0, or set to CORRECTED and treat it as an invalid
datetime string.", e)
}
+
+ /**
+ * When the new DateTimeFormatter failed to initialize because of invalid
datetime pattern, it
+ * will throw IllegalArgumentException. If the pattern can be recognized by
the legacy formatter
+ * it will raise SparkUpgradeException to tell users to restore the previous
behavior via LEGACY
+ * policy or follow our guide to correct their pattern. Otherwise, the
original
+ * IllegalArgumentException will be thrown.
+ *
+ * @param pattern the date time pattern
+ * @param tryLegacyFormatter a func to capture exception, identically which
forces a legacy
+ * datetime formatter to be initialized
+ */
+
+ protected def checkLegacyFormatter(
+ pattern: String,
+ tryLegacyFormatter: => Unit): PartialFunction[Throwable,
DateTimeFormatter] = {
+ case e: IllegalArgumentException =>
+ try {
+ tryLegacyFormatter
+ } catch {
+ case _: Throwable => throw e
+ }
+ throw new SparkUpgradeException("3.0", s"Fail to recognize '$pattern'
pattern in the" +
+ s" DateTimeFormatter. 1) You can set
${SQLConf.LEGACY_TIME_PARSER_POLICY.key} to LEGACY" +
+ s" to restore the behavior before Spark 3.0. 2) You can form a valid
datetime pattern" +
+ s" with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html";,
+ e)
+ }
}
private object DateTimeFormatterHelper {
@@ -190,7 +218,8 @@ private object DateTimeFormatterHelper {
}
final val unsupportedLetters = Set('A', 'c', 'e', 'n', 'N', 'p')
-
+ final val unsupportedNarrowTextStyle =
+ Set("GGGGG", "MMMMM", "LLLLL", "EEEEE", "uuuuu", "QQQQQ", "qqqqq", "uuuuu")
/**
* In Spark 3.0, we switch to the Proleptic Gregorian calendar and use
DateTimeFormatter for
* parsing/formatting datetime values. The pattern string is incompatible
with the one defined
@@ -211,6 +240,9 @@ private object DateTimeFormatterHelper {
for (c <- patternPart if unsupportedLetters.contains(c)) {
throw new IllegalArgumentException(s"Illegal pattern character:
$c")
}
+ for (style <- unsupportedNarrowTextStyle if
patternPart.contains(style)) {
+ throw new IllegalArgumentException(s"Too many pattern letters:
${style.head}")
+ }
// The meaning of 'u' was day number of week in SimpleDateFormat, it
was changed to year
// in DateTimeFormatter. Substitute 'u' to 'e' and use
DateTimeFormatter to parse the
// string. If parsable, return the result; otherwise, fall back to
'u', and then use the
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
index d7d6368..32b4dcd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala
@@ -54,6 +54,7 @@ sealed trait TimestampFormatter extends Serializable {
def format(us: Long): String
def format(ts: Timestamp): String
def format(instant: Instant): String
+ def validatePatternString(): Unit
}
class Iso8601TimestampFormatter(
@@ -99,6 +100,12 @@ class Iso8601TimestampFormatter(
override def format(ts: Timestamp): String = {
legacyFormatter.format(ts)
}
+
+ override def validatePatternString(): Unit = {
+ try {
+ formatter
+ } catch checkLegacyFormatter(pattern,
legacyFormatter.validatePatternString)
+ }
}
/**
@@ -202,6 +209,8 @@ class LegacyFastTimestampFormatter(
override def format(instant: Instant): String = {
format(instantToMicros(instant))
}
+
+ override def validatePatternString(): Unit = fastDateFormat
}
class LegacySimpleTimestampFormatter(
@@ -231,6 +240,8 @@ class LegacySimpleTimestampFormatter(
override def format(instant: Instant): String = {
format(instantToMicros(instant))
}
+
+ override def validatePatternString(): Unit = sdf
}
object LegacyDateFormats extends Enumeration {
@@ -255,8 +266,10 @@ object TimestampFormatter {
if (SQLConf.get.legacyTimeParserPolicy == LEGACY) {
getLegacyFormatter(pattern, zoneId, locale, legacyFormat)
} else {
- new Iso8601TimestampFormatter(
+ val tf = new Iso8601TimestampFormatter(
pattern, zoneId, locale, legacyFormat, needVarLengthSecondFraction)
+ tf.validatePatternString()
+ tf
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
index dd5a532..5b5c85f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
@@ -267,7 +267,7 @@ class DateExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
// Test escaping of format
GenerateUnsafeProjection.generate(
- DateFormatClass(Literal(ts), Literal("\"quote"), JST_OPT) :: Nil)
+ DateFormatClass(Literal(ts), Literal("\""), JST_OPT) :: Nil)
// SPARK-28072 The codegen path should work
checkEvaluation(
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
index 817e503..34a1ad2 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
@@ -40,6 +40,17 @@ class DateTimeFormatterHelperSuite extends SparkFunSuite {
val e =
intercept[IllegalArgumentException](convertIncompatiblePattern(s"yyyy-MM-dd $l
G"))
assert(e.getMessage === s"Illegal pattern character: $l")
}
+
+ unsupportedNarrowTextStyle.foreach { style =>
+ val e1 = intercept[IllegalArgumentException] {
+ convertIncompatiblePattern(s"yyyy-MM-dd $style")
+ }
+ assert(e1.getMessage === s"Too many pattern letters: ${style.head}")
+ val e2 = intercept[IllegalArgumentException] {
+ convertIncompatiblePattern(s"yyyy-MM-dd $style${style.head}")
+ }
+ assert(e2.getMessage === s"Too many pattern letters: ${style.head}")
+ }
assert(convertIncompatiblePattern("yyyy-MM-dd uuuu") === "uuuu-MM-dd eeee")
assert(convertIncompatiblePattern("yyyy-MM-dd EEEE") === "uuuu-MM-dd EEEE")
assert(convertIncompatiblePattern("yyyy-MM-dd'e'HH:mm:ss") ===
"uuuu-MM-dd'e'HH:mm:ss")
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
index dccb3de..4324d3c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/util/TimestampFormatterSuite.scala
@@ -396,4 +396,15 @@ class TimestampFormatterSuite extends SparkFunSuite with
SQLHelper with Matchers
val micros = formatter.parse("2009 11")
assert(micros === date(2009, 1, 1, 11))
}
+
+ test("explicitly forbidden datetime patterns") {
+ // not support by the legacy one too
+ Seq("QQQQQ", "qqqqq", "A", "c", "e", "n", "N", "p").foreach { pattern =>
+ intercept[IllegalArgumentException](TimestampFormatter(pattern,
UTC).format(0))
+ }
+ // supported by the legacy one, then we will suggest users with
SparkUpgradeException
+ Seq("GGGGG", "MMMMM", "LLLLL", "EEEEE", "uuuuu", "aa", "aaa").foreach {
pattern =>
+ intercept[SparkUpgradeException](TimestampFormatter(pattern,
UTC).format(0))
+ }
+ }
}
diff --git a/sql/core/src/test/resources/sql-tests/inputs/datetime-legacy.sql
b/sql/core/src/test/resources/sql-tests/inputs/datetime-legacy.sql
new file mode 100644
index 0000000..e573f8a
--- /dev/null
+++ b/sql/core/src/test/resources/sql-tests/inputs/datetime-legacy.sql
@@ -0,0 +1,2 @@
+--SET spark.sql.legacy.timeParserPolicy=LEGACY
+--IMPORT datetime.sql
\ No newline at end of file
diff --git a/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
index 9be857e..99b7cf9 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/datetime.sql
@@ -130,3 +130,23 @@ select to_date("16", "dd");
select to_date("02-29", "MM-dd");
select to_timestamp("2019 40", "yyyy mm");
select to_timestamp("2019 10:10:10", "yyyy hh:mm:ss");
+
+-- Unsupported narrow text style
+select date_format(date '2020-05-23', 'GGGGG');
+select date_format(date '2020-05-23', 'MMMMM');
+select date_format(date '2020-05-23', 'LLLLL');
+select date_format(timestamp '2020-05-23', 'EEEEE');
+select date_format(timestamp '2020-05-23', 'uuuuu');
+select date_format('2020-05-23', 'QQQQQ');
+select date_format('2020-05-23', 'qqqqq');
+select to_timestamp('2019-10-06 A', 'yyyy-MM-dd GGGGG');
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEEE');
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE');
+select unix_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE');
+select from_unixtime(12345, 'MMMMM');
+select from_unixtime(54321, 'QQQQQ');
+select from_unixtime(23456, 'aaaaa');
+select from_json('{"time":"26/October/2015"}', 'time Timestamp',
map('timestampFormat', 'dd/MMMMM/yyyy'));
+select from_json('{"date":"26/October/2015"}', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'));
+select from_csv('26/October/2015', 'time Timestamp', map('timestampFormat',
'dd/MMMMM/yyyy'));
+select from_csv('26/October/2015', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'));
diff --git
a/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
b/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
index 89596c7..ff30bb0 100644
--- a/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
--- Number of queries: 91
+-- Number of queries: 109
-- !query
@@ -778,3 +778,164 @@ select to_timestamp("2019 10:10:10", "yyyy hh:mm:ss")
struct<to_timestamp('2019 10:10:10', 'yyyy hh:mm:ss'):timestamp>
-- !query output
2019-01-01 10:10:10
+
+
+-- !query
+select date_format(date '2020-05-23', 'GGGGG')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'GGGGG' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(date '2020-05-23', 'MMMMM')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'MMMMM' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(date '2020-05-23', 'LLLLL')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'LLLLL' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'uuuuu')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'uuuuu' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format('2020-05-23', 'QQQQQ')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Too many pattern letters: Q
+
+
+-- !query
+select date_format('2020-05-23', 'qqqqq')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Too many pattern letters: q
+
+
+-- !query
+select to_timestamp('2019-10-06 A', 'yyyy-MM-dd GGGGG')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'yyyy-MM-dd GGGGG' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select unix_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_unixtime(12345, 'MMMMM')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'MMMMM' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_unixtime(54321, 'QQQQQ')
+-- !query schema
+struct<from_unixtime(CAST(54321 AS BIGINT), QQQQQ):string>
+-- !query output
+NULL
+
+
+-- !query
+select from_unixtime(23456, 'aaaaa')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'aaaaa' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_json('{"time":"26/October/2015"}', 'time Timestamp',
map('timestampFormat', 'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_json('{"date":"26/October/2015"}', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_csv('26/October/2015', 'time Timestamp', map('timestampFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_csv('26/October/2015', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
diff --git a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
b/sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
old mode 100755
new mode 100644
similarity index 84%
copy from sql/core/src/test/resources/sql-tests/results/datetime.sql.out
copy to sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
index b26eb2f..30743f8
--- a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
--- Number of queries: 91
+-- Number of queries: 109
-- !query
@@ -475,7 +475,7 @@ select to_timestamp('2019-10-06 10:11:12.0', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.0', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12
+NULL
-- !query
@@ -483,7 +483,7 @@ select to_timestamp('2019-10-06 10:11:12.1', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.1', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12.1
+NULL
-- !query
@@ -491,7 +491,7 @@ select to_timestamp('2019-10-06 10:11:12.12', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.12', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12.12
+NULL
-- !query
@@ -499,7 +499,7 @@ select to_timestamp('2019-10-06 10:11:12.123UTC',
'yyyy-MM-dd HH:mm:ss.SSSSSS[zz
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.123UTC', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 03:11:12.123
+NULL
-- !query
@@ -507,7 +507,7 @@ select to_timestamp('2019-10-06 10:11:12.1234', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.1234', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12.1234
+NULL
-- !query
@@ -515,7 +515,7 @@ select to_timestamp('2019-10-06 10:11:12.12345CST',
'yyyy-MM-dd HH:mm:ss.SSSSSS[
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.12345CST', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 08:11:12.12345
+NULL
-- !query
@@ -523,7 +523,7 @@ select to_timestamp('2019-10-06 10:11:12.123456PST',
'yyyy-MM-dd HH:mm:ss.SSSSSS
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.123456PST', 'yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12.123456
+NULL
-- !query
@@ -539,7 +539,7 @@ select to_timestamp('123456 2019-10-06 10:11:12.123456PST',
'SSSSSS yyyy-MM-dd H
-- !query schema
struct<to_timestamp('123456 2019-10-06 10:11:12.123456PST', 'SSSSSS yyyy-MM-dd
HH:mm:ss.SSSSSS[zzz]'):timestamp>
-- !query output
-2019-10-06 10:11:12.123456
+NULL
-- !query
@@ -555,7 +555,7 @@ select to_timestamp('2019-10-06 10:11:12.1234', 'yyyy-MM-dd
HH:mm:ss.[SSSSSS]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.1234', 'yyyy-MM-dd
HH:mm:ss.[SSSSSS]'):timestamp>
-- !query output
-2019-10-06 10:11:12.1234
+NULL
-- !query
@@ -563,7 +563,7 @@ select to_timestamp('2019-10-06 10:11:12.123', 'yyyy-MM-dd
HH:mm:ss[.SSSSSS]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.123', 'yyyy-MM-dd
HH:mm:ss[.SSSSSS]'):timestamp>
-- !query output
-2019-10-06 10:11:12.123
+NULL
-- !query
@@ -571,7 +571,7 @@ select to_timestamp('2019-10-06 10:11:12', 'yyyy-MM-dd
HH:mm:ss[.SSSSSS]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12', 'yyyy-MM-dd
HH:mm:ss[.SSSSSS]'):timestamp>
-- !query output
-2019-10-06 10:11:12
+NULL
-- !query
@@ -579,7 +579,7 @@ select to_timestamp('2019-10-06 10:11:12.12', 'yyyy-MM-dd
HH:mm[:ss.SSSSSS]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11:12.12', 'yyyy-MM-dd
HH:mm[:ss.SSSSSS]'):timestamp>
-- !query output
-2019-10-06 10:11:12.12
+NULL
-- !query
@@ -587,7 +587,7 @@ select to_timestamp('2019-10-06 10:11', 'yyyy-MM-dd
HH:mm[:ss.SSSSSS]')
-- !query schema
struct<to_timestamp('2019-10-06 10:11', 'yyyy-MM-dd
HH:mm[:ss.SSSSSS]'):timestamp>
-- !query output
-2019-10-06 10:11:00
+NULL
-- !query
@@ -595,7 +595,7 @@ select to_timestamp("2019-10-06S10:11:12.12345",
"yyyy-MM-dd'S'HH:mm:ss.SSSSSS")
-- !query schema
struct<to_timestamp('2019-10-06S10:11:12.12345',
'yyyy-MM-dd\'S\'HH:mm:ss.SSSSSS'):timestamp>
-- !query output
-2019-10-06 10:11:12.12345
+NULL
-- !query
@@ -603,7 +603,7 @@ select to_timestamp("12.12342019-10-06S10:11",
"ss.SSSSyyyy-MM-dd'S'HH:mm")
-- !query schema
struct<to_timestamp('12.12342019-10-06S10:11',
'ss.SSSSyyyy-MM-dd\'S\'HH:mm'):timestamp>
-- !query output
-2019-10-06 10:11:12.1234
+NULL
-- !query
@@ -627,7 +627,7 @@ select to_timestamp("12.1234019-10-06S10:11",
"ss.SSSSy-MM-dd'S'HH:mm")
-- !query schema
struct<to_timestamp('12.1234019-10-06S10:11',
'ss.SSSSy-MM-dd\'S\'HH:mm'):timestamp>
-- !query output
-0019-10-06 10:11:12.1234
+NULL
-- !query
@@ -652,7 +652,7 @@ select date_format(timestamp '2019-10-06', 'yyyy-MM-dd
uuee')
struct<>
-- !query output
java.lang.IllegalArgumentException
-Illegal pattern character: e
+Illegal pattern character 'e'
-- !query
@@ -661,7 +661,7 @@ select date_format(timestamp '2019-10-06', 'yyyy-MM-dd
uucc')
struct<>
-- !query output
java.lang.IllegalArgumentException
-Illegal pattern character: c
+Illegal pattern character 'c'
-- !query
@@ -669,7 +669,7 @@ select date_format(timestamp '2019-10-06', 'yyyy-MM-dd
uuuu')
-- !query schema
struct<date_format(TIMESTAMP '2019-10-06 00:00:00', yyyy-MM-dd uuuu):string>
-- !query output
-2019-10-06 Sunday
+2019-10-06 0007
-- !query
@@ -677,7 +677,7 @@ select to_timestamp("2019-10-06T10:11:12'12",
"yyyy-MM-dd'T'HH:mm:ss''SSSS")
-- !query schema
struct<to_timestamp('2019-10-06T10:11:12\'12',
'yyyy-MM-dd\'T\'HH:mm:ss\'\'SSSS'):timestamp>
-- !query output
-2019-10-06 10:11:12.12
+2019-10-06 10:11:12.012
-- !query
@@ -750,3 +750,149 @@ select to_timestamp("2019 10:10:10", "yyyy hh:mm:ss")
struct<to_timestamp('2019 10:10:10', 'yyyy hh:mm:ss'):timestamp>
-- !query output
2019-01-01 10:10:10
+
+
+-- !query
+select date_format(date '2020-05-23', 'GGGGG')
+-- !query schema
+struct<date_format(CAST(DATE '2020-05-23' AS TIMESTAMP), GGGGG):string>
+-- !query output
+AD
+
+
+-- !query
+select date_format(date '2020-05-23', 'MMMMM')
+-- !query schema
+struct<date_format(CAST(DATE '2020-05-23' AS TIMESTAMP), MMMMM):string>
+-- !query output
+May
+
+
+-- !query
+select date_format(date '2020-05-23', 'LLLLL')
+-- !query schema
+struct<date_format(CAST(DATE '2020-05-23' AS TIMESTAMP), LLLLL):string>
+-- !query output
+May
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'EEEEE')
+-- !query schema
+struct<date_format(TIMESTAMP '2020-05-23 00:00:00', EEEEE):string>
+-- !query output
+Saturday
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'uuuuu')
+-- !query schema
+struct<date_format(TIMESTAMP '2020-05-23 00:00:00', uuuuu):string>
+-- !query output
+00006
+
+
+-- !query
+select date_format('2020-05-23', 'QQQQQ')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Illegal pattern character 'Q'
+
+
+-- !query
+select date_format('2020-05-23', 'qqqqq')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Illegal pattern character 'q'
+
+
+-- !query
+select to_timestamp('2019-10-06 A', 'yyyy-MM-dd GGGGG')
+-- !query schema
+struct<to_timestamp('2019-10-06 A', 'yyyy-MM-dd GGGGG'):timestamp>
+-- !query output
+NULL
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEEE')
+-- !query schema
+struct<to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEEE'):timestamp>
+-- !query output
+2020-05-22 00:00:00
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE'):timestamp>
+-- !query output
+2020-05-22 00:00:00
+
+
+-- !query
+select unix_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<unix_timestamp(22 05 2020 Friday, dd MM yyyy EEEEE):bigint>
+-- !query output
+1590130800
+
+
+-- !query
+select from_unixtime(12345, 'MMMMM')
+-- !query schema
+struct<from_unixtime(CAST(12345 AS BIGINT), MMMMM):string>
+-- !query output
+December
+
+
+-- !query
+select from_unixtime(54321, 'QQQQQ')
+-- !query schema
+struct<from_unixtime(CAST(54321 AS BIGINT), QQQQQ):string>
+-- !query output
+NULL
+
+
+-- !query
+select from_unixtime(23456, 'aaaaa')
+-- !query schema
+struct<from_unixtime(CAST(23456 AS BIGINT), aaaaa):string>
+-- !query output
+PM
+
+
+-- !query
+select from_json('{"time":"26/October/2015"}', 'time Timestamp',
map('timestampFormat', 'dd/MMMMM/yyyy'))
+-- !query schema
+struct<from_json({"time":"26/October/2015"}):struct<time:timestamp>>
+-- !query output
+{"time":2015-10-26 00:00:00}
+
+
+-- !query
+select from_json('{"date":"26/October/2015"}', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<from_json({"date":"26/October/2015"}):struct<date:date>>
+-- !query output
+{"date":2015-10-26}
+
+
+-- !query
+select from_csv('26/October/2015', 'time Timestamp', map('timestampFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<from_csv(26/October/2015):struct<time:timestamp>>
+-- !query output
+{"time":2015-10-26 00:00:00}
+
+
+-- !query
+select from_csv('26/October/2015', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<from_csv(26/October/2015):struct<date:date>>
+-- !query output
+{"date":2015-10-26}
diff --git a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
b/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
index b26eb2f..dc466d1 100755
--- a/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/datetime.sql.out
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
--- Number of queries: 91
+-- Number of queries: 109
-- !query
@@ -750,3 +750,164 @@ select to_timestamp("2019 10:10:10", "yyyy hh:mm:ss")
struct<to_timestamp('2019 10:10:10', 'yyyy hh:mm:ss'):timestamp>
-- !query output
2019-01-01 10:10:10
+
+
+-- !query
+select date_format(date '2020-05-23', 'GGGGG')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'GGGGG' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(date '2020-05-23', 'MMMMM')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'MMMMM' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(date '2020-05-23', 'LLLLL')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'LLLLL' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format(timestamp '2020-05-23', 'uuuuu')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'uuuuu' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select date_format('2020-05-23', 'QQQQQ')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Too many pattern letters: Q
+
+
+-- !query
+select date_format('2020-05-23', 'qqqqq')
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Too many pattern letters: q
+
+
+-- !query
+select to_timestamp('2019-10-06 A', 'yyyy-MM-dd GGGGG')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'yyyy-MM-dd GGGGG' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select to_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select unix_timestamp('22 05 2020 Friday', 'dd MM yyyy EEEEE')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd MM yyyy EEEEE' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_unixtime(12345, 'MMMMM')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'MMMMM' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_unixtime(54321, 'QQQQQ')
+-- !query schema
+struct<from_unixtime(CAST(54321 AS BIGINT), QQQQQ):string>
+-- !query output
+NULL
+
+
+-- !query
+select from_unixtime(23456, 'aaaaa')
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'aaaaa' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_json('{"time":"26/October/2015"}', 'time Timestamp',
map('timestampFormat', 'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_json('{"date":"26/October/2015"}', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_csv('26/October/2015', 'time Timestamp', map('timestampFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
+
+
+-- !query
+select from_csv('26/October/2015', 'date Date', map('dateFormat',
'dd/MMMMM/yyyy'))
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'dd/MMMMM/yyyy' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
diff --git
a/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
b/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
index 5c56eff..b4c0525 100644
---
a/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/typeCoercion/native/stringCastAndExpressions.sql.out
@@ -136,9 +136,10 @@ NULL
-- !query
select to_timestamp('2018-01-01', a) from t
-- !query schema
-struct<to_timestamp('2018-01-01', t.`a`):timestamp>
+struct<>
-- !query output
-NULL
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'aa' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
-- !query
@@ -152,9 +153,10 @@ NULL
-- !query
select to_unix_timestamp('2018-01-01', a) from t
-- !query schema
-struct<to_unix_timestamp(2018-01-01, a):bigint>
+struct<>
-- !query output
-NULL
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'aa' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
-- !query
@@ -168,9 +170,10 @@ NULL
-- !query
select unix_timestamp('2018-01-01', a) from t
-- !query schema
-struct<unix_timestamp(2018-01-01, a):bigint>
+struct<>
-- !query output
-NULL
+org.apache.spark.SparkUpgradeException
+You may get a different result due to the upgrading of Spark 3.0: Fail to
recognize 'aa' pattern in the DateTimeFormatter. 1) You can set
spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before
Spark 3.0. 2) You can form a valid datetime pattern with the guide from
https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html
-- !query
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
index 5946536..247f815 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.time.{Instant, LocalDateTime}
import java.util.{Locale, TimeZone}
import java.util.concurrent.TimeUnit
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkException, SparkUpgradeException}
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{CEST, LA}
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.functions._
@@ -450,9 +450,9 @@ class DateFunctionsSuite extends QueryTest with
SharedSparkSession {
checkAnswer(
df.select(to_date(col("s"), "yyyy-hh-MM")),
Seq(Row(null), Row(null), Row(null)))
- checkAnswer(
- df.select(to_date(col("s"), "yyyy-dd-aa")),
- Seq(Row(null), Row(null), Row(null)))
+ val e = intercept[SparkUpgradeException](df.select(to_date(col("s"),
"yyyy-dd-aa")).collect())
+ assert(e.getCause.isInstanceOf[IllegalArgumentException])
+ assert(e.getMessage.contains("You may get a different result due to the
upgrading of Spark"))
// february
val x1 = "2016-02-29"
@@ -618,8 +618,16 @@ class DateFunctionsSuite extends QueryTest with
SharedSparkSession {
Row(secs(ts4.getTime)), Row(null), Row(secs(ts3.getTime)),
Row(null)))
// invalid format
- checkAnswer(df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd
aa:HH:ss')"), Seq(
- Row(null), Row(null), Row(null), Row(null)))
+ val invalid = df1.selectExpr(s"unix_timestamp(x, 'yyyy-MM-dd
aa:HH:ss')")
+ if (legacyParserPolicy == "legacy") {
+ checkAnswer(invalid,
+ Seq(Row(null), Row(null), Row(null), Row(null)))
+ } else {
+ val e = intercept[SparkUpgradeException](invalid.collect())
+ assert(e.getCause.isInstanceOf[IllegalArgumentException])
+ assert(
+ e.getMessage.contains("You may get a different result due to the
upgrading of Spark"))
+ }
// february
val y1 = "2016-02-29"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
