[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718: URL: https://github.com/apache/spark/pull/28718#discussion_r434718052 ## File path: sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql ## @@ -0,0 +1,43 @@ +--- TESTS FOR DATETIME FORMATTING FUNCTIONS WITH INVALID PATTERNS --- + +-- separating this from datetime-formatting.sql ,because the text form +-- for patterns with 5 letters in SimpleDateFormat varies from different JDKs +create temporary view v as select col from values + (timestamp '1582-06-01 11:33:33.123UTC+08'), + (timestamp '1970-01-01 00:00:00.000Europe/Paris'), + (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'), + (timestamp '1996-04-01 00:33:33.123Australia/Darwin'), + (timestamp '2018-11-17 13:33:33.123Z'), + (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'), + (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') t(col); + +select col, date_format(col, 'G') from v; +select col, date_format(col, 'yyy') from v; -- pattern letter count can not be greater than 10 Review comment: can we put the comment in a new line? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718: URL: https://github.com/apache/spark/pull/28718#discussion_r434717715 ## File path: sql/core/src/test/resources/sql-tests/inputs/datetime-formatting-invalid.sql ## @@ -0,0 +1,43 @@ +--- TESTS FOR DATETIME FORMATTING FUNCTIONS WITH INVALID PATTERNS --- + +-- separating this from datetime-formatting.sql ,because the text form +-- for patterns with 5 letters in SimpleDateFormat varies from different JDKs +create temporary view v as select col from values + (timestamp '1582-06-01 11:33:33.123UTC+08'), + (timestamp '1970-01-01 00:00:00.000Europe/Paris'), + (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'), + (timestamp '1996-04-01 00:33:33.123Australia/Darwin'), + (timestamp '2018-11-17 13:33:33.123Z'), + (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'), + (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') t(col); + +select col, date_format(col, 'G') from v; Review comment: for invalid testing, we can just use literals? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718: URL: https://github.com/apache/spark/pull/28718#discussion_r434593292 ## File path: sql/core/src/test/resources/sql-tests/inputs/datetime-formatting.sql ## @@ -0,0 +1,19 @@ +--- TESTS FOR DATETIME FORMATTING FUNCTIONS --- + +-- valid formatter pattern check +create temporary view ttt as select t from VALUES Review comment: and please lower case `VALUES`, or upper case all SQL keywords. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718: URL: https://github.com/apache/spark/pull/28718#discussion_r434592867 ## File path: sql/core/src/test/resources/sql-tests/inputs/datetime-formatting.sql ## @@ -0,0 +1,19 @@ +--- TESTS FOR DATETIME FORMATTING FUNCTIONS --- + +-- valid formatter pattern check +create temporary view ttt as select t from VALUES Review comment: nit: ``` create temporary view v as select col from ... t(col) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434590780
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
##
@@ -42,7 +46,7 @@ class DateTimeFormatterHelperSuite extends SparkFunSuite {
}
unsupportedLettersForParsing.foreach { l =>
val e = intercept[IllegalArgumentException] {
-convertIncompatiblePattern(s"$l", isParsing = true)
+DateTimeFormatterHelper.convertIncompatiblePattern(s"$l", isParsing =
true)
Review comment:
is this change needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434558722
##
File path: sql/core/src/test/resources/sql-tests/results/datetime.sql.out
##
@@ -1013,3 +1013,110 @@ select date_format('2020-01-01', '-MM-dd ')
struct
-- !query output
2020-01-01 Wednesday
+
+
+-- !query
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+select date_format(t, 'Y YY Y Y y yy yyy y') from ttt
+-- !query schema
+struct
+-- !query output
+1582 82 1582 01582 01582 1582 82 1582 1582 01582
+1970 70 1970 01970 01970 1969 69 1969 1969 01969
+1970 70 1970 01970 01970 1970 70 1970 1970 01970
+1996 96 1996 01996 01996 1996 96 1996 1996 01996
+2018 18 2018 02018 02018 2018 18 2018 2018 02018
+2020 20 2020 02020 02020 2019 19 2019 2019 02019
+2099 99 2099 02099 02099 2100 00 2100 2100 02100
+
+
+-- !query
+select date_format(t, 'q qq Q QQ QQQ ') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 Q1 1st quarter
+1 01 1 01 Q1 1st quarter
+2 02 2 02 Q2 2nd quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+
+
+-- !query
+select date_format(t, 'M MM MMM L LL') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 Jan January 1 01
+11 11 Nov November 11 11
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+3 03 Mar March 3 03
+5 05 May May 5 05
+
+
+-- !query
+select date_format(t, 'W ww d dd DDD u uu uuu F E EE EEE a') from ttt
+-- !query schema
+struct
+-- !query output
+0 53 1 01 001 5 05 Fri Friday 1 Fri Fri Fri Friday AM
+3 46 17 17 321 6 06 Sat Saturday 3 Sat Sat Sat Saturday AM
+4 13 31 31 091 7 07 Sun Sunday 3 Sun Sun Sun Sunday AM
+5 01 31 31 365 2 02 Tue Tuesday 3 Tue Tue Tue Tuesday AM
+5 01 31 31 365 3 03 Wed Wednesday 3 Wed Wed Wed Wednesday PM
+5 22 31 31 151 1 01 Mon Monday 3 Mon Mon Mon Monday PM
+5 53 31 31 365 4 04 Thu Thursday 3 Thu Thu Thu Thursday AM
+
+
+-- !query
+select date_format(t, 'h hh H HH k kk K KK m mm s ss SSS') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 1 01 1 01 33 33 33 33 123
+3 03 15 15 15 15 3 03 0 00 0 00 000
+4 04 4 04 4 04 4 04 59 59 59 59 999
+5 05 5 05 5 05 5 05 33 33 33 33 123
+7 07 19 19 19 19 7 07 40 40 35 35 123
+7 07 7 07 7 07 7 07 3 03 33 33 123
+9 09 9 09 9 09 9 09 33 33 33 33 123
+
+
+-- !query
+select date_format(t, 'VV z zz zzz O X XX XXX X x xx xxx
x Z ZZ ZZZ Z') from ttt
+-- !query schema
+struct
+-- !query output
+America/Los_Angeles PST PST PST Pacific Standard Time GMT-7:52:58 GMT-07:52:58
-0752 -0752 -07:52 -075258 -07:52:58 -0752 -0752 -07:52 -075258 -075258
-07:52:58 -0752 -0752 -0752 GMT-07:52:58 -07:52:58
Review comment:
actually, this is expected. The Spark test uses timezone
`America/Los_Angeles`, which is indeed GMT-7:52:58 long time before, see
https://www.politico.com/story/2017/11/18/railroads-initiative-synchronizes-us-clocks-nov-18-1883-244935
Spark's timestamp value doesn't have an associated timezone, so extracting
timezone means showing the session local timezone.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434548439
##
File path: sql/core/src/test/resources/sql-tests/results/datetime.sql.out
##
@@ -1013,3 +1013,110 @@ select date_format('2020-01-01', '-MM-dd ')
struct
-- !query output
2020-01-01 Wednesday
+
+
+-- !query
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+select date_format(t, 'Y YY Y Y y yy yyy y') from ttt
+-- !query schema
+struct
+-- !query output
+1582 82 1582 01582 01582 1582 82 1582 1582 01582
+1970 70 1970 01970 01970 1969 69 1969 1969 01969
+1970 70 1970 01970 01970 1970 70 1970 1970 01970
+1996 96 1996 01996 01996 1996 96 1996 1996 01996
+2018 18 2018 02018 02018 2018 18 2018 2018 02018
+2020 20 2020 02020 02020 2019 19 2019 2019 02019
+2099 99 2099 02099 02099 2100 00 2100 2100 02100
+
+
+-- !query
+select date_format(t, 'q qq Q QQ QQQ ') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 Q1 1st quarter
+1 01 1 01 Q1 1st quarter
+2 02 2 02 Q2 2nd quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+
+
+-- !query
+select date_format(t, 'M MM MMM L LL') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 Jan January 1 01
+11 11 Nov November 11 11
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+3 03 Mar March 3 03
+5 05 May May 5 05
+
+
+-- !query
+select date_format(t, 'W ww d dd DDD u uu uuu F E EE EEE a') from ttt
+-- !query schema
+struct
+-- !query output
+0 53 1 01 001 5 05 Fri Friday 1 Fri Fri Fri Friday AM
+3 46 17 17 321 6 06 Sat Saturday 3 Sat Sat Sat Saturday AM
+4 13 31 31 091 7 07 Sun Sunday 3 Sun Sun Sun Sunday AM
+5 01 31 31 365 2 02 Tue Tuesday 3 Tue Tue Tue Tuesday AM
+5 01 31 31 365 3 03 Wed Wednesday 3 Wed Wed Wed Wednesday PM
+5 22 31 31 151 1 01 Mon Monday 3 Mon Mon Mon Monday PM
+5 53 31 31 365 4 04 Thu Thursday 3 Thu Thu Thu Thursday AM
+
+
+-- !query
+select date_format(t, 'h hh H HH k kk K KK m mm s ss SSS') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 1 01 1 01 33 33 33 33 123
+3 03 15 15 15 15 3 03 0 00 0 00 000
+4 04 4 04 4 04 4 04 59 59 59 59 999
+5 05 5 05 5 05 5 05 33 33 33 33 123
+7 07 19 19 19 19 7 07 40 40 35 35 123
+7 07 7 07 7 07 7 07 3 03 33 33 123
+9 09 9 09 9 09 9 09 33 33 33 33 123
+
+
+-- !query
+select date_format(t, 'VV z zz zzz O X XX XXX X x xx xxx
x Z ZZ ZZZ Z') from ttt
+-- !query schema
+struct
+-- !query output
+America/Los_Angeles PST PST PST Pacific Standard Time GMT-7:52:58 GMT-07:52:58
-0752 -0752 -07:52 -075258 -07:52:58 -0752 -0752 -07:52 -075258 -075258
-07:52:58 -0752 -0752 -0752 GMT-07:52:58 -07:52:58
Review comment:
I'm not sure, but it seems like we should ban all the timezone letters
during formatting. Spark's timestamp value doesn't have an associated timezone,
it makes no sense to extract timezone from it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434548545
##
File path: sql/core/src/test/resources/sql-tests/results/datetime.sql.out
##
@@ -1013,3 +1013,110 @@ select date_format('2020-01-01', '-MM-dd ')
struct
-- !query output
2020-01-01 Wednesday
+
+
+-- !query
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+select date_format(t, 'Y YY Y Y y yy yyy y') from ttt
+-- !query schema
+struct
+-- !query output
+1582 82 1582 01582 01582 1582 82 1582 1582 01582
+1970 70 1970 01970 01970 1969 69 1969 1969 01969
+1970 70 1970 01970 01970 1970 70 1970 1970 01970
+1996 96 1996 01996 01996 1996 96 1996 1996 01996
+2018 18 2018 02018 02018 2018 18 2018 2018 02018
+2020 20 2020 02020 02020 2019 19 2019 2019 02019
+2099 99 2099 02099 02099 2100 00 2100 2100 02100
+
+
+-- !query
+select date_format(t, 'q qq Q QQ QQQ ') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 Q1 1st quarter
+1 01 1 01 Q1 1st quarter
+2 02 2 02 Q2 2nd quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+
+
+-- !query
+select date_format(t, 'M MM MMM L LL') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 Jan January 1 01
+11 11 Nov November 11 11
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+3 03 Mar March 3 03
+5 05 May May 5 05
+
+
+-- !query
+select date_format(t, 'W ww d dd DDD u uu uuu F E EE EEE a') from ttt
+-- !query schema
+struct
+-- !query output
+0 53 1 01 001 5 05 Fri Friday 1 Fri Fri Fri Friday AM
+3 46 17 17 321 6 06 Sat Saturday 3 Sat Sat Sat Saturday AM
+4 13 31 31 091 7 07 Sun Sunday 3 Sun Sun Sun Sunday AM
+5 01 31 31 365 2 02 Tue Tuesday 3 Tue Tue Tue Tuesday AM
+5 01 31 31 365 3 03 Wed Wednesday 3 Wed Wed Wed Wednesday PM
+5 22 31 31 151 1 01 Mon Monday 3 Mon Mon Mon Monday PM
+5 53 31 31 365 4 04 Thu Thursday 3 Thu Thu Thu Thursday AM
+
+
+-- !query
+select date_format(t, 'h hh H HH k kk K KK m mm s ss SSS') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 1 01 1 01 33 33 33 33 123
+3 03 15 15 15 15 3 03 0 00 0 00 000
+4 04 4 04 4 04 4 04 59 59 59 59 999
+5 05 5 05 5 05 5 05 33 33 33 33 123
+7 07 19 19 19 19 7 07 40 40 35 35 123
+7 07 7 07 7 07 7 07 3 03 33 33 123
+9 09 9 09 9 09 9 09 33 33 33 33 123
+
+
+-- !query
+select date_format(t, 'VV z zz zzz O X XX XXX X x xx xxx
x Z ZZ ZZZ Z') from ttt
+-- !query schema
+struct
+-- !query output
+America/Los_Angeles PST PST PST Pacific Standard Time GMT-7:52:58 GMT-07:52:58
-0752 -0752 -07:52 -075258 -07:52:58 -0752 -0752 -07:52 -075258 -075258
-07:52:58 -0752 -0752 -0752 GMT-07:52:58 -07:52:58
Review comment:
cc @MaxGekk
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434548439
##
File path: sql/core/src/test/resources/sql-tests/results/datetime.sql.out
##
@@ -1013,3 +1013,110 @@ select date_format('2020-01-01', '-MM-dd ')
struct
-- !query output
2020-01-01 Wednesday
+
+
+-- !query
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+select date_format(t, 'Y YY Y Y y yy yyy y') from ttt
+-- !query schema
+struct
+-- !query output
+1582 82 1582 01582 01582 1582 82 1582 1582 01582
+1970 70 1970 01970 01970 1969 69 1969 1969 01969
+1970 70 1970 01970 01970 1970 70 1970 1970 01970
+1996 96 1996 01996 01996 1996 96 1996 1996 01996
+2018 18 2018 02018 02018 2018 18 2018 2018 02018
+2020 20 2020 02020 02020 2019 19 2019 2019 02019
+2099 99 2099 02099 02099 2100 00 2100 2100 02100
+
+
+-- !query
+select date_format(t, 'q qq Q QQ QQQ ') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 Q1 1st quarter
+1 01 1 01 Q1 1st quarter
+2 02 2 02 Q2 2nd quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+4 04 4 04 Q4 4th quarter
+
+
+-- !query
+select date_format(t, 'M MM MMM L LL') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 Jan January 1 01
+11 11 Nov November 11 11
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+12 12 Dec December 12 12
+3 03 Mar March 3 03
+5 05 May May 5 05
+
+
+-- !query
+select date_format(t, 'W ww d dd DDD u uu uuu F E EE EEE a') from ttt
+-- !query schema
+struct
+-- !query output
+0 53 1 01 001 5 05 Fri Friday 1 Fri Fri Fri Friday AM
+3 46 17 17 321 6 06 Sat Saturday 3 Sat Sat Sat Saturday AM
+4 13 31 31 091 7 07 Sun Sunday 3 Sun Sun Sun Sunday AM
+5 01 31 31 365 2 02 Tue Tuesday 3 Tue Tue Tue Tuesday AM
+5 01 31 31 365 3 03 Wed Wednesday 3 Wed Wed Wed Wednesday PM
+5 22 31 31 151 1 01 Mon Monday 3 Mon Mon Mon Monday PM
+5 53 31 31 365 4 04 Thu Thursday 3 Thu Thu Thu Thursday AM
+
+
+-- !query
+select date_format(t, 'h hh H HH k kk K KK m mm s ss SSS') from ttt
+-- !query schema
+struct
+-- !query output
+1 01 1 01 1 01 1 01 33 33 33 33 123
+3 03 15 15 15 15 3 03 0 00 0 00 000
+4 04 4 04 4 04 4 04 59 59 59 59 999
+5 05 5 05 5 05 5 05 33 33 33 33 123
+7 07 19 19 19 19 7 07 40 40 35 35 123
+7 07 7 07 7 07 7 07 3 03 33 33 123
+9 09 9 09 9 09 9 09 33 33 33 33 123
+
+
+-- !query
+select date_format(t, 'VV z zz zzz O X XX XXX X x xx xxx
x Z ZZ ZZZ Z') from ttt
+-- !query schema
+struct
+-- !query output
+America/Los_Angeles PST PST PST Pacific Standard Time GMT-7:52:58 GMT-07:52:58
-0752 -0752 -07:52 -075258 -07:52:58 -0752 -0752 -07:52 -075258 -075258
-07:52:58 -0752 -0752 -0752 GMT-07:52:58 -07:52:58
Review comment:
I'm not sure, but it seems like we should ban all the timezone letters
during formatting. Spark's timestamp value doesn't have an associated timezone,
it makes no sense to extract timezone from it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434547591
##
File path: sql/core/src/test/resources/sql-tests/inputs/datetime.sql
##
@@ -168,3 +168,18 @@ select date_format(date '2018-11-17', 'yyy-MM-dd');
-- SPARK-31879: the first day of week
select date_format('2020-01-01', '-MM-dd uu');
select date_format('2020-01-01', '-MM-dd ');
+
+-- valid formatter pattern check
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t);
+select date_format(t, 'Y-w-u -ww-uu YYY-W-uuu YY Y E EE EEE
') from ttt;
Review comment:
it's better to have a column to show the value of `t` as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434546839
##
File path: sql/core/src/test/resources/sql-tests/results/datetime-legacy.sql.out
##
@@ -996,3 +996,72 @@ select date_format('2020-01-01', '-MM-dd ')
struct
-- !query output
2020-01-01 0003
+
+
+-- !query
+create temporary view ttt as select t from VALUES
+ (timestamp '1582-06-01 11:33:33.123UTC+08'),
+ (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
+ (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
+ (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
+ (timestamp '2018-11-17 13:33:33.123Z'),
+ (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
+ (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') tt(t)
+-- !query schema
+struct<>
+-- !query output
+
+
+
+-- !query
+select date_format(t, 'Y-w-u -ww-uu YYY-W-uuu YY Y E EE EEE
') from ttt
+-- !query schema
+struct
+-- !query output
+1582-22-4 1582-22-04 1582-5-004 82 01582 0004 Thu Thu Thu Thursday
+1970-1-3 1970-01-03 1970-5-003 70 01970 0003 Wed Wed Wed Wednesday
+1970-53-4 1970-53-04 1970-5-004 70 01970 0004 Thu Thu Thu Thursday
+1996-13-7 1996-13-07 1996-4-007 96 01996 0007 Sun Sun Sun Sunday
+2018-46-6 2018-46-06 2018-3-006 18 02018 0006 Sat Sat Sat Saturday
+2020-1-2 2020-01-02 2020-5-002 20 02020 0002 Tue Tue Tue Tuesday
+2099-53-5 2099-53-05 2099-0-005 99 02099 0005 Fri Fri Fri Friday
+
+
+-- !query
+select date_format(t, 'q qq Q QQ QQQ ') from ttt
+-- !query schema
+struct<>
+-- !query output
+java.lang.IllegalArgumentException
+Illegal pattern character 'q'
+
+
+-- !query
+select date_format(t, 'y-M-d H:m:s -MM-dd HH:mm:ss.SSS yy yyy y MMM
L LL F h hh k kk K KK a') from ttt
+-- !query schema
+struct
+-- !query output
+1582-5-31 19:40:35 1582-05-31 19:40:35.123 82 1582 01582 May May 5 05 5 7 07
19 19 7 07 PM
+1969-12-31 15:0:0 1969-12-31 15:00:00.000 69 1969 01969 Dec December 12 12 5 3
03 15 15 3 03 PM
+1970-12-31 4:59:59 1970-12-31 04:59:59.999 70 1970 01970 Dec December 12 12 5
4 04 4 04 4 04 AM
+1996-3-31 7:3:33 1996-03-31 07:03:33.123 96 1996 01996 Mar March 3 03 5 7 07 7
07 7 07 AM
+2018-11-17 5:33:33 2018-11-17 05:33:33.123 18 2018 02018 Nov November 11 11 3
5 05 5 05 5 05 AM
+2019-12-31 9:33:33 2019-12-31 09:33:33.123 19 2019 02019 Dec December 12 12 5
9 09 9 09 9 09 AM
+2100-1-1 1:33:33 2100-01-01 01:33:33.123 00 2100 02100 Jan January 1 01 1 1 01
1 01 1 01 AM
+
+
+-- !query
+select date_format(t, 'VV z zz zzz O X XX XXX X x xx xxx
x Z ZZ ZZZ Z') from ttt
Review comment:
can we test `VV` separately? otherwise all other patterns listed here
can't be tested with the legacy formatter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28718: [SPARK-31892][SQL][FOLLOWUP] Improve test coverage for valid pattern …
cloud-fan commented on a change in pull request #28718:
URL: https://github.com/apache/spark/pull/28718#discussion_r434542000
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeFormatterHelperSuite.scala
##
@@ -23,43 +23,49 @@ import
org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper._
class DateTimeFormatterHelperSuite extends SparkFunSuite {
test("check incompatible pattern") {
-assert(convertIncompatiblePattern("MM-DD-u") === "MM-DD-e")
-assert(convertIncompatiblePattern("-MM-dd'T'HH:mm:ss.SSSz")
- === "-MM-dd'T'HH:mm:ss.SSSz")
-assert(convertIncompatiblePattern("-MM'y contains in quoted
text'HH:mm:ss")
- === "-MM'y contains in quoted text'HH:mm:ss")
-assert(convertIncompatiblePattern("-MM-dd-u'T'HH:mm:ss.SSSz")
+assert(convertIncompatiblePattern("MM-DD-u", isParsing = false) ===
"MM-DD-e")
Review comment:
In this test suite, how about we add
```
def convertIncompatiblePattern(pattern: String): String = {
convertIncompatiblePattern(pattern, isParsing = false)
}
```
To reduce the code diff?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
