zabetak commented on code in PR #4777:
URL: https://github.com/apache/hive/pull/4777#discussion_r1354460818
##########
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.java:
##########
@@ -91,7 +95,22 @@ public void testEvaluate() throws HiveException,
InterruptedException {
SessionState.endStart(state);
}
+ @Test
+ public void testEvaluateUnixTimeStamp() throws HiveException,
InterruptedException {
+ HiveConf conf = new HiveConf();
+ conf.setVar(HiveConf.ConfVars.HIVE_DATETIME_FORMATTER, formatter);
+ conf.setVar(HiveConf.ConfVars.HIVE_LOCAL_TIME_ZONE, zone);
+ conf.setVar(HiveConf.ConfVars.HIVE_DATETIME_RESOLVERSTYLE, resolverStyle);
+ SessionState state = SessionState.start(conf);
+ udfUnixTimeStamp.initialize(argInspectors);
+ LongWritable result = (LongWritable) udfUnixTimeStamp.evaluate(
+ new DeferredObject[] { new DeferredJavaObject(new Text(value)), new
DeferredJavaObject(new Text(pattern)) });
+ assertEquals(udfDisplayWithInputs(), expectedResult, result);
+ SessionState.endStart(state);
+ }
+
Review Comment:
This duplicates the amount of tests that are run without adding much in
terms of test coverage. The evaluate is simply redirecting to
`GenericUDFToUnixTimeStamp`. It's not that long so we can keep it but if we do
let's at lest do some refactoring so that we don't duplicate the code from
above.
If we keep it, we may need to change the `udfDisplayWithInputs()` so that it
doesn't return the exact same thing for both tests.
##########
ql/src/test/resources/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.csv:
##########
@@ -1,54 +1,128 @@
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;0
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;0
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;SIMPLE;3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;-3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;SIMPLE;-3600
-1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;0
-1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;0
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;3600
-1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;-3600
-1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;-3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Europe/Paris;DATETIME;3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Europe/Paris;SIMPLE;3600
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;-5364662400
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;-5364662400
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;-5364683608
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;SIMPLE;-5364682200
-Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;null
-Jul 9 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1688860800
-Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;1688860800
-Jul 09 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1688860800
-Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;1689897600
-Jul 21 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1689897600
-2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;null
-2023-07-21;YYYY-MM-DD;Etc/GMT;SIMPLE;1672531200
-Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;1689930780
-Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;1689930780
-Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;null
-Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;1689930780
-2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;null
-2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;1689973980
-2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;SIMPLE;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;1689930780
-2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;null
-2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;null
-2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;1689973980
-2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;1689930790
-2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;1689930790
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;null
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;SIMPLE;1689930903
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.SSS;Etc/GMT;DATETIME;1689930790
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.SSS;Etc/GMT;DATETIME;1689930790
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;SIMPLE;SMART;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;SMART;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;STRICT;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;SIMPLE;SMART;-3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;SMART;-3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Europe/Paris;DATETIME;STRICT;-3600
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;0
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;SMART;0
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT;uuuu-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;STRICT;0
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT-01:00;uuuu-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;-3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;SMART;-3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT+01:00;uuuu-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;-3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;SIMPLE;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT-01:00;uuuu-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;STRICT;3600
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+1800-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;SIMPLE;SMART;-5364682200
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;SMART;-5364683608
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;STRICT;null
+1800-01-01 00:00:00;uuuu-MM-dd
HH:mm:ss;Asia/Kolkata;DATETIME;STRICT;-5364683608
+Jul 9 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1688860800
+Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;null
+Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 9 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;null
+Jul 09 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1688860800
+Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;1688860800
+Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 09 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;1688860800
+Jul 21 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1689897600
+Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;1689897600
+Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;1689897600
+2023-07-21;YYYY-MM-DD;Etc/GMT;SIMPLE;SMART;1672531200
+2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;SMART;null
+2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;STRICT;null
+2023-07-21;UUUU-MM-DD;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;SMART;1689930780
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 09:13;MMM dd uuuu HH:mm;Etc/GMT;DATETIME;STRICT;1689930780
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;SMART;null
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 9:13;MMM dd uuuu HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;SMART;1689930780
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13;uuuu-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;1689930780
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;SMART;null
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13;uuuu-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;SMART;1689973980
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13PM;uuuu-MM-dd h:mma;Etc/GMT;DATETIME;STRICT;1689973980
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;SIMPLE;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mmAA;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;1689930780
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd hh:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd hh:mma;Etc/GMT;DATETIME;STRICT;1689973980
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;1689930790
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;1689930790
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13:10;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;1689930790
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;SIMPLE;SMART;1689930903
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13:10.123;uuuu-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;STRICT;null
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;983318400
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;983318400
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2001-02-28 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;983318400
+2001-02-29;yyyy-MM-dd;Etc/GMT;SIMPLE;SMART;983404800
+2001-02-29;yyyy-MM-dd;Etc/GMT;DATETIME;SMART;983318400
+2001-02-29;yyyy-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-02-29;uuuu-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-02-31;yyyy-MM-dd;Etc/GMT;SIMPLE;SMART;983577600
+2001-02-31;yyyy-MM-dd;Etc/GMT;DATETIME;SMART;983318400
+2001-02-31;yyyy-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-02-31;uuuu-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-04-31;yyyy-MM-dd;Etc/GMT;SIMPLE;SMART;988675200
+2001-04-31;yyyy-MM-dd;Etc/GMT;DATETIME;SMART;988588800
+2001-04-31;yyyy-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-04-31;uuuu-MM-dd;Etc/GMT;DATETIME;STRICT;null
+2001-06-31 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;993945600
+2001-06-31 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;993859200
+2001-06-31 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2001-06-31 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2001-09-31 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;1001894400
+2001-09-31 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;1001808000
+2001-09-31 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2001-09-31 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
Review Comment:
nit: Missing new line at the end of file.
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/InstantDateTimeFormatter.java:
##########
@@ -25,15 +25,20 @@
import java.time.LocalDate;
import java.time.ZoneId;
import java.time.ZonedDateTime;
+import java.time.chrono.IsoEra;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
+import java.time.format.ResolverStyle;
+import java.time.temporal.ChronoField;
import java.util.Objects;
final class InstantDateTimeFormatter extends
InstantFormatterCache<DateTimeFormatter> {
InstantDateTimeFormatter(final ZoneId zoneId) {
super(zoneId,
- s -> new
DateTimeFormatterBuilder().parseCaseInsensitive().appendPattern(s).toFormatter().withZone(zoneId));
+ s -> new DateTimeFormatterBuilder().parseCaseInsensitive()
+ .parseDefaulting(ChronoField.ERA, IsoEra.CE.getValue())
Review Comment:
I see your point and I guess it makes sense but if we don't do this
consistently in other places as well we may bump into some weird behavior.
##########
ql/src/test/resources/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.csv:
##########
@@ -1,54 +1,128 @@
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;0
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;0
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;SIMPLE;3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;-3600
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;SIMPLE;-3600
-1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;0
-1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;0
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;3600
-1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;-3600
-1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;-3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Europe/Paris;DATETIME;3600
-1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Europe/Paris;SIMPLE;3600
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;-5364662400
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;-5364662400
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;-5364683608
-1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;SIMPLE;-5364682200
-Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;null
-Jul 9 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1688860800
-Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;1688860800
-Jul 09 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1688860800
-Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;1689897600
-Jul 21 2023;MMM dd yyyy;Etc/GMT;SIMPLE;1689897600
-2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;null
-2023-07-21;YYYY-MM-DD;Etc/GMT;SIMPLE;1672531200
-Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;1689930780
-Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;1689930780
-Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;null
-Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;1689930780
-2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;null
-2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;1689930780
-2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;1689973980
-2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;SIMPLE;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;null
-2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;1689930780
-2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;null
-2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;1689930780
-2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;null
-2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;1689973980
-2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;SIMPLE;1689973980
-2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;1689930790
-2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;1689930790
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;null
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;SIMPLE;1689930903
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.SSS;Etc/GMT;DATETIME;1689930790
-2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.SSS;Etc/GMT;DATETIME;1689930790
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;0
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;SIMPLE;SMART;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;SMART;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Atlantic/Azores;DATETIME;STRICT;3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;SIMPLE;SMART;-3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;SMART;-3600
+1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Europe/Paris;DATETIME;STRICT;null
+1970-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Europe/Paris;DATETIME;STRICT;-3600
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;0
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;SMART;0
+1970-01-01 00:00:00 GMT;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT;uuuu-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;STRICT;0
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;DATETIME;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT-01:00;uuuu-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss z;Etc/GMT;SIMPLE;SMART;-3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;SMART;-3600
+1970-01-01 00:00:00 GMT+01:00;yyyy-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT+01:00;uuuu-MM-dd HH:mm:ss
z;Etc/GMT;DATETIME;STRICT;-3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;SIMPLE;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;SMART;3600
+1970-01-01 00:00:00 GMT-01:00;yyyy-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;STRICT;null
+1970-01-01 00:00:00 GMT-01:00;uuuu-MM-dd HH:mm:ss
z;Europe/Paris;DATETIME;STRICT;3600
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+1800-01-01 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;-5364662400
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;SIMPLE;SMART;-5364682200
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;SMART;-5364683608
+1800-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Asia/Kolkata;DATETIME;STRICT;null
+1800-01-01 00:00:00;uuuu-MM-dd
HH:mm:ss;Asia/Kolkata;DATETIME;STRICT;-5364683608
+Jul 9 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1688860800
+Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;null
+Jul 9 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 9 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;null
+Jul 09 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1688860800
+Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;1688860800
+Jul 09 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 09 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;1688860800
+Jul 21 2023;MMM dd yyyy;Etc/GMT;SIMPLE;SMART;1689897600
+Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;SMART;1689897600
+Jul 21 2023;MMM dd yyyy;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023;MMM dd uuuu;Etc/GMT;DATETIME;STRICT;1689897600
+2023-07-21;YYYY-MM-DD;Etc/GMT;SIMPLE;SMART;1672531200
+2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;SMART;null
+2023-07-21;YYYY-MM-DD;Etc/GMT;DATETIME;STRICT;null
+2023-07-21;UUUU-MM-DD;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;SMART;1689930780
+Jul 21 2023 09:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 09:13;MMM dd uuuu HH:mm;Etc/GMT;DATETIME;STRICT;1689930780
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;SMART;null
+Jul 21 2023 9:13;MMM dd yyyy HH:mm;Etc/GMT;DATETIME;STRICT;null
+Jul 21 2023 9:13;MMM dd uuuu HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;SMART;1689930780
+2023-07-21 09:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13;uuuu-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;1689930780
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;SMART;null
+2023-07-21 9:13;yyyy-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13;uuuu-MM-dd HH:mm;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;SMART;1689973980
+2023-07-21 9:13PM;yyyy-MM-dd h:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 9:13PM;uuuu-MM-dd h:mma;Etc/GMT;DATETIME;STRICT;1689973980
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;SIMPLE;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmAA;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mmAA;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;SMART;1689930780
+2023-07-21 09:13AM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13AM;uuuu-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;1689930780
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;SIMPLE;SMART;1689930780
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13PM;yyyy-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd HH:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd hh:mmaa;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;SIMPLE;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;SMART;1689973980
+2023-07-21 09:13PM;yyyy-MM-dd hh:mma;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13PM;uuuu-MM-dd hh:mma;Etc/GMT;DATETIME;STRICT;1689973980
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;1689930790
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;1689930790
+2023-07-21 09:13:10;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13:10;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;1689930790
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;SIMPLE;SMART;1689930903
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;SMART;null
+2023-07-21 09:13:10.123;yyyy-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;STRICT;null
+2023-07-21 09:13:10.123;uuuu-MM-dd HH:mm:ss.sss;Etc/GMT;DATETIME;STRICT;null
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;SIMPLE;SMART;983318400
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;SMART;983318400
+2001-02-28 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;null
+2001-02-28 00:00:00;uuuu-MM-dd HH:mm:ss;Etc/GMT;DATETIME;STRICT;983318400
+2001-02-29;yyyy-MM-dd;Etc/GMT;SIMPLE;SMART;983404800
+2001-02-29;yyyy-MM-dd;Etc/GMT;DATETIME;SMART;983318400
Review Comment:
I am curious if the `LENIENT` value allows the SIMPLE and DATETIME parser to
output the same result for invalid dates. If that's the case then it may be a
good idea to keep and test that option as well.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3860,6 +3860,17 @@ public static enum ConfVars {
"is discouraged. It suffers from known bugs that are unlikely to be
fixed in subsequent versions of the product." +
"Furthermore, using SIMPLE formatter may lead to strange behavior, and
unexpected results when combined " +
"with SQL functions/operators that are using the new DATETIME
formatter."),
+
+ HIVE_DATETIME_RESOLVERSTYLE("hive.datetime.resolver.style", "SMART", new
StringSet("SMART", "STRICT", "LENIENT"),
Review Comment:
Please add some property specific tests in `TestHiveConfVarsValidate` class.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3860,6 +3860,17 @@ public static enum ConfVars {
"is discouraged. It suffers from known bugs that are unlikely to be
fixed in subsequent versions of the product." +
"Furthermore, using SIMPLE formatter may lead to strange behavior, and
unexpected results when combined " +
"with SQL functions/operators that are using the new DATETIME
formatter."),
+
+ HIVE_DATETIME_RESOLVERSTYLE("hive.datetime.resolver.style", "SMART", new
StringSet("SMART", "STRICT", "LENIENT"),
Review Comment:
Since this is a property that affects the formatter maybe for namespacing
purposes we could rename it to:
`hive.datetime.formatter.resolver.style`
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3860,6 +3860,17 @@ public static enum ConfVars {
"is discouraged. It suffers from known bugs that are unlikely to be
fixed in subsequent versions of the product." +
"Furthermore, using SIMPLE formatter may lead to strange behavior, and
unexpected results when combined " +
"with SQL functions/operators that are using the new DATETIME
formatter."),
+
+ HIVE_DATETIME_RESOLVERSTYLE("hive.datetime.resolver.style", "SMART", new
StringSet("SMART", "STRICT", "LENIENT"),
Review Comment:
I think we should mention here that this property only takes effect when the
formatter is set to `DATETIME`. Alternatively, we could also do some kind of
mapping for SimpleDateFormat but I don't think we want to add new functionality
for a kind of deprecated config unless someone ask us to do it.
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3860,6 +3860,17 @@ public static enum ConfVars {
"is discouraged. It suffers from known bugs that are unlikely to be
fixed in subsequent versions of the product." +
"Furthermore, using SIMPLE formatter may lead to strange behavior, and
unexpected results when combined " +
"with SQL functions/operators that are using the new DATETIME
formatter."),
+
+ HIVE_DATETIME_RESOLVERSTYLE("hive.datetime.resolver.style", "SMART", new
StringSet("SMART", "STRICT", "LENIENT"),
+ "* SMART: Using smart resolution will perform the sensible default for
each field,For example," +
+ " Any value beyond the last valid day-of-month will be converted
to the last valid day-of-month." +
+ "* STRICT: Using strict resolution will ensure that all parsed
values are within the outer range of valid" +
+ " values for the field. For example, resolving year-month and
day-of-month will ensure that" +
+ " the day-of-month is valid for the year-month, rejecting invalid
values." +
+ "* LENIENT: Using lenient resolution will resolve the values in an
appropriate lenient manner. For " +
+ "example, lenient mode allows the month to be outside the range 1
to 12. " +
+ "For example, month 15 is treated as being 3 months after month
12."),
Review Comment:
As far as I see, LENIENT is not tested so in that case I feel that it can be
dropped. Let's not add extra functionality if we don't need to. Anything that
we add becomes something that we need to support from now onwards.
##########
ql/src/test/resources/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFToUnixTimestampEvaluateStringString.csv:
##########
@@ -1,54 +1,128 @@
-1970-01-01 00:00:00;yyyy-MM-dd HH:mm:ss;Etc/GMT;DATETIME;0
Review Comment:
In order to make the tests easier to read and facilitate the diff I would
suggest the following. Keep existing lines as they are and just add SMART at
the end demonstrating that this value retains backward compatibility.
Add the new tests for invalid dates/patterns at the end of the file.
Also consider removing repetitive test cases such as the combination `yyyy
with STRICT`. Showing once that this is invalid and returns null is sufficient.
Moreover, if SMART returns null then I guess by definition STRICT will return
null so these tests seem a bit redundant.
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/InstantFormatter.java:
##########
@@ -56,16 +57,16 @@ InstantFormatter newFormatter(ZoneId zone) {
*/
DATETIME {
@Override
- InstantFormatter newFormatter(ZoneId zone) {
- return new InstantDateTimeFormatter(zone);
+ InstantFormatter newFormatter(ZoneId zone, ResolverStyle resolverStyle) {
+ return new InstantDateTimeFormatter(zone, resolverStyle);
}
};
/**
* Creates a new formatter with the specified zone id.
* @param zone - the zone id
* @return a new formatter with the specified zone id.
*/
- abstract InstantFormatter newFormatter(ZoneId zone);
+ abstract InstantFormatter newFormatter(ZoneId zone, ResolverStyle
resolverStyle);
Review Comment:
nit: Please add a small entry to the Javadoc just above.
##########
ql/src/test/queries/clientpositive/test_unixtimestamp_for_invaild_dates.q:
##########
@@ -0,0 +1,20 @@
+DESCRIBE FUNCTION UNIX_TIMESTAMP;
Review Comment:
The `TestGenericUDFToUnixTimestampEvaluateStringStringCsv` only tests the
UDF using two inputs (date + pattern). I was wondering if we have or if we
should add tests (with invalid dates) where the UDF has only one input (date
and no pattern).
I would like to see if there are inconsistencies among the following
```
unix_timestamp('2001-02-31')
unix_timestamp('2001-02-31', 'uuuu-MM-dd')
```
after the changes in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
