LuciferYang commented on pull request #35825:
URL: https://github.com/apache/spark/pull/35825#issuecomment-1065880942
@dongjoon-hyun @gengliangwang @MaxGekk
Similar to the current pr, the following SQL will also have behavioral
differences between Java 8 and Java 11 / 17
```
"select to_timestamp('365', 'DD')"
```
Java 8 failed as:
```
You may get a different result due to the upgrading to Spark >= 3.0: Fail to
parse '365' in the new parser. You can set spark.sql.legacy.timeParserPolicy to
LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat
it as an invalid datetime string.
org.apache.spark.SparkUpgradeException: You may get a different result due
to the upgrading to Spark >= 3.0: Fail to parse '365' in the new parser. You
can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior
before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime
string.
at
org.apache.spark.sql.errors.QueryExecutionErrors$.failToParseDateTimeInNewParserError(QueryExecutionErrors.scala:1008)
at
org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:148)
at
org.apache.spark.sql.catalyst.util.DateTimeFormatterHelper$$anonfun$checkParsedDiff$1.applyOrElse(DateTimeFormatterHelper.scala:141)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at
org.apache.spark.sql.catalyst.util.Iso8601TimestampFormatter.parse(TimestampFormatter.scala:133)
at
org.apache.spark.sql.catalyst.expressions.ToTimestamp.eval(datetimeExpressions.scala:1211)
```
Java 11/17 will successed and return `1970-12-31 00:00:00.0`
The reason for the difference may be that 'DD' has the following definitions
in the new Java version:
```
* D 1 appendValue(ChronoField.DAY_OF_YEAR)
* DD 2 appendValue(ChronoField.DAY_OF_YEAR, 2, 3,
SignStyle.NOT_NEGATIVE)
* DDD 3 appendValue(ChronoField.DAY_OF_YEAR, 3)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
