andygrove opened a new issue, #7210:
URL: https://github.com/apache/datafusion-comet/issues/7210
### Describe the bug
In an `America/Los_Angeles` session, casting the string `-290308-12-21
19:59:05.224192` to `TIMESTAMP` gives a value 422 seconds off from Spark's:
- Spark: `-9223372008476775808` micros. Java's zone rules use the local mean
time offset, -07:52:58, for dates before 1883.
- Comet: `-9223372008054775808` micros, which is the -08:00 standard offset.
`1800-06-01 00:00:00` in the same session matches Spark
(`-5351587622000000`), so local mean time works for ordinary dates. The
difference shows up at the far end of the range.
The cast code and chrono-tz are unchanged since 1.1.0, so the 1.1.0 release
has the same bug by code; it was not run against 1.1.0.
### Steps to reproduce
```scala
spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
Seq("-290308-12-21 19:59:05.224192").toDF("s").write.parquet("/tmp/lmt")
spark.read.parquet("/tmp/lmt").selectExpr("CAST(s AS TIMESTAMP) AS
t").write.parquet("/tmp/lmt_out")
// read back with Comet off
spark.read.parquet("/tmp/lmt_out").selectExpr("unix_micros(t)").show()
```
The round trip through Parquet is needed for two reasons. Collecting the
timestamp needs a `java.sql.Timestamp` that can hold this year. And
`unix_micros` over the cast goes through the codegen dispatcher, which
evaluates the cast in Spark's code.
### Expected behavior
`-9223372008476775808`, as in Spark.
### Additional context
Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression.
#6816 is a related gap at the far-future end (DST rules after 2099).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
