andygrove opened a new issue, #6739:
URL: https://github.com/apache/datafusion-comet/issues/6739
### Describe the bug
`expressions/datetime/trunc_timestamp.sql` fails on the Spark 4.2 profile in
both dictionary variants:
```
Error executing SQL 'SELECT date_trunc('SECOND', ts),
date_trunc('MILLISECOND', ts)
FROM test_trunc_ts_overflow' Job aborted due to stage failure: ...
java.lang.ArithmeticException: long overflow
at java.base/java.lang.Math.subtractExact(Math.java:945)
at
org.apache.spark.sql.catalyst.util.DateTimeUtils$.truncTimestamp(DateTimeUtils.scala:542)
```
The query came in with #5956. Its fixture expects Spark to wrap the Long
subtraction when SECOND or MILLISECOND truncation of
`timestamp_micros(-9223372036854775808)` falls below `Long.MinValue`. Spark
4.1.3 and earlier do wrap. Spark 4.2.0 uses `Math.subtractExact` there and
raises `long overflow`, like every other unit (SPARK-56663,
apache/spark@da095f9e63). The pull request tier only runs Spark 4.1, so this
first showed up on #6664, which has `run-all-spark-profiles`. The nightly will
fail the same way.
The fixture is the visible half. Comet's native kernel wraps on every Spark
version (`timestamp_trunc_fine_boundary` in
`native/spark-expr/src/kernels/temporal.rs`), so on Spark 4.2 Comet returns a
timestamp in the year 294247 where Spark raises.
### Steps to reproduce
```shell
./mvnw test -Pspark-4.2 -Dtest=none -DwildcardSuites=none \
-Dsuites="org.apache.comet.CometSqlFileTestSuite trunc_timestamp.sql"
```
### Expected behavior
On Spark 4.2, Comet raises `long overflow` when SECOND or MILLISECOND
truncation falls below the smallest timestamp. On earlier versions it keeps
wrapping, as Spark does.
### Additional context
Spark 4.1.3 `DateTimeUtils.truncTimestamp`:
```scala
case TRUNC_TO_SECOND =>
micros - Math.floorMod(micros, MICROS_PER_SECOND)
```
Spark 4.2.0:
```scala
case TRUNC_TO_SECOND =>
Math.subtractExact(micros, Math.floorMod(micros, MICROS_PER_SECOND))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
