andygrove opened a new issue, #464:
URL: https://github.com/apache/datafusion-comet/issues/464
### Describe the bug
Comet currently fails with an overflow when ANSI mode is disabled, but we
should return the original value instead.
## Create test data
```
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
```
## Test with ansi mode disabled
With ansi mode disabled we expect abs to return the original value if there
would be an overflow calculating the abs value, but Comet fails with an
overflow.
```
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, abs(a) from t").show
+-----------+-----------+
| a| abs(a)|
+-----------+-----------+
| 2147483647| 2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, abs(a) from t").show
24/05/23 13:26:17 WARN CometSparkSessionExtensions$CometExecRule: Comet
cannot execute some parts of this plan natively because CollectLimit is not
supported
24/05/23 13:26:17 ERROR Executor: Exception in task 0.0 in stage 7.0 (TID 8)
org.apache.comet.CometNativeException: Arrow error: Compute error:
Int32Array overflow on abs(-2147483648)
```
With ansi mode enabled, we do expect an exception, but we should try and
match Spark's exception.
```
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] integer
overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this
error.
```
Comet currently has this error:
```
org.apache.comet.CometNativeException: Arrow error: Compute error:
Int32Array overflow on abs(-2147483648)
```
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
