andygrove opened a new issue, #465:
URL: https://github.com/apache/datafusion-comet/issues/465
### What is the problem the feature request solves?
Comet does not support ANSI mode for UnaryMinus.
## Create test data
```
val df = Seq(Int.MaxValue, Int.MinValue).toDF("a")
df.write.parquet("/tmp/int.parquet")
spark.read.parquet("/tmp/int.parquet").createTempView("t")
```
## Test with ANSI mode disabled
Behavior is correct with ANSI mode disabled:
```
scala> spark.conf.set("spark.sql.ansi.enabled", false)
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, -a from t").show
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:00 WARN CometSparkSessionExtensions$CometExecRule: Comet
cannot execute some parts of this plan natively because CollectLimit is not
supported
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
```
## Test with ANSI mode enabled
With ANSI mode enabled, Spark throws an exception, but Comet does not.
```
spark.conf.set("spark.sql.ansi.enabled", true)
spark.conf.set("spark.comet.ansi.enabled", true)
scala> spark.conf.set("spark.comet.enabled", false)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:36 WARN CometSparkSessionExtensions$CometExecRule: Using
Comet's experimental support for ANSI mode.
24/05/23 13:55:36 ERROR Executor: Exception in task 0.0 in stage 18.0 (TID
18)
org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] integer
overflow. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this
error.
scala> spark.conf.set("spark.comet.enabled", true)
scala> spark.sql("select a, -a from t").show
24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Using
Comet's experimental support for ANSI mode.
24/05/23 13:55:48 WARN CometSparkSessionExtensions$CometExecRule: Comet
cannot execute some parts of this plan natively because CollectLimit is not
supported
+-----------+-----------+
| a| (- a)|
+-----------+-----------+
| 2147483647|-2147483647|
|-2147483648|-2147483648|
+-----------+-----------+
```
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
