andygrove opened a new issue, #5013:
URL: https://github.com/apache/datafusion-comet/issues/5013
### Describe the bug
With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, casting any
integer column to a negative-scale decimal crashes the native engine:
```
Comet native panic: panicked at library/core/src/num/mod.rs:475:5:
attempt to multiply with overflow
```
`spark.sql.legacy.allowNegativeScaleOfDecimal` is one of the configs listed
under #4180. `CometCast` does consult it (`CometCast.scala:279-289`), but only
for the **decimal → string** direction: a negative-scale target is reported
`Compatible()` when the flag is on. Casting *into* a negative-scale decimal,
and arithmetic on negative-scale decimals, are not guarded at all.
Note that the native workspace sets `overflow-checks = false` for the
release profile (`native/Cargo.toml:67`), so a release build will not panic
here. The same multiplication wraps instead, which would turn this into
silently wrong values rather than a crash. I verified the panic on a debug
build and the profile setting by inspection; I did not run a release build to
confirm what the wrapped result looks like.
### Steps to reproduce
Negative scale cannot be written in SQL (the parser rejects `decimal(20,-5)`
outright), so this needs the DataFrame API:
```scala
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types.DecimalType
spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", "true")
val path = "/tmp/negscale"
spark.sql("SELECT cast(n as bigint) as n FROM VALUES (1),(2),(3) AS v(n)")
.write.mode("overwrite").parquet(path)
val df = spark.read.parquet(path)
spark.conf.set("spark.comet.enabled", "false")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// [0E+1], [0E+1], [0E+1] (Spark rounds to the negative scale)
spark.conf.set("spark.comet.enabled", "true")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// org.apache.comet.CometNativeException: native panic: attempt to multiply
with overflow
```
Also reproduced with:
* `DecimalType(20, -5)` — same panic;
* arithmetic on the result, `n + n` / `n * 2` — `attempt to subtract with
overflow`;
* casting the negative-scale decimal back to string — same panic, i.e. the
one direction `CometCast` does guard is unreachable because producing the value
crashes first.
With `spark.sql.legacy.allowNegativeScaleOfDecimal=false` both engines raise
`NEGATIVE_SCALE_DISALLOWED` identically, so only the `true` case is affected.
### Expected behavior
Comet should either handle negative-scale decimals correctly, or report
`Unsupported`/`Incompatible` for any cast or arithmetic involving a
negative-scale decimal so the expression falls back to Spark. It should not
panic, and it should not wrap silently in release builds.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark
4.1.2), macOS aarch64, debug build.
* Found while auditing the config inventory in #4180.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
