andygrove opened a new issue, #7007:
URL: https://github.com/apache/datafusion-comet/issues/7007
### Describe the bug
Native decimal `AVG` returns `NULL` when the sum is large for its scale.
`avg()` in `native/spark-expr/src/agg_funcs/avg_decimal.rs` multiplies the
unscaled `i128` sum by `10^(result scale - sum scale)`, which is `10^4` for
most inputs, before it divides by the count. It turns a `checked_mul` overflow
into `NULL`. Spark divides with `BigDecimal` and only checks that the final
average fits the result type.
For `DECIMAL(38,18)`, Spark's type for Scala and Java `BigDecimal` fields,
any group whose sum passes about `1.7e16` returns `NULL`. That covers global
and grouped `AVG`, with ANSI on and off. The threshold is about `1.7 * 10^(34 -
scale)`, so higher scales hit it sooner.
The same path also hides a real range error under ANSI. A single
`100000000000000000` row in a `DECIMAL(38,18)` column averages to a value that
doesn't fit `DECIMAL(38,22)`. Spark raises `NUMERIC_VALUE_OUT_OF_RANGE` there,
but Comet returns `NULL`.
### Steps to reproduce
With Comet installed and Comet shuffle enabled:
```scala
val path = java.nio.file.Files.createTempDirectory("avg-scale").toString +
"/data"
spark.conf.set("spark.comet.enabled", "false")
Seq((1, "9000000000000000"), (1, "9000000000000000"))
.toDF("k", "raw")
.selectExpr("k", "CAST(raw AS DECIMAL(38,18)) AS v")
.coalesce(1)
.write
.parquet(path)
spark.conf.set("spark.comet.enabled", "true")
spark.read.parquet(path).createOrReplaceTempView("t")
spark.sql("SELECT avg(v) FROM t").show(false)
spark.sql("SELECT k, avg(v) FROM t GROUP BY k").show(false)
```
Both queries run as a native partial and a native final
`CometHashAggregateExec`, and both return `NULL`.
### Expected behavior
Both queries return `9000000000000000.0000000000000000000000`, as Spark
does, with ANSI on and off.
### Additional context
Reproduced on main at 66c63bd307 with Spark 4.1.3. `avg()` is unchanged at
1.0.0 and 1.1.0, so both releases have it, and it isn't a regression.
#5420 widens an overflowing scaled numerator to `i256` and raises
`NUMERIC_VALUE_OUT_OF_RANGE` under ANSI, with grouped regressions for both
rows. This issue tracks the defect on main apart from the AVG partial-buffer
work in #5418.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
