andygrove opened a new issue, #5148: URL: https://github.com/apache/datafusion-comet/issues/5148
## Describe the bug In ANSI mode, `avg` on a decimal column raises `[ARITHMETIC_OVERFLOW] Overflow in sum of decimals` when an ungrouped aggregate has nothing to average — either an empty input or an input where every value is null. Spark returns `NULL` in both cases. The error comes from the non-grouped `AvgDecimalAccumulator` in `native/spark-expr/src/agg_funcs/avg_decimal.rs`: - `merge_batch` sets `is_empty = partial_counts.len() == partial_counts.null_count()`. A global partial aggregate over zero rows still emits one row with `sum = NULL, count = 0`, and that count is not null, so `is_empty` flips to `false`. - `evaluate` then sees `sum.is_none() && !is_empty` and reports an overflow. The adjacent `is_not_null` check a few lines below guards with `count > 0`; this one does not. The `GroupsAccumulator` path guards its equivalent check with `count > 0`, so only aggregates without `GROUP BY` are affected. ## Steps to reproduce ```sql set spark.sql.ansi.enabled = true; create table t(a decimal(38, 2)) using parquet; select avg(a) from t; ``` ``` org.apache.spark.SparkArithmeticException: [ARITHMETIC_OVERFLOW] Overflow in sum of decimals. Use 'try_avg' to tolerate overflow and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error. SQLSTATE: 22003 ``` The same happens for a table that has rows where every `a` value is null. ## Expected behavior `NULL`, matching Spark. There is no sum to overflow when nothing was accumulated. ## Additional context Found while investigating the Spark 4.1 SQL test failures on https://github.com/apache/datafusion-comet/pull/5114. `LogicalPlanTagInSparkPlanSuite` q24a/q24b/q24-v2.7 hit this error because that suite creates the TPC-DS tables with no rows and the q24 scalar subquery is `SELECT 0.05 * avg(netpaid) FROM ssales`. The Spark 3.5 SQL lane does not catch it because ANSI is off by default there. A fix is included in https://github.com/apache/datafusion-comet/pull/5114 (separate commit), since that PR needs it to get the Spark 4.1 lane green. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
