sunchao opened a new issue, #5418:
URL: https://github.com/apache/datafusion-comet/issues/5418
### Describe the bug
An empty native scalar AVG partial currently exports `(sum = NULL, count =
0)`. Spark initializes AVG buffers to `(sum = 0, count = 0)` and adds partial
sums without coalescing nulls. In the supported native-partial / Spark-final
path for non-decimal AVG, an empty partial can therefore turn a nonempty query
result into `NULL`.
The same scalar-state distinction matters for decimal AVG: a legitimate
empty buffer must be distinguishable from an overflow buffer. A fix must keep
the final result of an empty AVG as `NULL` and preserve decimal overflow
behavior.
### Steps to reproduce
With Comet installed and enabled in a fresh local Spark session (for
example, `local[4]`), create four separate Parquet scan partitions and filter
out all rows in three of them:
```python
import tempfile
for key, value in {
"spark.sql.adaptive.enabled": "false",
"spark.sql.files.maxPartitionBytes": "1048576",
"spark.sql.parquet.filterPushdown": "false",
"spark.comet.scan.enabled": "false",
"spark.comet.convert.parquet.enabled": "true",
"spark.comet.shuffle.enabled": "false",
"spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
spark.conf.set(key, value)
with tempfile.TemporaryDirectory() as tmp:
path = f"{tmp}/data"
(spark.range(8, numPartitions=4)
.selectExpr("id", "cast(id + 1 as int) quantity")
.write.parquet(path))
spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id =
1").show()
```
The final-mode testing switch isolates the native-partial / Spark-final
boundary. The query has a native partial aggregate and a Spark final aggregate,
but returns `NULL` instead of `2.0`.
### Expected behavior
The result is `2.0`, matching Spark. If the filter removes every row, the
final result remains `NULL`.
### Additional context
Reproduced against Apache Comet `main` at
`2699f59b71788e17a2714910e166a3f83deed937` using Spark 4.0.4. This is a
partial-buffer representation problem, not a request to enable mixed execution
for decimal AVG.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
