jayzhan211 opened a new issue, #26471:
URL: https://github.com/apache/datafusion/issues/26471
### Describe the bug
`corr`, `covar_pop`, `covar_samp` and the `regr_*` functions return the same
result with `DISTINCT` as without it. The planner accepts the query, but the
accumulators never read `is_distinct`.
### To Reproduce
```sql
SELECT regr_count(x, y) AS rc,
regr_count(DISTINCT x, y) AS rc_d,
covar_pop(x, y) AS cv,
covar_pop(DISTINCT x, y) AS cv_d
FROM (VALUES (1.0, 1.0), (1.0, 1.0), (2.0, 3.0)) t(x, y);
```
DataFusion returns `rc = 3, rc_d = 3, cv = 0.444..., cv_d = 0.444...`.
### Expected behavior
`DISTINCT` applies to the argument tuple: `rc_d = 2`, `cv_d = 0.5`. DuckDB
returns these values. If DataFusion does not support it, the query should fail
with a clear error instead of returning the non-distinct answer.
### Additional context
#25288 tagged these functions with `DistinctHandling::Unsupported`, and the
code says so directly: "the accumulator does not read `is_distinct` and today
silently returns the non-distinct answer. The tag records the intent;
enforcement is a follow-up change."
https://github.com/apache/datafusion/blob/3470bd0fbc/datafusion/functions-aggregate/src/covariance.rs#L132-L137
I couldn't find an issue tracking that follow-up. Both options look
reasonable: reject `f(DISTINCT ..)` at planning time when the function reports
`Unsupported`, or deduplicate the input for it. Rejecting is the smaller
change, and it's at least correct.
Related: #25374 and #25375 (forwarding `distinct_handling`).
### Willingness to contribute
Feel free to take it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
