comphead commented on code in PR #7194:
URL: https://github.com/apache/datafusion-comet/pull/7194#discussion_r4637038141
##########
spark/src/test/resources/sql-tests/expressions/aggregate/regr.sql:
##########
@@ -174,6 +174,34 @@ SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y,
x) FROM test_regr_cons
query
SELECT regr_sxx(y, x), regr_syy(y, x), regr_sxy(y, x) FROM test_regr_const_y
+-- Nullable aggregate FILTERs must preserve group positions and combine with
+-- each function's pair-null rules. Group 1 exists but has no qualifying rows;
+-- group 2 has x values whose y is NULL, which slope/intercept counted before
SPARK-48719.
+statement
+CREATE TABLE test_regr_filtered(g int, y double, x double, keep boolean) USING
parquet
+
+statement
+INSERT INTO test_regr_filtered VALUES
+ (0, 3.0, 1.0, true), (0, 9.0, 4.0, true), (0, 8.0, NULL, true),
+ (0, NULL, 20.0, true), (0, 90.0, 8.0, false), (0, 70.0, 9.0, NULL),
+ (1, 2.0, 5.0, false), (1, 8.0, 9.0, NULL),
+ (2, NULL, 1.0, true), (2, NULL, 4.0, true),
+ (3, 7.0, 1.0, true), (3, 7.0, 4.0, true)
+
+query
+SELECT g,
+ regr_sxx(y, x) FILTER (WHERE keep), regr_syy(y, x) FILTER (WHERE keep),
+ regr_sxy(y, x) FILTER (WHERE keep), regr_slope(y, x) FILTER (WHERE keep),
+ regr_intercept(y, x) FILTER (WHERE keep), regr_r2(y, x) FILTER (WHERE keep)
+FROM test_regr_filtered GROUP BY g ORDER BY g
+
+query
+SELECT g,
+ regr_sxx(x, y) FILTER (WHERE keep), regr_syy(x, y) FILTER (WHERE keep),
+ regr_sxy(x, y) FILTER (WHERE keep), regr_slope(x, y) FILTER (WHERE keep),
+ regr_intercept(x, y) FILTER (WHERE keep), regr_r2(x, y) FILTER (WHERE keep)
+FROM test_regr_filtered GROUP BY g ORDER BY g
Review Comment:
The grouped path is only compared with Comet's own scalar accumulator for
`NaN`, infinities and `-0.0` (in the Rust tests), never with Spark. `regr.sql`
has no `NaN` or `Infinity` input at all, while `corr.sql` has a `NaN`
permutation table. Could you add a small grouped table with `cast('NaN' as
double)`, `cast('Infinity' as double)` and `-0.0` in `y` and `x`, and run the
six functions with `GROUP BY` and `FILTER`? That would check the new loop
against Spark directly. I expect it to pass because the update and merge
arithmetic is shared with the scalar path, but I have not run it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]