andygrove opened a new pull request, #6866: URL: https://github.com/apache/datafusion-comet/pull/6866
## Which issue does this PR close? Part of #6385. Follow-up to #6447, which kept this rule until #6519 was fixed, and #6829, which fixed it. ## Rationale for this change `CometExecRule` wrapped the divisor of a `FLOAT` or `DOUBLE` `Divide` or `Remainder` in a Project or Filter in `NormalizeNaNAndZero`. Comparisons don't need it. Since #6447, the native comparison builder follows Spark's ordering, including in the check for a zero divisor. The rule stayed for a side effect: it canonicalized a NaN quotient. Without it, `1.0D / (-d)` of a NaN `d` is a NaN with the sign bit set, which `percentile_approx` ordered below every other value (#6519). #6829 fixed `percentile_approx`, so the wrapper now only adds a copy of each divisor. ## What changes are included in this PR? - `normalizeDivisors` and its `normalizeNaNAndZero` helper are removed from `CometExecRule`, along with the imports that only they used. - `nan_divisor.sql` keeps its `percentile_approx` query, which now checks a quotient whose NaN has the sign bit set. The `hash`, `min`/`max` and `greatest`/`least` queries over the same quotients stay as coverage for those consumers. - The 11 TPC-DS approved plans that hold a float division in a Project or Filter now count one native expression fewer. q78 no longer lists `knownfloatingpointnormalized` and `normalizenanandzero` among the expressions its projection dispatches. ## How are these changes tested? - On Spark 4.1, 553 tests pass: `nan_divisor.sql` with ANSI off and on, `CometFloatSemanticsSuite`, and the `expressions/math/`, `expressions/aggregate/` and `expressions/conditional/` fixtures. - The TPC-DS plans were regenerated on Spark 3.4, 3.5, 4.0, 4.1 and 4.2. Spark 3.4 rewrote the 9 base plans, 3.5 and 4.0 rewrote their own `q83` overrides, and 4.1 and 4.2 needed no override. Both plan stability suites pass on 4.1 and 4.2 without regeneration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
