cloud-fan opened a new pull request, #59121: URL: https://github.com/apache/spark/pull/59121
### What changes were proposed in this pull request? This PR reverts the conditional COUNT(DISTINCT) rewrite introduced by https://github.com/apache/spark/commit/9deb4037705a6a6e8d548ee803d323ddf75c693b (#55925). It restores the previous RewriteDistinctAggregates behavior and removes the associated internal configuration and tests. It also reverts the dependent benchmark-only follow-up https://github.com/apache/spark/commit/364fa9b3d576a0d3773053e51ec1b7321d0d085e (#57826), whose benchmark matrix references the removed configuration. ### Why are the changes needed? The rewrite is not always neutral or beneficial once the cardinality of the first, pre-shuffle aggregate is considered. For example: ```sql SELECT g, COUNT(DISTINCT IF(p = 1, x, NULL)), COUNT(DISTINCT IF(p = 2, x, NULL)) FROM t GROUP BY g ``` When x is high-cardinality and most rows match neither predicate, the old plan projects NULL as the distinct key in each Expand branch. The map-side aggregate collapses unmatched rows to one NULL key per gid. After the rewrite, Expand projects raw x for every row. The predicates are reduced with max in the first aggregate, but they do not prevent unmatched x values from entering the grouping key (g, x, gid). The rewrite can therefore reduce Expand input from 2N rows to N while increasing map-side aggregate output from roughly two keys to N keys, causing severe spill and shuffle growth. This failure mode has been observed in a downstream production workload. The original optimization is default-on and is already present in the Spark 4.3.0 release candidate. Reverting is safer before GA. A future reintroduction should cover high-cardinality, low-selectivity inputs and preserve an early union-of-filters guard or apply a safe cost/selectivity condition. Discussion: https://github.com/apache/spark/pull/55925#issuecomment-5846595588 ### Does this PR introduce _any_ user-facing change? Yes, on the unreleased master branch. Conditional COUNT(DISTINCT IF/CASE WHEN) expressions are no longer normalized to COUNT(DISTINCT) with FILTER, and the internal spark.sql.optimizer.rewriteCountDistinctConditional.enabled configuration is removed. This restores the behavior from before #55925; there is no change relative to a GA Spark release. ### How was this patch tested? - `git diff --check upstream/master...HEAD` - Verified that no references to the removed configuration or rewrite remain. - Attempted `build/sbt catalyst/Test/compile sql/Test/compile`; local dependency resolution could not start because Maven Central and the SBT plugin repositories failed DNS resolution. CI will provide the compile and test coverage. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
