cloud-fan commented on PR #55925:
URL: https://github.com/apache/spark/pull/55925#issuecomment-5846595588
I propose reverting #55925 from `master` and `branch-4.3` before Spark 4.3.0
GA.
The rewrite is not neutral-or-better once we consider the cardinality of the
first, pre-shuffle aggregate. For example:
```sql
SELECT g,
COUNT(DISTINCT IF(p = 1, x, NULL)),
COUNT(DISTINCT IF(p = 2, x, NULL))
FROM t
GROUP BY g
```
Assume `x` is high-cardinality and most rows match neither predicate.
Before #55925, `Expand` has two branches, but an unmatched row projects
`NULL` as the distinct key in each branch. The map-side aggregate therefore
collapses all such rows to one NULL key per `gid`.
After #55925, the counts share one distinct group, but `Expand` projects raw
`x` for every row. The predicates are reduced with `max` in the first
aggregate; they do not prevent unmatched `x` values from entering its grouping
key `(g, x, gid)`. Consequently, the rewrite reduces Expand input from `2N`
rows to `N`, while potentially increasing map-side aggregate output from
roughly 2 keys to `N` keys. With selective predicates and mostly unique `x`,
this causes severe map-side spill and shuffle growth. We have observed this
exact failure mode in a downstream production workload.
This is entirely in the upstream Catalyst plan, the configuration defaults
to `true`, and the change is already included in `v4.3.0-rc1`.
The current benchmark does not cover this case: it uses low-cardinality `pt`
as the distinct child and includes a predicate that covers all rows. A
reintroduction should include a high-cardinality, low-selectivity benchmark and
either preserve an early union-of-filters guard around the shared base or apply
a safe cost/selectivity condition.
Until then, I think reverting is safer than shipping this default-on. The
temporary workaround is:
```text
spark.sql.optimizer.rewriteCountDistinctConditional.enabled=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
