cloud-fan commented on PR #59402:
URL: https://github.com/apache/spark/pull/59402#issuecomment-6141186974
Thanks for re-landing this with the guard. The cardinality argument holds
for the groups the guard covers: IF(Or(conds), base, NULL) preserves results,
and its key set is bounded by the per-condition keys. The determinism
precondition always holds after analysis, since non-deterministic FILTERs and
non-deterministic aggregate arguments are both rejected. One gap in 'never
worse than the pre-rewrite plan': rewrite() normalizes every matching COUNT
even when that does not reduce the number of distinct groups. COUNT(DISTINCT
IF(c, x, NULL)) next to SUM(DISTINCT IF(c, x, NULL)) used to be one distinct
group with no Expand; now it is two groups with a 2x Expand. A repeated
COUNT(DISTINCT IF(c, x, NULL)) next to COUNT(*) likewise moves from the
single-distinct path onto the Expand path. Could rewrite() keep the normalized
aggregate only when it has fewer distinct groups than the original?
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6124805791","thread_id":"conversation:6124805791","verdict_sha256":"acbbef33321a9ebe90df2b63de54ebb93a89a11d9ea16704d30e5d7e50f00504"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
