cloud-fan commented on code in PR #59402:
URL: https://github.com/apache/spark/pull/59402#discussion_r4292169103
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3350,6 +3350,29 @@ object SQLConf {
"The threshold must be >= 2 or -1 (disabled)")
.createWithDefault(-1)
+ val REWRITE_COUNT_DISTINCT_CONDITIONAL_ENABLED =
+ buildConf("spark.sql.optimizer.rewriteCountDistinctConditional.enabled")
+ .internal()
+ .doc("When true, rewrites COUNT(DISTINCT IF(cond, base, NULL)) and " +
+ "COUNT(DISTINCT CASE WHEN cond THEN base END) into " +
+ "COUNT(DISTINCT base) FILTER (WHERE cond). This reduces the Expand
factor " +
+ "in RewriteDistinctAggregates from Nx to 1x when multiple conditional
distinct " +
+ "counts share the same base column. Also guards the shared distinct
Expand " +
+ "key with the union of the filters (IF(Or(cond1, ..., condN), base,
NULL)) " +
+ "for an all-filtered distinct group whose children are safe to
evaluate " +
Review Comment:
**Nit (P3):** This says the union-of-filters guard applies to "an
all-filtered distinct group whose children are safe to evaluate unconditionally
(including user-written FILTER on a plain column)". `canGuard` also requires
every filter to be deterministic and free of subqueries
(`RewriteDistinctAggregates.scala:342-344`), and `extractCondAndBase` leaves a
conditional count whose condition contains a subquery un-normalized (`:537`).
So `COUNT(DISTINCT c) FILTER (WHERE b IN (SELECT ...))` on a plain column meets
every condition listed here yet keeps the raw Expand key, and two
`COUNT(DISTINCT IF(b IN (SELECT ...), c, NULL))` on one base are not rewritten.
The class scaladoc and the `normalizeCountDistinctConditional` scaladoc already
state both exclusions. Could this doc state them too? For example: "... whose
filters are deterministic and subquery-free and whose children are safe to
evaluate unconditionally ...", plus a note that conditions containing a
subquery are not rewritten.
**Verification:**
- **Inspection:** The configuration doc's preconditions for the guard and
the IF/CASE rewrite match the checks the rule performs and the exclusions its
scaladoc already lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
