Xi Liang created SPARK-61819:
--------------------------------
Summary: Do not normalize conditional distinct counts with
non-deterministic conditions, and read the config once
Key: SPARK-61819
URL: https://issues.apache.org/jira/browse/SPARK-61819
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Xi Liang
SPARK-56898 (fix version 4.4.0, not released yet) made
{{RewriteDistinctAggregates}} normalize {{COUNT(DISTINCT IF(cond, base,
NULL))}} and {{COUNT(DISTINCT CASE WHEN cond THEN base END)}} to
{{COUNT(DISTINCT base) FILTER (WHERE cond)}} while
{{spark.sql.optimizer.rewriteCountDistinctConditional.enabled}} is true, which
is its default. Several such counts over one base then share one group of the
{{Expand}} that the rule builds. The rule also guards that group's distinct
key, the value that the {{Expand}} projects for it, with the union of the
counts' filters: {{IF(cond_1 OR ... OR cond_n, base, NULL)}} (the
union-of-filters guard). Rows that match no filter then project NULL.
For each value of the GROUP BY keys, the first aggregate above the {{Expand}}
then outputs, for that group, one row per distinct base value among the rows
that match some condition, plus at most one NULL row. Without the
normalization, it outputs no fewer: for each condition, one row per distinct
base value among the rows that match that condition, plus at most one NULL row.
Without the guard, the distinct key is the base itself on every row, so the
aggregate outputs, for that group, one row per distinct base value of all rows.
Results stay correct, because the FILTER clauses still select the rows of each
count, but that output can be much larger.
The rule can still normalize counts and then build their group without the
guard, in two cases:
# *A non-deterministic condition.* The guard applies only when every filter of
the group is deterministic, because it evaluates the filters a second time, in
the distinct key. The normalization does not check determinism, so a count
whose condition is non-deterministic is normalized and joins the group of the
other counts over its base, and then no count of that group is guarded.
Analysis rejects non-deterministic expressions in aggregate arguments, and
{{CollapseProject}} inlines only deterministic expressions, so we know of no
query that reaches this case; the rule should not depend on that.
# *The config changing while the rule runs*, for example when another thread
sets it on the same session. For one {{Aggregate}}, the config is read twice:
when the counts are normalized, and when the guard is decided. If it is turned
off in between, the normalized counts are kept and their guard is skipped.
Proposed change:
* Do not normalize a count whose condition is non-deterministic. It keeps its
own group, whose distinct key is its original {{IF}} or {{CASE WHEN}}; the
other counts over the same base can still share a guarded group.
* Read the config once for each {{Aggregate}} that the rule rewrites, and use
that value for both the normalization and the guard.
* Add tests for both cases, for the existing check that only a count with one
argument is normalized, for the number of rows that the first aggregate above
the {{Expand}} outputs, and for a deterministic condition over a column that a
subquery computes with {{input_file_name()}}.
* Set the config to true in the subexpression elimination benchmark of
{{ExpandBenchmark}}, which compares two settings of subexpression elimination
on the normalized plan, so that it measures that plan whatever the default. The
file's two other benchmarks of this rewrite already set the config in each of
their cases.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]