[ 
https://issues.apache.org/jira/browse/SPARK-61819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xi Liang updated SPARK-61819:
-----------------------------
    Affects Version/s: 4.4.0

> Do not normalize conditional distinct counts with non-deterministic 
> conditions, and read the config once
> --------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61819
>                 URL: https://issues.apache.org/jira/browse/SPARK-61819
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Xi Liang
>            Priority: Major
>
> SPARK-56898 (fix version 4.4.0, not released yet) made 
> {{RewriteDistinctAggregates}} normalize {{COUNT(DISTINCT IF(cond, base, 
> NULL))}} and {{COUNT(DISTINCT CASE WHEN cond THEN base END)}} to 
> {{COUNT(DISTINCT base) FILTER (WHERE cond)}} while 
> {{spark.sql.optimizer.rewriteCountDistinctConditional.enabled}} is true, 
> which is its default. Several such counts over one base then share one group 
> of the {{Expand}} that the rule builds. The rule also guards that group's 
> distinct key, the value that the {{Expand}} projects for it, with the union 
> of the counts' filters: {{IF(cond_1 OR ... OR cond_n, base, NULL)}} (the 
> union-of-filters guard). Rows that match no filter then project NULL.
> For each value of the GROUP BY keys, the first aggregate above the {{Expand}} 
> then outputs, for that group, one row per distinct base value among the rows 
> that match some condition, plus at most one NULL row. Without the 
> normalization, it outputs no fewer: for each condition, one row per distinct 
> base value among the rows that match that condition, plus at most one NULL 
> row. Without the guard, the distinct key is the base itself on every row, so 
> the aggregate outputs, for that group, one row per distinct base value of all 
> rows. Results stay correct, because the FILTER clauses still select the rows 
> of each count, but that output can be much larger.
> The rule can still normalize counts and then build their group without the 
> guard, in two cases:
> # *A non-deterministic condition.* The guard applies only when every filter 
> of the group is deterministic, because it evaluates the filters a second 
> time, in the distinct key. The normalization does not check determinism, so a 
> count whose condition is non-deterministic is normalized and joins the group 
> of the other counts over its base, and then no count of that group is 
> guarded. Analysis rejects non-deterministic expressions in aggregate 
> arguments, and {{CollapseProject}} inlines only deterministic expressions, so 
> we know of no query that reaches this case; the rule should not depend on 
> that.
> # *The config changing while the rule runs*, for example when another thread 
> sets it on the same session. For one {{Aggregate}}, the config is read twice: 
> when the counts are normalized, and when the guard is decided. If it is 
> turned off in between, the normalized counts are kept and their guard is 
> skipped.
> Proposed change:
> * Do not normalize a count whose condition is non-deterministic. It keeps its 
> own group, whose distinct key is its original {{IF}} or {{CASE WHEN}}; the 
> other counts over the same base can still share a guarded group.
> * Read the config once for each {{Aggregate}} that the rule rewrites, and use 
> that value for both the normalization and the guard.
> * Add tests for both cases, for the existing check that only a count with one 
> argument is normalized, for the number of rows that the first aggregate above 
> the {{Expand}} outputs, and for a deterministic condition over a column that 
> a subquery computes with {{input_file_name()}}.
> * Set the config to true in the subexpression elimination benchmark of 
> {{ExpandBenchmark}}, which compares two settings of subexpression elimination 
> on the normalized plan, so that it measures that plan whatever the default. 
> The file's two other benchmarks of this rewrite already set the config in 
> each of their cases.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to