Vivek1106-04 opened a new pull request, #58154:
URL: https://github.com/apache/spark/pull/58154
### What changes were proposed in this pull request?
This PR makes `OptimizeExpand` reject the rewrite when the pre-aggregate it
would insert has an empty grouping.
`OptimizeExpand` inserts a grouping-only `Aggregate` below the `Expand`
produced by `RewriteDistinctAggregates`, so that rows are de-duplicated before
the `Expand` amplifies them. The pre-aggregate's grouping is collected from the
attributes the `Expand` references:
```scala
expand.child.output.filter(expand.references.contains)
```
When the query has no `GROUP BY` and every distinct argument is foldable,
the `Expand` references no attribute of its child, so the inserted node is
`Aggregate(Nil, Nil, child)` -- a *global* aggregate. A global aggregate emits
one row even for an empty input instead of de-duplicating rows, so the `Expand`
above it sees one row where it previously saw none.
The new check rejects that case; a non-empty grouping is what makes the
inserted node a de-duplication rather than a global aggregate.
### Why are the changes needed?
`COUNT(DISTINCT <constant>)` over an empty input returns 1 instead of 0:
```sql
-- an empty source, e.g. a daily partition with no rows yet
CREATE OR REPLACE TEMP VIEW events AS
SELECT * FROM VALUES (1, 'x', 'y') AS t(id, c1, c2) WHERE 1 = 0;
SET spark.sql.optimizer.optimizeExpandRatio=2;
SELECT COUNT(DISTINCT region), COUNT(DISTINCT channel)
FROM (SELECT 'US' AS region, 'web' AS channel, id FROM events);
-- returns [1,1], expected [0,0]
```
No constant literal has to appear in the query text: the distinct arguments
only need to be *foldable*, which the optimizer proves for the common pattern
of tagging constant dimensions in a subquery, view, or `UNION ALL` branch. The
minimal form is `SELECT COUNT(DISTINCT 1), COUNT(DISTINCT 2) FROM events`.
The optimized plan shows the inserted global aggregate:
```
Aggregate [count(1) FILTER (gid=2) AS count(DISTINCT region), count(1)
FILTER (gid=1) AS count(DISTINCT channel)]
+- Aggregate ['web', 'US', gid], [gid]
+- Expand [[web, null, 1], [null, US, 2]], ['web', 'US', gid]
+- Aggregate <- inserted by OptimizeExpand: no
grouping, no aggregates
+- LocalRelation <empty>
```
Excluding the rule by name
(`spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.OptimizeExpand`)
gives the correct answer, confirming the rule is the cause and not the
surrounding rewrite.
This is the same empty-table failure that
`RewriteDistinctAggregates.mustRewrite` exists to prevent -- its comment names
`SELECT COUNT(DISTINCT 1)` and "wrong results when working with empty tables"
as the reason the rewrite is forced for this shape. `OptimizeExpand` then
undoes that protection.
The bug was introduced by SPARK-56315 (4.2.0). It is reachable only when the
internal conf `spark.sql.optimizer.optimizeExpandRatio` is explicitly set,
since it defaults to `-1` (disabled).
This is independent of SPARK-58387: there the inner `Aggregate` computes a
duplicate-sensitive aggregate, here it computes no aggregate at all, so that
fix's duplicate-sensitivity guard does not reject this plan.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a correctness bug. With
`spark.sql.optimizer.optimizeExpandRatio` set, a query with two or more
foldable-argument distinct aggregates and no `GROUP BY` returned 1 per distinct
aggregate over an empty input; it now returns 0, matching the result with the
rule disabled. No behavior change under the default conf value (`-1`).
### How was this patch tested?
New tests, verified to fail before the fix and pass after:
- `OptimizeExpandSuite`
- "SPARK-58888: skips when every distinct argument is foldable" -- asserts
no pre-aggregate is inserted.
- "SPARK-58888: applies when one distinct argument is a real column" --
guards against over-rejecting: the column enters the pre-aggregate's `GROUP
BY`, so the rewrite still applies.
- `OptimizeExpandQuerySuite`
- "SPARK-58888: count distinct of constants over an empty input"
- "SPARK-58888: count distinct of constant-projected columns over an empty
input" (the reported repro)
- "SPARK-58888: count distinct of constants over a non-empty input" --
unchanged result of 1.
Both suites pass in full (`OptimizeExpandSuite` 10/10,
`OptimizeExpandQuerySuite` 11/11).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
