924060929 opened a new pull request, #70409:
URL: https://github.com/apache/doris/pull/70409
### Background
Nereids infers `IS NOT NULL` predicates below global aggregates when rows
with NULL arguments can be discarded. The top-down rewriter retries a changed
aggregate until the rewrite rules stop changing its plan.
### What problem does this PR solve?
Issue Number: None
Related PR: #31123 (introduced the non-convergent predicate construction),
#33598 (added the CASE rewrite trigger)
PR Type: Bugfix
Problem Summary:
A global aggregate can repeatedly add filters when its child already filters
only some of the inputs inferred to be non-null. For example, with nullable `a`
and `b`, the following query can keep growing the FE plan and heap during
optimization:
```sql
SELECT SUM(CASE WHEN b IS NOT NULL THEN a * b END) FROM t;
```
Merge newly inferred predicates into an existing logical filter so the next
rule application sees all satisfied conditions and stops.
### Details
- **Frequency:** Deterministic for a supported aggregate whose argument
infers multiple non-null predicates when the immediate filter contains only a
subset. CASE elimination can produce this shape from the example above.
- **Root cause:** `InferAggNotNull` checks only the immediate filter but
wraps the missing predicates in a separate filter. For inferred conditions `{a
IS NOT NULL, b IS NOT NULL}`, an existing filter on `b` becomes a new filter on
`a`; the next application then adds `b` again. The top-down rewriter keeps
retrying the aggregate before visiting the filter children.
- **Why existing tests missed it:** They did not cover an existing filter
containing only part of the inferred non-null conditions, nor the interaction
with CASE elimination.
- **Fix:** Combine the original conjuncts and the missing non-null
predicates with `LogicalFilter.withConjuncts`. Preserve existing predicates and
their generated flags. The rule returns the same plan on the next application.
- **Other paths and branches:** This is a shared Nereids rule used in both
cloud and shared-nothing deployments. Release branches need separate backports.
- **Behavior change:** Affected aggregate queries finish rewriting instead
of growing an unbounded chain of filters and aggregate expressions.
### Testing
- Before the fix, `./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.InferAggNotNullTest#testMergeInferredNotNullWithExistingFilter'`
fails because the rewritten filter still has another filter as its child. This
is a bounded check that does not enter the infinite rewrite loop.
- After the fix, `./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.InferAggNotNullTest,org.apache.doris.nereids.rules.rewrite.EliminateAggCaseWhenTest'`
runs 18 tests with 0 failures, 0 errors and 0 skips. The new tests cover
partial predicates (both generated and user-written), existing complete
predicates, no existing filter, and CASE elimination followed by the Visitor
rewrite path.
- `mvn -Ddisable.obs=true -Ddisable.cos=true checkstyle:check -pl fe-core`
passes with 0 violations.
- Local environment: macOS, JDK 17. Optional OBS and COS dependencies are
excluded from the local build and test reactor. Validation covers planner
rewrites, including the Visitor path.
### Release note
Fix FE planner hangs and excessive heap growth for some global aggregate
queries with existing filters, including conditional SUM expressions.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready
### AI Disclosure
- Tool: Codex
- Generated: Code, tests, and this description
- Human review: Pending author review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
