This is an automated email from the ASF dual-hosted git repository.
924060929 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 70fc3282b39 [fix](nereids) Merge existing filters when adding
predicates (#70409)
70fc3282b39 is described below
commit 70fc3282b393c86f81957a127886d5c375c675ee
Author: 924060929 <[email protected]>
AuthorDate: Mon Nov 16 14:23:13 2026 +0800
[fix](nereids) Merge existing filters when adding predicates (#70409)
### Background
Nereids rules add filtering predicates through `PlanUtils.filter` and
`filterOrSelf`. Incremental inference rules can add only the conditions
that are missing from an existing filter. The top-down rewriter retries
a changed node until the rules stop changing it.
### What problem does this PR solve?
Issue Number: None
Related PR: #31123 (introduced the non-convergent aggregate inference
path), #33598 (added the CASE rewrite trigger)
PR Type: Bugfix
Problem Summary:
The shared filter-construction helper always creates a new filter, even
when its input is already a filter. An inference rule that checks only
its immediate filter and adds missing conditions can therefore alternate
between subsets indefinitely, growing the FE plan and heap. With
nullable `a` and `b`, CASE elimination can trigger this behavior for:
```sql
SELECT SUM(CASE WHEN b IS NOT NULL THEN a * b END) FROM t;
```
Make the shared construction helper merge predicates into an immediately
adjacent logical filter and reuse the existing node when all requested
conditions and their original SQL provenance are already represented.
The aggregate inference rule needs no special handling.
### Details
- **Frequency:** Deterministic for the aggregate example when the
immediate filter contains only part of the inferred non-null conditions.
Other rules also share the same filter-construction helper.
- **Root cause:** After CASE elimination produces a filter on `b`,
aggregate inference requests the missing condition on `a`. Wrapping it
in a separate filter hides `b` from the next inference pass, which
requests `b` again and hides `a` in turn.
- **Why existing tests missed it:** They did not cover partial
conditions below an aggregate or repeated predicate insertion through
the shared helper.
- **Fix:** `PlanUtils.filter` combines the old and new conjuncts before
constructing the filter. If every requested conjunct is already present,
it returns the original filter. `filterOrSelf` delegates to this helper.
Newly combined filters derive fresh logical properties. Predicate
deduplication preserves generated not-null flags and gives original SQL
predicates precedence over equivalent inferred predicates, whose
provenance is ignored by expression equality.
- **Other paths and branches:** The helper is used by non-null
inference, predicate inference, predicate pushdown, subquery rewrites
and MV predicate handling. Only directly adjacent logical filters are
combined; operator boundaries such as Project are preserved, as is the
empty-predicate contract. This shared Nereids code applies to cloud and
shared-nothing deployments; release branches need separate backports.
- **Behavior change:** Affected queries finish rewriting instead of
growing an unbounded filter chain. Incremental filtering through the
helper produces a combined filter rather than another wrapper.
### Testing
- Before the fix, `./run-fe-ut.sh --run
'org.apache.doris.nereids.util.PlanUtilsTest#filterMergesConjuncts+filterReusesExistingPredicates'`
fails both bounded tests: it adds another filter layer and recreates
already-present conditions.
- The standard `run-fe-ut.sh --run` runner passes 97 tests across 13
suites, with 0 failures, 0 errors and 0 skips. Suites: `StructInfoTest`,
`EliminateAggCaseWhenTest`, `EliminateNotNullTest`,
`InferAggNotNullTest`, `InferFilterNotNullTest`, `InferJoinNotNullTest`,
`InferPredicatesTest`, `PushDownFilterThroughAggregationTest`,
`PushDownFilterThroughJoinTest`, `PushDownJoinOtherConditionTest`,
`PushFilterInsideJoinTest`, `UnCorrelatedApplyFilterTest`,
`PlanUtilsTest`. Coverage includes aggregate convergence through
CASE/Visitor, shared filter insertion, operator boundaries, generated
not-null flags and original-vs-inferred predicate provenance.
- Final Java Checkstyle passes with 0 violations; `git diff --check`
passes.
- Local environment: macOS, JDK 17; optional OBS and COS dependencies
are excluded. Validation covers planner rewrites, including the CASE and
Visitor path.
### Release note
Fix FE planner hangs and excessive heap growth caused by repeatedly
adding filter predicates during optimization, including some conditional
SUM queries.
### AI Disclosure
- Tool: Codex
- Generated: Code, tests, and this description
- Human review: Pending author review of the final code
---
.../org/apache/doris/nereids/util/PlanUtils.java | 24 ++++++
.../nereids/rules/rewrite/InferAggNotNullTest.java | 90 ++++++++++++++++++++++
.../apache/doris/nereids/util/PlanUtilsTest.java | 71 +++++++++++++++++
3 files changed, 185 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/PlanUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/PlanUtils.java
index 11773fd4c03..a1f80a20daa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/util/PlanUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/util/PlanUtils.java
@@ -83,10 +83,34 @@ import java.util.stream.Collectors;
* Util for plan
*/
public class PlanUtils {
+ /** Create a filter, merging predicates into a directly adjacent logical
filter. */
public static Optional<LogicalFilter<? extends Plan>>
filter(Set<Expression> predicates, Plan plan) {
if (predicates.isEmpty()) {
return Optional.empty();
}
+ if (plan instanceof LogicalFilter) {
+ LogicalFilter<?> filter = (LogicalFilter<?>) plan;
+ Map<Expression, Expression> conjuncts =
Maps.newLinkedHashMapWithExpectedSize(
+ filter.getConjuncts().size() + predicates.size());
+ for (Expression predicate : filter.getConjuncts()) {
+ conjuncts.put(predicate, predicate);
+ }
+ boolean changed = false;
+ for (Expression predicate : predicates) {
+ Expression existing = conjuncts.get(predicate);
+ // Original SQL predicates take precedence over equal inferred
predicates.
+ if (existing == null || (ExpressionUtils.isInferred(existing)
+ && !ExpressionUtils.isInferred(predicate))) {
+ conjuncts.put(predicate, predicate);
+ changed = true;
+ }
+ }
+ if (!changed) {
+ return Optional.of(filter);
+ }
+ predicates = ImmutableSet.copyOf(conjuncts.values());
+ plan = filter.child();
+ }
return Optional.of(new LogicalFilter<>(predicates, plan));
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNullTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNullTest.java
index eee7c008631..6b1a361001c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNullTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNullTest.java
@@ -23,9 +23,15 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.Type;
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.jobs.rewrite.TopDownVisitorRewriteJob;
+import org.apache.doris.nereids.rules.FilteredRules;
+import org.apache.doris.nereids.rules.Rule;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.Multiply;
import org.apache.doris.nereids.trees.expressions.Not;
import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
import org.apache.doris.nereids.trees.expressions.functions.agg.ArrayAgg;
@@ -42,11 +48,15 @@ import
org.apache.doris.nereids.trees.expressions.functions.agg.Min;
import
org.apache.doris.nereids.trees.expressions.functions.agg.NullIgnoringAggregateFunction;
import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
import org.apache.doris.nereids.trees.expressions.functions.agg.Sum0;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.plans.RelationId;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.types.IntegerType;
import org.apache.doris.nereids.util.LogicalPlanBuilder;
import org.apache.doris.nereids.util.MemoPatternMatchSupported;
import org.apache.doris.nereids.util.MemoTestUtils;
@@ -66,6 +76,86 @@ import java.util.Set;
class InferAggNotNullTest implements MemoPatternMatchSupported {
private final LogicalOlapScan scan1 =
PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ @Test
+ void testMergeInferredNotNullWithExistingFilter() {
+ LogicalOlapScan scan = newWideNullableScan(2);
+ Expression a = scan.getOutput().get(0);
+ Expression b = scan.getOutput().get(1);
+ Expression existingPredicate = new GreaterThan(a, new
IntegerLiteral(0));
+ for (boolean generated : ImmutableList.of(false, true)) {
+ Expression bNotNull = new Not(new IsNull(b), generated);
+ LogicalAggregate<?> aggregate = new
LogicalAggregate<>(ImmutableList.of(),
+ ImmutableList.of(new Alias(new Sum(new Multiply(a, b)),
"total")),
+ new LogicalFilter<>(ImmutableSet.of(bNotNull,
existingPredicate), scan));
+ CascadesContext context =
MemoTestUtils.createCascadesContext(aggregate);
+ Rule rule = new InferAggNotNull().build();
+
+ LogicalAggregate<?> rewritten = (LogicalAggregate<?>)
rule.transform(aggregate, context).get(0);
+ LogicalFilter<?> filter = (LogicalFilter<?>) rewritten.child();
+ Assertions.assertSame(scan, filter.child());
+ Assertions.assertEquals(ImmutableSet.of(bNotNull,
existingPredicate,
+ new Not(new IsNull(a), true)), filter.getConjuncts());
+ Assertions.assertSame(rewritten, rule.transform(rewritten,
context).get(0));
+ }
+ }
+
+ @Test
+ void testInferMultipleInputsWithoutFilter() {
+ LogicalOlapScan scan = newWideNullableScan(2);
+ Expression a = scan.getOutput().get(0);
+ Expression b = scan.getOutput().get(1);
+ LogicalAggregate<?> aggregate = new
LogicalAggregate<>(ImmutableList.of(),
+ ImmutableList.of(new Alias(new Sum(new Multiply(a, b)),
"total")), scan);
+ CascadesContext context =
MemoTestUtils.createCascadesContext(aggregate);
+ Rule rule = new InferAggNotNull().build();
+
+ LogicalAggregate<?> rewritten = (LogicalAggregate<?>)
rule.transform(aggregate, context).get(0);
+ LogicalFilter<?> filter = (LogicalFilter<?>) rewritten.child();
+ Assertions.assertSame(scan, filter.child());
+ Assertions.assertEquals(ImmutableSet.of(new Not(new IsNull(a), true),
+ new Not(new IsNull(b), true)), filter.getConjuncts());
+ Assertions.assertSame(rewritten, rule.transform(rewritten,
context).get(0));
+ }
+
+ @Test
+ void testKeepFilterWithAllInferredNotNullPredicates() {
+ LogicalOlapScan scan = newWideNullableScan(2);
+ Expression a = scan.getOutput().get(0);
+ Expression b = scan.getOutput().get(1);
+ LogicalAggregate<?> aggregate = new
LogicalAggregate<>(ImmutableList.of(),
+ ImmutableList.of(new Alias(new Sum(new Multiply(a, b)),
"total")),
+ new LogicalFilter<>(ImmutableSet.of(new Not(new IsNull(a)),
new Not(new IsNull(b))), scan));
+ CascadesContext context =
MemoTestUtils.createCascadesContext(aggregate);
+
+ Assertions.assertSame(aggregate, new
InferAggNotNull().build().transform(aggregate, context).get(0));
+ }
+
+ @Test
+ void testInferAfterEliminateAggCaseWhen() {
+ LogicalOlapScan scan = newWideNullableScan(2);
+ Expression a = scan.getOutput().get(0);
+ Expression b = scan.getOutput().get(1);
+ Expression bNotNull = new Not(new IsNull(b));
+ If conditional = new If(bNotNull, new Multiply(a, b), new
NullLiteral(IntegerType.INSTANCE));
+ LogicalAggregate<?> aggregate = new
LogicalAggregate<>(ImmutableList.of(),
+ ImmutableList.of(new Alias(new Sum(conditional), "total")),
scan);
+ CascadesContext context =
MemoTestUtils.createCascadesContext(aggregate);
+ Rule eliminateCase = new EliminateAggCaseWhen().build();
+ Rule inferNotNull = new InferAggNotNull().build();
+
+ // Check one rewrite first so a regression fails without entering an
unbounded visitor loop.
+ LogicalAggregate<?> withoutCase = (LogicalAggregate<?>)
eliminateCase.transform(aggregate, context).get(0);
+ LogicalAggregate<?> rewritten = (LogicalAggregate<?>)
inferNotNull.transform(withoutCase, context).get(0);
+ LogicalFilter<?> filter = (LogicalFilter<?>) rewritten.child();
+ Assertions.assertSame(scan, filter.child());
+ Assertions.assertEquals(ImmutableSet.of(bNotNull, new Not(new
IsNull(a), true)), filter.getConjuncts());
+ Assertions.assertSame(rewritten, inferNotNull.transform(rewritten,
context).get(0));
+
+ new TopDownVisitorRewriteJob(new
FilteredRules(ImmutableList.of(eliminateCase, inferNotNull)), plan -> true)
+ .execute(context.getCurrentJobContext());
+ Assertions.assertTrue(rewritten.deepEquals(context.getRewritePlan()));
+ }
+
@Test
void testInfer() {
LogicalPlan plan = new LogicalPlanBuilder(scan1)
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanUtilsTest.java
index e3d744f2e40..31cd7fc100a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanUtilsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanUtilsTest.java
@@ -24,19 +24,90 @@ import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.load.NereidsLoadUtils;
import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.GreaterThan;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.OriginStatement;
import org.apache.doris.utframe.UtFrameUtils;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableSet;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
class PlanUtilsTest {
+ @Test
+ void filterMergesConjuncts() {
+ LogicalOlapScan scan = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ Expression aNotNull = new Not(new IsNull(scan.getOutput().get(0)),
true);
+ Expression bNotNull = new Not(new IsNull(scan.getOutput().get(1)));
+ Expression generatedBNotNull = new Not(new
IsNull(scan.getOutput().get(1)), true);
+ Expression existingPredicate = new
GreaterThan(scan.getOutput().get(0), new IntegerLiteral(0));
+ ImmutableSet<Expression> existingPredicates =
ImmutableSet.of(bNotNull, existingPredicate);
+ LogicalFilter<?> original = new LogicalFilter<>(existingPredicates,
scan);
+
+ LogicalFilter<?> merged = PlanUtils.filter(ImmutableSet.of(aNotNull,
generatedBNotNull), original).get();
+
+ Assertions.assertSame(scan, merged.child());
+ Assertions.assertEquals(ImmutableSet.of(aNotNull, bNotNull,
generatedBNotNull, existingPredicate),
+ merged.getConjuncts());
+ Assertions.assertEquals(existingPredicates, original.getConjuncts());
+ }
+
+ @Test
+ void filterReusesExistingPredicates() {
+ LogicalOlapScan scan = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ Expression aNotNull = new Not(new IsNull(scan.getOutput().get(0)),
true);
+ Expression bNotNull = new Not(new IsNull(scan.getOutput().get(1)),
true);
+ ImmutableSet<Expression> predicates = ImmutableSet.of(aNotNull,
bNotNull);
+ LogicalFilter<?> filter = new LogicalFilter<>(predicates, scan);
+
+ Assertions.assertSame(filter, PlanUtils.filter(predicates,
filter).get());
+ Assertions.assertSame(filter,
PlanUtils.filter(ImmutableSet.of(aNotNull), filter).get());
+ Assertions.assertSame(filter,
PlanUtils.filterOrSelf(ImmutableSet.of(bNotNull), filter));
+ Assertions.assertSame(filter,
PlanUtils.filterOrSelf(ImmutableSet.of(), filter));
+ Assertions.assertTrue(PlanUtils.filter(ImmutableSet.of(),
filter).isEmpty());
+ }
+
+ @Test
+ void filterPreservesOriginalPredicate() {
+ LogicalOlapScan scan = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ Expression originalPredicate = new
GreaterThan(scan.getOutput().get(0), new IntegerLiteral(0));
+ Expression inferredPredicate = originalPredicate.withInferred(true);
+ Assertions.assertEquals(originalPredicate, inferredPredicate);
+
+ LogicalFilter<?> inferredFilter = new
LogicalFilter<>(ImmutableSet.of(inferredPredicate), scan);
+ LogicalFilter<?> merged =
PlanUtils.filter(ImmutableSet.of(originalPredicate), inferredFilter).get();
+ Assertions.assertSame(scan, merged.child());
+ Assertions.assertSame(originalPredicate,
merged.getConjuncts().iterator().next());
+
Assertions.assertTrue(inferredFilter.getConjuncts().iterator().next().isInferred());
+
+ LogicalFilter<?> originalFilter = new
LogicalFilter<>(ImmutableSet.of(originalPredicate), scan);
+ Assertions.assertSame(originalFilter,
PlanUtils.filter(ImmutableSet.of(inferredPredicate), originalFilter).get());
+ }
+
+ @Test
+ void filterDoesNotMergeThroughProject() {
+ LogicalOlapScan scan = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ Expression aNotNull = new Not(new IsNull(scan.getOutput().get(0)),
true);
+ Expression bNotNull = new Not(new IsNull(scan.getOutput().get(1)),
true);
+ LogicalFilter<?> original = new
LogicalFilter<>(ImmutableSet.of(bNotNull), scan);
+ LogicalProject<?> project = new
LogicalProject<>(ImmutableList.copyOf(scan.getOutput()), original);
+
+ LogicalFilter<?> filter = PlanUtils.filter(ImmutableSet.of(aNotNull),
project).get();
+
+ Assertions.assertSame(project, filter.child());
+ Assertions.assertEquals(ImmutableSet.of(aNotNull),
filter.getConjuncts());
+ Assertions.assertSame(original, project.child());
+ }
+
@Test
void filterOrSelf() {
LogicalOlapScan scan = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]