This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 220b3623058 [fix](ivm) Reject a mark join on the append-only detail
IVM paths (#69211)
220b3623058 is described below
commit 220b3623058a418033c6aa789baeadb4f02e4316
Author: yujun <[email protected]>
AuthorDate: Mon Oct 19 15:22:36 2026 +0800
[fix](ivm) Reject a mark join on the append-only detail IVM paths (#69211)
The append-only detail (DUP) IVM model maintains only inner and cross
joins, but it tested the join type alone. A join's mark is orthogonal to
its type: an apply rewrite puts a mark on a `LEFT_SEMI`, a `LEFT_ANTI`,
a `NULL_AWARE_LEFT_ANTI`, or even a `CROSS` join (`ExistsApplyToJoin` /
`ScalarApplyToJoin`), so an `EXISTS` that lands on a `CROSS` join passes
a type-only test. A mark join can stop matching a row the MV already
holds, so no append can express the refresh, yet the detail normalize
and the detail delta rewriter both accepted it while the unique-key
model rejected the same query.
Reproduction: on a DUP-key base table, create an IVM whose query
contains a `NOT IN` / `EXISTS` subquery that the optimizer rewrites to a
mark join whose type is `CROSS_JOIN`; before the fix the detail model
accepted it, and the refresh could not maintain it.
Root cause: `IvmNormalizeDupKeysMTMV#visitLogicalJoin` and
`IvmDupDeltaRewriter#visitLogicalJoin` checked only
`join.getJoinType().isInnerOrCrossJoin()`, with no `join.isMarkJoin()`
gate; `IvmNormalizeUniqKeysMTMV` already had both gates.
---
.../main/java/org/apache/doris/mtmv/ivm/AGENTS.md | 14 +++
.../apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java | 10 +-
.../doris/mtmv/ivm/IvmNormalizeCommonChecker.java | 132 ++++++++++++++++++++
.../doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java | 31 ++---
.../doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java | 31 +----
.../java/org/apache/doris/mtmv/ivm/IvmUtil.java | 19 ---
.../doris/mtmv/ivm/IvmDupDeltaRewriterTest.java | 28 +++++
.../mtmv/ivm/IvmNormalizeCommonCheckerTest.java | 133 +++++++++++++++++++++
8 files changed, 324 insertions(+), 74 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/AGENTS.md
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/AGENTS.md
index 8b0a0ea1953..62bdb646f20 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/AGENTS.md
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/AGENTS.md
@@ -137,6 +137,20 @@ side. Only the legacy fallback for a DUP_KEYS table
without row binlog -- the no
trigger table -- is still rejected on the retained side. FULL OUTER JOIN
requires deterministic
row-ids on both sides.
+## Shared Acceptance Gates
+
+Both normalize paths and the append-only detail delta rewriter accept the same
query shapes, and the
+gates that reject the rest live in one place, `IvmNormalizeCommonChecker` (a
DISTINCT project, a
+UNION DISTINCT or constant-arm UNION ALL, a reserved output column name, an
unsupported node type).
+Add a new acceptance gate there rather than to a single class, or the paths
drift apart on what they
+accept -- which is how the join gate below went unshared.
+
+The join gate carries a per-model accepted set (the detail model maintains
only inner and cross joins,
+the unique-key model also the outer joins) but is otherwise shared, and it
tests the mark-join mark as
+well as the join type. The mark is orthogonal to the type -- an apply rewrite
puts one on a LEFT_SEMI,
+a LEFT_ANTI, or a CROSS join -- so a type-only test accepts a mark join whose
type happens to be an
+accepted one, and a mark join can stop matching a row the MV already holds.
+
## Recursive Delta Rewrite
- `IvmNormalizeUniqKeysMTMV` rejects an input `LogicalOlapTableStreamScan`;
normalized MV queries start
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java
index a494817d969..aea39b69f79 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriter.java
@@ -155,12 +155,10 @@ class IvmDupDeltaRewriter extends
PlanVisitor<Optional<Plan>, Void> {
@Override
public Optional<Plan> visitLogicalJoin(LogicalJoin<? extends Plan, ?
extends Plan> join, Void context) {
- if (!join.getJoinType().isInnerOrCrossJoin()) {
- // An outer join retracts the rows its null side stopped matching,
and the normalizer rejects
- // that shape before any refresh runs.
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM append-only detail MV delta rewrite does not support:
" + join.getJoinType());
- }
+ // An outer join retracts the rows its null side stopped matching, and
a mark join reaches those
+ // rows through the mark; the shared gate rejects both, so this
rewriter accepts exactly what the
+ // detail normalize accepts.
+ IvmNormalizeCommonChecker.checkJoin(join,
IvmNormalizeCommonChecker.DETAIL_JOIN_TYPES);
List<Plan> contributions = new ArrayList<>(join.children().size());
Optional<Plan> leftDelta = join.left().accept(this, context);
if (leftDelta.isPresent()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonChecker.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonChecker.java
new file mode 100644
index 00000000000..f5a989e19c3
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonChecker.java
@@ -0,0 +1,132 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv.ivm;
+
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalResultSink;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+
+import com.google.common.collect.ImmutableSet;
+
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * The query shapes an IVM MV cannot maintain, checked once for every
normalize.
+ *
+ * <p>Each shape here has to be rejected while a query is normalized, and the
reason is the same
+ * whichever model runs the normalize: a DISTINCT project folds rows together,
a UNION DISTINCT or a
+ * UNION ALL with constant arms reaches rows the MV does not hold as the
query's own, and a column name
+ * Doris reserves for a hidden one would leave the MV holding fewer columns
than the query produced.
+ * Keeping each such gate here rather than in a normalizer is what stops the
two normalize paths -- the
+ * unique-key one and the append-only detail one, plus the delta rewriter that
accepts the same shapes the
+ * detail normalize does -- from drifting apart on what they accept.
+ *
+ * <p>The join gate is the one gate that carries a per-model policy: how many
join types a model can
+ * maintain differs. An append-only detail MV maintains only {@link
#DETAIL_JOIN_TYPES}, a unique-key MV
+ * also maintains the outer joins ({@link #UNIQ_JOIN_TYPES}). The accepted set
is therefore the caller's,
+ * but the gate as a whole is shared, because a mark join's mark is orthogonal
to its type: the apply
+ * rewrites put one on a LEFT_SEMI, a LEFT_ANTI, a NULL_AWARE_LEFT_ANTI, or
even a CROSS join, so a test
+ * of the join type alone does not reject it. A mark join reaches rows the MV
does not hold as the query's
+ * own, exactly as an outer join does, so it must be rejected by name before
the type is looked at.
+ *
+ * <p>The class holds no state: every gate is a pure check that throws when it
rejects.
+ */
+final class IvmNormalizeCommonChecker {
+
+ /** Join types an append-only detail (DUP) MV can maintain; see {@link
IvmNormalizeDupKeysMTMV}. */
+ static final Set<JoinType> DETAIL_JOIN_TYPES =
+ ImmutableSet.of(JoinType.INNER_JOIN, JoinType.CROSS_JOIN);
+
+ /** Join types a unique-key MV can maintain; see {@link
IvmNormalizeUniqKeysMTMV}. */
+ static final Set<JoinType> UNIQ_JOIN_TYPES =
+ ImmutableSet.of(JoinType.INNER_JOIN, JoinType.CROSS_JOIN,
+ JoinType.LEFT_OUTER_JOIN, JoinType.RIGHT_OUTER_JOIN,
JoinType.FULL_OUTER_JOIN);
+
+ private IvmNormalizeCommonChecker() {
+ }
+
+ /**
+ * The rejection every normalize's {@code visit} default throws: a node
type no model accepts lands
+ * there, and the caller throws what this returns.
+ */
+ static IvmException unsupportedNode(Plan plan) {
+ return new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support plan node: " +
plan.getClass().getSimpleName());
+ }
+
+ static void checkProject(LogicalProject<? extends Plan> project) {
+ if (project.isDistinct()) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support DISTINCT project.");
+ }
+ }
+
+ static void checkUnion(LogicalUnion union) {
+ if (union.getQualifier() != Qualifier.ALL) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support UNION DISTINCT. Only UNION ALL is
supported.");
+ }
+ if (!union.getConstantExprsList().isEmpty()) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support UNION ALL with constant
expressions.");
+ }
+ }
+
+ /**
+ * Rejects an MV output column whose name a Doris hidden column already
owns. Both models lay hidden
+ * columns on the MV they build -- the unique-key model its row-id, any
model that enables row binlog
+ * its own commit tso and row lsn -- so a user column taking a reserved
prefix would be filtered out of
+ * the schema the user reads, leaving an MV that holds fewer columns than
its query produced.
+ */
+ static void checkResultSinkOutputNames(LogicalResultSink<? extends Plan>
sink) {
+ for (NamedExpression output : sink.getOutputExprs()) {
+ try {
+ FeNameFormat.checkColumnName(output.getName());
+ } catch (AnalysisException e) {
+ throw new
IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED, e.getMessage(), e);
+ }
+ }
+ }
+
+ /**
+ * Rejects a join no model with {@code allowedJoinTypes} can maintain. A
mark join is rejected before
+ * its type is looked at: it passes a type test whenever its type happens
to be an accepted one (a
+ * CROSS join carrying an EXISTS, say), which is exactly the hole a
type-only check leaves.
+ */
+ static void checkJoin(LogicalJoin<? extends Plan, ? extends Plan> join,
Set<JoinType> allowedJoinTypes) {
+ if (join.isMarkJoin()) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support a mark join (a subquery with
disjunction): the mark can drop "
+ + "a row the MV already holds.");
+ }
+ if (!allowedJoinTypes.contains(join.getJoinType())) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM does not support join type " + join.getJoinType() +
". Only "
+ +
allowedJoinTypes.stream().map(JoinType::toString).collect(Collectors.joining(",
"))
+ + " are supported.");
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java
index 5b9d6ce0c52..385cba08aec 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeDupKeysMTMV.java
@@ -22,7 +22,6 @@ import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.info.TableNameInfo;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
@@ -73,8 +72,7 @@ public class IvmNormalizeDupKeysMTMV extends
DefaultPlanRewriter<Set<TableNameIn
public Plan visit(Plan plan, Set<TableNameInfo> excludedTriggerTables) {
// Every node type not accepted below lands here, which is where the
unique-key normalize
// rejects it too: a detail MV's query may hold no other shape.
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support plan node: " +
plan.getClass().getSimpleName());
+ throw IvmNormalizeCommonChecker.unsupportedNode(plan);
}
@Override
@@ -102,10 +100,7 @@ public class IvmNormalizeDupKeysMTMV extends
DefaultPlanRewriter<Set<TableNameIn
@Override
public Plan visitLogicalProject(LogicalProject<? extends Plan> project,
Set<TableNameInfo> excludedTriggerTables) {
- if (project.isDistinct()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support DISTINCT project.");
- }
+ IvmNormalizeCommonChecker.checkProject(project);
return visitChildren(this, project, excludedTriggerTables);
}
@@ -124,26 +119,16 @@ public class IvmNormalizeDupKeysMTMV extends
DefaultPlanRewriter<Set<TableNameIn
@Override
public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan>
join,
Set<TableNameInfo> excludedTriggerTables) {
- if (!join.getJoinType().isInnerOrCrossJoin()) {
- // An outer join retracts the rows its null side stopped matching,
which means reaching
- // rows the MV already holds.
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM detail MV error: a detail MV can only maintain
INNER_JOIN and CROSS_JOIN, "
- + "but the query has " + join.getJoinType());
- }
+ // An outer join retracts the rows its null side stopped matching, and
a mark join reaches the
+ // same rows through the mark: both mean reaching rows the MV already
holds, so the shared gate
+ // rejects them by name as well as by type.
+ IvmNormalizeCommonChecker.checkJoin(join,
IvmNormalizeCommonChecker.DETAIL_JOIN_TYPES);
return visitChildren(this, join, excludedTriggerTables);
}
@Override
public Plan visitLogicalUnion(LogicalUnion union, Set<TableNameInfo>
excludedTriggerTables) {
- if (union.getQualifier() != Qualifier.ALL) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support UNION DISTINCT. Only UNION ALL is
supported.");
- }
- if (!union.getConstantExprsList().isEmpty()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support UNION ALL with constant
expressions.");
- }
+ IvmNormalizeCommonChecker.checkUnion(union);
return visitChildren(this, union, excludedTriggerTables);
}
@@ -153,7 +138,7 @@ public class IvmNormalizeDupKeysMTMV extends
DefaultPlanRewriter<Set<TableNameIn
// The MV is built with the query's own columns, so a column name
Doris reserves for a hidden one
// would leave the MV holding fewer columns than the query produced;
the unique-key normalize
// rejects the same names, and both models lay hidden columns on the
MV they build.
- IvmUtil.validateUserOutputColumnNames(sink.getOutputExprs());
+ IvmNormalizeCommonChecker.checkResultSinkOutputNames(sink);
return visitChildren(this, sink, excludedTriggerTables);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
index 36ed9203892..b0eaf93b59e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
@@ -47,7 +47,6 @@ import
org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier;
import org.apache.doris.nereids.trees.plans.commands.info.ColumnDefinition;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
@@ -278,8 +277,7 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
// unsupported: any plan node not explicitly whitelisted below
@Override
public Plan visit(Plan plan, NormalizeContext context) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED, "IVM
does not support plan node: "
- + plan.getClass().getSimpleName());
+ throw IvmNormalizeCommonChecker.unsupportedNode(plan);
}
// whitelisted: only OlapScan — inject IVM row-id at index 0
@@ -349,10 +347,7 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
// whitelisted: project — recurse into child, then propagate row-id if not
already present
@Override
public Plan visitLogicalProject(LogicalProject<? extends Plan> project,
NormalizeContext context) {
- if (project.isDistinct()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support DISTINCT project.");
- }
+ IvmNormalizeCommonChecker.checkProject(project);
Plan newChild = project.child().accept(this, context);
List<NamedExpression> baseOutputs = materializeDroppedAggState(
rewriteOutputsWithIvmHiddenColumns(newChild,
project.getProjects(), context.isFirstNonSink));
@@ -465,16 +460,7 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
@Override
public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan>
join, NormalizeContext context) {
JoinType joinType = join.getJoinType();
- if (!joinType.isInnerOrCrossJoin() && !joinType.isOuterJoin()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support join type: " + joinType
- + ". Only INNER_JOIN, CROSS_JOIN, LEFT_OUTER_JOIN,
RIGHT_OUTER_JOIN"
- + " and FULL_OUTER_JOIN are supported.");
- }
- if (join.isMarkJoin()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support mark join (subquery with
disjunction).");
- }
+ IvmNormalizeCommonChecker.checkJoin(join,
IvmNormalizeCommonChecker.UNIQ_JOIN_TYPES);
NormalizeContext childContext = context.enterJoin();
Plan newLeft = join.left().accept(this, childContext);
Plan newRight = join.right().accept(this, childContext);
@@ -599,14 +585,7 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
*/
@Override
public Plan visitLogicalUnion(LogicalUnion union, NormalizeContext
context) {
- if (union.getQualifier() != Qualifier.ALL) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support UNION DISTINCT. Only UNION ALL is
supported.");
- }
- if (!union.getConstantExprsList().isEmpty()) {
- throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM does not support UNION ALL with constant
expressions.");
- }
+ IvmNormalizeCommonChecker.checkUnion(union);
NormalizeContext childContext = context.afterNonSink();
boolean useUnionKeys = useFullKeys && !context.isInsideAggregate;
@@ -929,7 +908,7 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
// whitelisted: result sink — recurse into child, then prepend row-id to
output exprs
@Override
public Plan visitLogicalResultSink(LogicalResultSink<? extends Plan> sink,
NormalizeContext context) {
- IvmUtil.validateUserOutputColumnNames(sink.getOutputExprs());
+ IvmNormalizeCommonChecker.checkResultSinkOutputNames(sink);
Plan newChild = sink.child().accept(this, context);
List<NamedExpression> baseOutputs = materializeDroppedAggState(
rewriteOutputsWithIvmHiddenColumns(newChild,
sink.getOutputExprs(), context.isFirstNonSink));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmUtil.java
index dc602278819..f39db09eb21 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmUtil.java
@@ -25,7 +25,6 @@ import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.info.TableNameInfo;
import org.apache.doris.catalog.stream.BaseTableStream;
import org.apache.doris.catalog.stream.OlapTableStream;
-import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.info.TableNameInfoUtils;
@@ -33,7 +32,6 @@ import org.apache.doris.mtmv.MTMVPartitionUtil;
import org.apache.doris.nereids.trees.expressions.Cast;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.IsNull;
-import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.functions.scalar.BitmapEmpty;
import
org.apache.doris.nereids.trees.expressions.functions.scalar.MurmurHash3128;
@@ -389,21 +387,4 @@ public class IvmUtil {
+ table.getName() + ". The base MTMV is refreshed
by partition overwrite.");
}
}
-
- /**
- * Rejects an MV output column whose name a Doris hidden column already
owns. Both table models lay
- * hidden columns on the MV they build -- the unique-key model its row-id,
any model that enables row
- * binlog its own commit tso and row lsn -- so a user column taking one of
the reserved prefixes would
- * be indistinguishable from an internal one: it is filtered out of the
schema the user reads, leaving
- * an MV that holds fewer columns than its query produced.
- */
- public static void validateUserOutputColumnNames(List<NamedExpression>
outputs) {
- for (NamedExpression output : outputs) {
- try {
- FeNameFormat.checkColumnName(output.getName());
- } catch (org.apache.doris.common.AnalysisException e) {
- throw new
IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED, e.getMessage(), e);
- }
- }
- }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriterTest.java
index 658575a912a..d3d0c822daa 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmDupDeltaRewriterTest.java
@@ -22,10 +22,13 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.nereids.hint.DistributeHint;
import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.DistributeType;
import org.apache.doris.nereids.trees.plans.JoinType;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
@@ -34,6 +37,7 @@ import
org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalOlapTableStreamScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+import org.apache.doris.nereids.util.ExpressionUtils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -44,6 +48,7 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
+import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@@ -170,6 +175,17 @@ class IvmDupDeltaRewriterTest extends IvmDeltaTestBase {
Assertions.assertThrows(IvmException.class, () ->
generateDelta(buildOuterJoin(scan, scan), mtmv));
}
+ @Test
+ void testMarkJoinIsRejected() {
+ LogicalOlapScan scan = buildScanForTable(712, "dup_mark_reject");
+ Plan query = buildScanPlan(scan).child();
+ MTMV mtmv = dupMtmvWithPendingRows(query, 712_001L,
ImmutableList.of(scan.getTable()));
+
+ // A mark join's mark can drop a row the MV already holds, so a
refresh cannot express it by
+ // appending, and the shared join gate the detail normalize uses
rejects it here as well.
+ Assertions.assertThrows(IvmException.class, () ->
generateDelta(buildMarkJoin(scan, scan), mtmv));
+ }
+
private Plan generateDelta(Plan query, MTMV mtmv) {
ConnectContext connectContext = newConnectContext();
IvmRewriteResult rewriteResult = new IvmRewriteResult();
@@ -221,6 +237,18 @@ class IvmDupDeltaRewriterTest extends IvmDeltaTestBase {
leftArm, rightArm, JoinReorderContext.EMPTY);
}
+ private Plan buildMarkJoin(LogicalOlapScan leftScan, LogicalOlapScan
rightScan) {
+ LogicalProject<LogicalOlapScan> leftArm = new LogicalProject<>(
+ ImmutableList.copyOf(leftScan.getOutput()), leftScan);
+ LogicalProject<LogicalOlapScan> rightArm = new LogicalProject<>(
+ ImmutableList.copyOf(rightScan.getOutput()), rightScan);
+ return new LogicalJoin<>(JoinType.INNER_JOIN,
+ ImmutableList.of(), ExpressionUtils.EMPTY_CONDITION,
+ new DistributeHint(DistributeType.NONE),
+ Optional.of(new MarkJoinSlotReference("$mark")),
+ leftArm, rightArm, JoinReorderContext.EMPTY);
+ }
+
private Plan buildUnionAll(LogicalOlapScan firstScan, LogicalOlapScan
secondScan) {
Plan firstArm = buildScanPlan(firstScan).child();
Plan secondArm = new LogicalProject<>(ImmutableList.of(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonCheckerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonCheckerTest.java
new file mode 100644
index 00000000000..1cc46c2139d
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeCommonCheckerTest.java
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.mtmv.ivm;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.KeysType;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.DistributeType;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalResultSink;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.PlanConstructor;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.function.Executable;
+
+import java.util.Optional;
+
+/**
+ * Tests the join gate of {@link IvmNormalizeCommonChecker}, which is the one
gate the unique-key and
+ * append-only detail models share with a per-model accepted set, and that the
models reach it.
+ */
+class IvmNormalizeCommonCheckerTest extends IvmDeltaTestBase {
+
+ private LogicalOlapScan buildDupScan(long tableId, String name) {
+ OlapTable table = PlanConstructor.newOlapTable(tableId, name, 0,
KeysType.DUP_KEYS);
+ enableRowBinlog(table);
+ table.setQualifiedDbName("test_db");
+ return new LogicalOlapScan(PlanConstructor.getNextRelationId(), table,
ImmutableList.of("test_db"));
+ }
+
+ private LogicalJoin<?, ?> crossJoin(LogicalOlapScan left, LogicalOlapScan
right) {
+ return new LogicalJoin<>(JoinType.CROSS_JOIN, left, right,
JoinReorderContext.EMPTY);
+ }
+
+ private LogicalJoin<?, ?> semiJoin(LogicalOlapScan left, LogicalOlapScan
right) {
+ return new LogicalJoin<>(JoinType.LEFT_SEMI_JOIN,
+ ImmutableList.of(new EqualTo(left.getOutput().get(0),
right.getOutput().get(0))),
+ left, right, JoinReorderContext.EMPTY);
+ }
+
+ private LogicalJoin<?, ?> markJoin(JoinType joinType, LogicalOlapScan
left, LogicalOlapScan right) {
+ return new LogicalJoin<>(joinType,
+ ImmutableList.of(), ExpressionUtils.EMPTY_CONDITION,
+ new DistributeHint(DistributeType.NONE),
+ Optional.of(new MarkJoinSlotReference("$mark")),
+ left, right, JoinReorderContext.EMPTY);
+ }
+
+ @Test
+ void testAcceptedJoinTypePasses() {
+ LogicalOlapScan left = buildDupScan(1, "a");
+ LogicalOlapScan right = buildDupScan(2, "b");
+
+ Assertions.assertDoesNotThrow(() ->
IvmNormalizeCommonChecker.checkJoin(crossJoin(left, right),
+ IvmNormalizeCommonChecker.DETAIL_JOIN_TYPES));
+ }
+
+ @Test
+ void testUnacceptedJoinTypeIsRejected() {
+ LogicalOlapScan left = buildDupScan(1, "a");
+ LogicalOlapScan right = buildDupScan(2, "b");
+
+ assertPlanPatternUnsupported(() ->
IvmNormalizeCommonChecker.checkJoin(semiJoin(left, right),
+ IvmNormalizeCommonChecker.DETAIL_JOIN_TYPES));
+ }
+
+ @Test
+ void testMarkJoinOfAcceptedTypeIsRejected() {
+ LogicalOlapScan left = buildDupScan(1, "a");
+ LogicalOlapScan right = buildDupScan(2, "b");
+
+ // The mark is orthogonal to the join type: a CROSS join can carry
one, so a type test alone
+ // accepts it, and the mark would let the join reach rows the MV
already holds.
+ assertPlanPatternUnsupported(() -> IvmNormalizeCommonChecker.checkJoin(
+ markJoin(JoinType.CROSS_JOIN, left, right),
+ IvmNormalizeCommonChecker.DETAIL_JOIN_TYPES));
+ assertPlanPatternUnsupported(() -> IvmNormalizeCommonChecker.checkJoin(
+ markJoin(JoinType.INNER_JOIN, left, right),
+ IvmNormalizeCommonChecker.UNIQ_JOIN_TYPES));
+ }
+
+ @Test
+ void testDetailNormalizeRejectsMarkJoin() {
+ LogicalOlapScan left = buildDupScan(1, "a");
+ LogicalOlapScan right = buildDupScan(2, "b");
+
+ assertPlanPatternUnsupported(() -> new
IvmNormalizeDupKeysMTMV().normalize(
+ wrapForDupNormalize(markJoin(JoinType.CROSS_JOIN, left,
right)), ImmutableSet.of()));
+ }
+
+ private Plan wrapForDupNormalize(Plan plan) {
+ ImmutableList<NamedExpression> exprs = plan.getOutput().stream()
+ .filter(slot -> !Column.ROW_LSN_COL.equals(slot.getName()))
+ .filter(slot -> !(slot instanceof MarkJoinSlotReference))
+ .map(NamedExpression.class::cast)
+ .collect(ImmutableList.toImmutableList());
+ LogicalProject<?> project = new LogicalProject<>(exprs, plan);
+ return new LogicalResultSink<>(exprs, project);
+ }
+
+ private void assertPlanPatternUnsupported(Executable executable) {
+ IvmException exception = Assertions.assertThrows(IvmException.class,
executable);
+ Assertions.assertEquals(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
exception.getFailureReason());
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
