This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4f3abce2c25 [fix](ivm) Resolve IVM identity keys by slot identity and
materialize unprojected keys (#67669)
4f3abce2c25 is described below
commit 4f3abce2c25e7b1e3d24eeaa97cffcfd76abe531
Author: yujun <[email protected]>
AuthorDate: Thu Sep 10 11:13:13 2026 +0800
[fix](ivm) Resolve IVM identity keys by slot identity and materialize
unprojected keys (#67669)
Two follow-ups from the #62606 review round, both around same-named
identity keys of aggregate/full-keys MVs:
1. **Resolve IVM aggregate group keys by slot identity instead of name**
— an aggregate delta over a join of two tables that both expose the same
column name (`GROUP BY l.id, r.id`) resolved its group keys through
name-keyed lookups. A name index collapses same-named slots onto the
last match, so both keys were bound to one delta slot and a `(10, 20)`
group was written as `(20, 20)`, diverging from COMPLETE. The delta-side
group keys are now resolved and carried by slot identity (ordered with
the aggregate metadata), the apply project emits each key output from
its own delta slot, and the `ivm_use_full_keys` identity conjuncts
resolve their delta side by identity first. The full-keys regression
with a non-full-keys counterpart covers plain INSERT, full-keys, and MOW
UPDATE refresh, each cross-checked against COMPLETE.
2. **Materialize unprojected full-keys identity keys at the project
layer** — with `ivm_use_full_keys` every identity key must be
materialized in the stored layout, either as a visible output or under a
hidden key column. The result sink judged "already projected" by column
name, so a same-named key from another table (selecting only `l.id`
while grouping by `l.id, r.id`) was silently dropped and the key set
lost one dimension. The sink now checks projection by slot identity, and
materialization moves down to the first project that drops a key, so
CREATE (result sink) and refresh (olap-table sink) plans produce the
same hidden-key layout. A new regression asserts the hidden key column
exists (DESC with hidden columns) and that INCREMENTAL matches COMPLETE.
Tests: `IvmAggDeltaHandlerTest` (33) and `IvmNormalizeMTMVJoinTest` new
cases pass; the new regression cases pass and the full ivm regression
set (85 suites) is green on the materialization change.
Trace issue: https://github.com/apache/doris/issues/65418
---
.../apache/doris/mtmv/ivm/IvmAggDeltaHandler.java | 90 +++++--
.../doris/mtmv/ivm/IvmDeltaRewriteHelper.java | 14 ++
.../nereids/rules/analysis/IvmNormalizeMTMV.java | 44 +++-
.../doris/mtmv/ivm/IvmAggDeltaHandlerTest.java | 104 ++++++++
.../doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java | 67 ++++++
.../rules/analysis/IvmNormalizeMTMVTest.java | 15 +-
.../ivm/test_ivm_agg_join_same_name_group_key.out | 50 ++++
...est_ivm_full_keys_same_name_unprojected_key.out | 30 +++
.../test_ivm_agg_join_same_name_group_key.groovy | 263 +++++++++++++++++++++
..._ivm_full_keys_same_name_unprojected_key.groovy | 106 +++++++++
10 files changed, 750 insertions(+), 33 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandler.java
index 86b2d011f28..a72e073413f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandler.java
@@ -110,15 +110,26 @@ class IvmAggDeltaHandler {
private final Slot deltaGroupCountSlot;
/** Per-target delta slots consumed by aggregate function processors
during apply. */
private final Map<IvmAggDeltaSlotRef, Slot> applyDeltaSlots;
- /** Group key slots resolved from topDeltaProject output, keyed by
column name. */
+ /**
+ * Group key slots resolved from the topDeltaProject output, ordered
like
+ * {@link IvmAggMeta#getGroupKeySlots()}. Same-named keys (e.g. l.id
and r.id after
+ * a join) must be addressed by position/identity through this list,
never by name.
+ */
+ private final List<Slot> deltaGroupKeySlots;
+ /**
+ * Name-keyed view of {@link #deltaGroupKeySlots}; with same-named
group keys the map
+ * keeps only the last slot, so it is only usable when all key names
are distinct.
+ */
private final Map<String, Slot> groupKeySlotsByName;
private DeltaPlanParts(LogicalProject<?> topDeltaProject, Slot
rowIdSlot, Slot deltaGroupCountSlot,
- Map<IvmAggDeltaSlotRef, Slot> applyDeltaSlots, Map<String,
Slot> groupKeySlotsByName) {
+ Map<IvmAggDeltaSlotRef, Slot> applyDeltaSlots, List<Slot>
deltaGroupKeySlots,
+ Map<String, Slot> groupKeySlotsByName) {
this.topDeltaProject = topDeltaProject;
this.rowIdSlot = rowIdSlot;
this.deltaGroupCountSlot = deltaGroupCountSlot;
this.applyDeltaSlots = applyDeltaSlots;
+ this.deltaGroupKeySlots = deltaGroupKeySlots;
this.groupKeySlotsByName = groupKeySlotsByName;
}
}
@@ -237,19 +248,27 @@ class IvmAggDeltaHandler {
aggFunctionRegistry.mapApplyDeltaSlots(
target, outputByName, applyDeltaSlots,
deltaGroupCountSlot, aggExpressionBuilder);
}
- Map<String, Slot> groupKeySlotsByName = new LinkedHashMap<>();
+ // Group keys are resolved by slot identity, not by name: same-named
keys (e.g.
+ // l.id / r.id after a join) collapse to the last matching slot under
a name index.
+ List<Slot> deltaOutputSlots = topDeltaProject.getOutput();
+ List<Slot> deltaGroupKeySlots = new ArrayList<>(groupKeySize);
for (Slot groupKey : aggMeta.getGroupKeySlots()) {
- Slot resolved = outputByName.get(groupKey.getName());
+ Slot resolved = helper.findSlotByExprId(deltaOutputSlots,
groupKey);
if (resolved == null) {
throw new IvmException(IvmFailureReason.PLAN_REWRITE_FAILED,
"IVM agg delta rewrite failed to resolve delta group
key slot: "
- + groupKey.getName());
+ + groupKey);
}
- groupKeySlotsByName.put(groupKey.getName(), resolved);
+ deltaGroupKeySlots.add(resolved);
+ }
+ // Name-keyed view, kept only for lookups on distinct key names (see
field comment).
+ Map<String, Slot> groupKeySlotsByName = new LinkedHashMap<>();
+ for (Slot resolved : deltaGroupKeySlots) {
+ groupKeySlotsByName.put(resolved.getName(), resolved);
}
return new DeltaPlanParts(topDeltaProject,
outputByName.get(Column.IVM_ROW_ID_COL),
- deltaGroupCountSlot, applyDeltaSlots, groupKeySlotsByName);
+ deltaGroupCountSlot, applyDeltaSlots, deltaGroupKeySlots,
groupKeySlotsByName);
}
/**
@@ -304,9 +323,8 @@ class IvmAggDeltaHandler {
"negative group count");
finalByColumnName.put(Column.IVM_ROW_ID_COL, delta.rowIdSlot);
finalByColumnName.put(aggMeta.getGroupCountSlot().getName(),
newGroupCount);
- for (Slot groupKey : aggMeta.getGroupKeySlots()) {
- finalByColumnName.put(groupKey.getName(), deltaGroupKey(delta,
groupKey.getName()));
- }
+ // Group keys are emitted from the delta side below, resolved by slot
identity, and
+ // must not enter the name-keyed map: same-named keys would overwrite
each other.
Set<String> visibleColumnNames = new HashSet<>();
for (IvmAggTarget target : aggMeta.getAggTargets()) {
@@ -330,7 +348,13 @@ class IvmAggDeltaHandler {
// Keep the normalized aggregate schema here. The normalize-added top
project computes row-id above this
// project, and the final sink project reorders columns by MV schema.
for (Slot target : normalizedAgg.getOutput()) {
- Expression expr = finalByColumnName.get(target.getName());
+ // A group key output takes the delta-side key slot: a group that
is new in the
+ // delta has no MV row to copy from. Same-named keys are
disambiguated by slot
+ // identity against aggMeta's ordered group keys.
+ int groupKeyIndex = groupKeyIndexIn(target, aggMeta);
+ Expression expr = groupKeyIndex >= 0
+ ? delta.deltaGroupKeySlots.get(groupKeyIndex)
+ : finalByColumnName.get(target.getName());
if (expr == null) {
throw new IvmException(IvmFailureReason.PLAN_REWRITE_FAILED,
"IVM agg delta rewrite missing output expression for
column: "
@@ -350,11 +374,18 @@ class IvmAggDeltaHandler {
List<Slot> identityKeys = rewriteResult == null ? null :
rewriteResult.getIdentityKeySlots();
if (identityKeys != null) {
for (Slot identityKey : identityKeys) {
+ // The MV side is matched by column name (the MV key columns
are the
+ // normalized outputs). When an MV selects same-named group
keys under
+ // output aliases, the alias differs from the key slot name
and this lookup
+ // misses: the identity conjunct is skipped and the row-id
conjunct still
+ // guards the join, so no wrong value is produced. The delta
side below is
+ // always resolved by slot identity (findDeltaKeyByIdentity),
so a matched
+ // identity key can never be paired with the wrong delta slot.
Slot mvKey =
helper.findSlotByNameOrNull(rawMvScan.getOutput(), identityKey.getName());
if (mvKey == null) {
continue;
}
- Slot deltaKey = findDeltaKeyByName(delta,
identityKey.getName());
+ Slot deltaKey = findDeltaKeyByIdentity(delta, identityKey);
if (deltaKey == null) {
continue;
}
@@ -367,6 +398,21 @@ class IvmAggDeltaHandler {
return conjuncts.build();
}
+ /**
+ * Resolves the delta-side key slot for an identity key by slot identity
first (both
+ * live in the same normalized plan, so same-named keys such as l.id /
r.id match their
+ * own delta slot), falling back to the name-keyed lookup for legacy
callers whose keys
+ * have distinct names.
+ */
+ private Slot findDeltaKeyByIdentity(DeltaPlanParts delta, Slot
identityKey) {
+ for (Slot keySlot : delta.deltaGroupKeySlots) {
+ if (keySlot.getExprId().equals(identityKey.getExprId())) {
+ return keySlot;
+ }
+ }
+ return findDeltaKeyByName(delta, identityKey.getName());
+ }
+
private Slot findDeltaKeyByName(DeltaPlanParts delta, String name) {
Slot direct = delta.groupKeySlotsByName.get(name);
if (direct != null) {
@@ -449,15 +495,6 @@ class IvmAggDeltaHandler {
return delta.deltaGroupCountSlot;
}
- private Expression deltaGroupKey(DeltaPlanParts delta, String name) {
- Slot slot = delta.groupKeySlotsByName.get(name);
- if (slot == null) {
- throw new IvmException(IvmFailureReason.PLAN_REWRITE_FAILED,
- "IVM agg delta rewrite failed to resolve delta group key:
" + name);
- }
- return slot;
- }
-
/**
* Collects delta output names where NULL should be normalized to zero
before apply.
*
@@ -487,4 +524,15 @@ class IvmAggDeltaHandler {
return slotByName;
}
+ /** Index of {@code slot} among the aggregate group keys, or -1 when it is
not a group key. */
+ private int groupKeyIndexIn(Slot slot, IvmAggMeta aggMeta) {
+ List<Slot> groupKeys = aggMeta.getGroupKeySlots();
+ for (int i = 0; i < groupKeys.size(); i++) {
+ if (groupKeys.get(i).getExprId().equals(slot.getExprId())) {
+ return i;
+ }
+ }
+ return -1;
+ }
+
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriteHelper.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriteHelper.java
index 2ed7d14931e..7a6bc0f7558 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriteHelper.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriteHelper.java
@@ -81,6 +81,20 @@ public class IvmDeltaRewriteHelper {
return null;
}
+ /**
+ * Finds the slot in {@code slots} whose ExprId equals {@code target}'s,
or null. Slots
+ * that share a name (e.g. l.id / r.id after a join) must be resolved by
slot identity
+ * instead of a name-keyed lookup, which collapses same-named slots onto
one of them.
+ */
+ public Slot findSlotByExprId(List<Slot> slots, Slot target) {
+ for (Slot slot : slots) {
+ if (slot.getExprId().equals(target.getExprId())) {
+ return slot;
+ }
+ }
+ return null;
+ }
+
/**
* Adds the root-level fallback guard for a non-deterministic MV row-id.
Inserts remain valid;
* any delete delta fails so the caller can fall back to a full refresh.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
index 1c92e8017d2..9ce790d31e5 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
@@ -383,9 +383,19 @@ public class IvmNormalizeMTMV extends
DefaultPlanRewriter<IvmNormalizeMTMV.Norma
NamedExpression projected = findProjectedKey(baseOutputs,
keySlot);
if (projected != null) {
survivingKeys.add(projected.toSlot());
- } else {
- extendedOutputs.add(keySlot);
+ } else if (IvmUtil.isIvmHiddenColumn(keySlot.getName())) {
+ // Already materialized by a lower layer: keep the hidden
key slot as is.
survivingKeys.add(keySlot);
+ } else {
+ // An unprojected business key is materialized here as a
hidden key column
+ // instead of being passed through: the result sink
(CREATE) and the
+ // olap-table sink (refresh) then see the same hidden
layout and agree on
+ // the hidden column names. Keys are matched by slot
identity, so a
+ // same-named key from another table (e.g. r.id next to an
output l.id)
+ // still gets its own hidden column.
+ Alias hiddenAlias = materializeHiddenKey(keySlot);
+ extendedOutputs.add(hiddenAlias);
+ survivingKeys.add(hiddenAlias.toSlot());
}
}
finalOutputs = ImmutableList.copyOf(extendedOutputs);
@@ -416,6 +426,21 @@ public class IvmNormalizeMTMV extends
DefaultPlanRewriter<IvmNormalizeMTMV.Norma
return null;
}
+ /**
+ * Materializes an unprojected identity key under a unique hidden key
column name. The
+ * same helper is used by the project layer (where refresh and CREATE
plans first drop a
+ * key) and by the sink (direct-child fallback), so both paths agree on
hidden names.
+ *
+ * <p>Note: the returned alias owns a fresh ExprId, so a later delta
rewrite cannot match
+ * the hidden column to the original key slot by identity; it resolves it
through the
+ * hidden column name (see the agg handler's sanitized-name fallback).
+ */
+ private Alias materializeHiddenKey(Slot keySlot) {
+ String hiddenName = Column.IVM_KEY_COL_PREFIX + (++sinkKeyCounter) +
"_"
+ + IvmUtil.sanitizeIvmKeyName(keySlot.getName()) + "_COL__";
+ return new Alias(keySlot, hiddenName);
+ }
+
@Override
public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter,
NormalizeContext context) {
Plan result = filter.withChildren(child -> child.accept(this,
context.afterNonSink()));
@@ -927,19 +952,20 @@ public class IvmNormalizeMTMV extends
DefaultPlanRewriter<IvmNormalizeMTMV.Norma
List<Slot> sinkKeys = new ArrayList<>();
List<NamedExpression> finalOutputs = new ArrayList<>(baseOutputs);
if (childKeys != null && !childKeys.isEmpty()) {
- Set<String> outputNames = finalOutputs.stream()
- .map(NamedExpression::getName)
- .collect(Collectors.toSet());
for (Slot keySlot : childKeys) {
String keyName = keySlot.getName();
if (IvmUtil.isIvmHiddenColumn(keyName)) {
sinkKeys.add(keySlot);
- } else if (outputNames.contains(keyName)) {
+ } else if (findProjectedKey(finalOutputs, keySlot) != null) {
+ // The key is already materialized by an output that
directly emits the
+ // key slot (bare slot or alias over it). Matching by slot
identity -
+ // as the project layer does - instead of by name is
required: a
+ // same-named key from another table (e.g. r.id next to an
output l.id)
+ // must still be materialized under its own hidden column,
or the
+ // full-keys identity set silently loses one dimension.
sinkKeys.add(keySlot);
} else {
- String hiddenName = Column.IVM_KEY_COL_PREFIX +
(++sinkKeyCounter) + "_"
- + IvmUtil.sanitizeIvmKeyName(keyName) + "_COL__";
- Alias hiddenAlias = new Alias(keySlot, hiddenName);
+ Alias hiddenAlias = materializeHiddenKey(keySlot);
finalOutputs.add(hiddenAlias);
sinkKeys.add(hiddenAlias.toSlot());
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandlerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandlerTest.java
index 9f6fb5c5b89..44b2597e3c3 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandlerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmAggDeltaHandlerTest.java
@@ -22,7 +22,10 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.nereids.analyzer.UnboundTableSink;
+import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext;
import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.expressions.Not;
@@ -59,6 +62,7 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
+import java.util.Set;
import java.util.stream.Collectors;
class IvmAggDeltaHandlerTest extends IvmDeltaTestBase {
@@ -734,6 +738,106 @@ class IvmAggDeltaHandlerTest extends IvmDeltaTestBase {
"MIN/MAX with expression args should still have assert_true
guards");
}
+ @Test
+ void testGroupedAggWithSameNamedGroupKeysKeepsDistinctDeltaSlots() {
+ // GROUP BY l.id, r.id over a join of two tables that both expose
"id": the
+ // same-named group keys must keep their identity through the delta
rewrite,
+ // otherwise both keys collapse onto one slot and a (10, 20) group
becomes
+ // (20, 20).
+ LogicalOlapScan leftScan = buildScanForTable(1L, "l");
+ LogicalOlapScan rightScan = buildScanForTable(2L, "r");
+ Slot leftId = leftScan.getOutput().get(0);
+ Slot rightId = rightScan.getOutput().get(0);
+ Assertions.assertEquals(leftId.getName(), rightId.getName(),
+ "test setup expects same-named key columns on both base
tables");
+ Assertions.assertNotEquals(leftId.getExprId(), rightId.getExprId());
+
+ LogicalJoin<LogicalOlapScan, LogicalOlapScan> join = new LogicalJoin<>(
+ JoinType.INNER_JOIN, ImmutableList.of(new EqualTo(leftId,
rightId)),
+ leftScan, rightScan, JoinReorderContext.EMPTY);
+ Alias countAlias = new Alias(new Count(), "cnt");
+ LogicalAggregate<LogicalJoin<LogicalOlapScan, LogicalOlapScan>> agg =
new LogicalAggregate<>(
+ ImmutableList.of(leftId, rightId),
+ ImmutableList.of(leftId, rightId, countAlias),
+ true, Optional.empty(), join);
+
+ AggRewriteResult result = rewriteAgg(agg);
+ List<Slot> aggGroupKeys =
result.bundle.rewriteResult.getAggMeta().getGroupKeySlots();
+ Assertions.assertEquals(2, aggGroupKeys.size());
+ Assertions.assertEquals(leftId.getName(),
aggGroupKeys.get(0).getName());
+ Assertions.assertEquals(leftId.getName(),
aggGroupKeys.get(1).getName());
+
+ // Apply project must emit each group key from its own delta-side slot
(the apply
+ // project output order follows the aggregate outputs, keys first).
+ List<Expression> applyKeyExprs =
getApplyProject(result).getProjects().stream()
+ .filter(projection ->
leftId.getName().equals(projection.getName()))
+ .map(projection -> projection instanceof Alias ? ((Alias)
projection).child() : projection)
+ .collect(Collectors.toList());
+ Assertions.assertEquals(2, applyKeyExprs.size(), "expected two
same-named group key outputs");
+ Assertions.assertInstanceOf(Slot.class, applyKeyExprs.get(0));
+ Assertions.assertInstanceOf(Slot.class, applyKeyExprs.get(1));
+ Assertions.assertNotEquals(((Slot) applyKeyExprs.get(0)).getExprId(),
+ ((Slot) applyKeyExprs.get(1)).getExprId(),
+ "same-named group keys must resolve to distinct delta slots");
+
+ // The apply keys must be exactly the delta group key slots in aggMeta
order. The
+ // delta top project output is laid out as [row_id, group keys...,
other outputs]
+ // (the row-id hash relies on the same layout), so group key i sits at
index 1 + i.
+ LogicalProject<?> deltaTop = getDeltaTopProject(result);
+ Assertions.assertEquals(((Slot) applyKeyExprs.get(0)).getExprId(),
+ deltaTop.getOutput().get(1).getExprId());
+ Assertions.assertEquals(((Slot) applyKeyExprs.get(1)).getExprId(),
+ deltaTop.getOutput().get(2).getExprId());
+ }
+
+ @Test
+ void testGroupedAggFullKeysWithSameNamedGroupKeysKeepsDistinctDeltaSlots()
{
+ // With ivm_use_full_keys the MV key is row_id + group by keys; the
same-named
+ // l.id / r.id identity keys must each resolve to their own delta slot
in the
+ // apply-join identity conjuncts, not both to the last same-named slot.
+ LogicalOlapScan leftScan = buildScanForTable(1L, "l");
+ LogicalOlapScan rightScan = buildScanForTable(2L, "r");
+ Slot leftId = leftScan.getOutput().get(0);
+ Slot rightId = rightScan.getOutput().get(0);
+ LogicalJoin<LogicalOlapScan, LogicalOlapScan> joinInput = new
LogicalJoin<>(
+ JoinType.INNER_JOIN, ImmutableList.of(new EqualTo(leftId,
rightId)),
+ leftScan, rightScan, JoinReorderContext.EMPTY);
+ Alias countAlias = new Alias(new Count(), "cnt");
+ LogicalAggregate<LogicalJoin<LogicalOlapScan, LogicalOlapScan>> agg =
new LogicalAggregate<>(
+ ImmutableList.of(leftId, rightId),
+ ImmutableList.of(leftId, rightId, countAlias),
+ true, Optional.empty(), joinInput);
+
+ AggRewriteResult result = rewriteAggWithIdentityKeys(agg,
+ ImmutableList.of(leftId, rightId));
+ LogicalJoin<?, ?> join = getJoin(result);
+
+ LogicalProject<?> deltaTop = getDeltaTopProject(result);
+ List<Slot> deltaKeySlots =
ImmutableList.of(deltaTop.getOutput().get(1),
+ deltaTop.getOutput().get(2));
+
+ // Identity conjuncts are the NullSafeEquals that are not the row-id
conjunct;
+ // each same-named identity key must be joined against its own delta
key slot.
+ List<Expression> identityConjuncts =
join.getHashJoinConjuncts().stream()
+ .filter(condition -> condition instanceof NullSafeEqual)
+ .filter(condition ->
!condition.toSql().contains(Column.IVM_ROW_ID_COL))
+ .collect(Collectors.toList());
+ Assertions.assertEquals(2, identityConjuncts.size(),
+ "expected one identity conjunct per group key, but got: " +
identityConjuncts);
+ Set<ExprId> deltaSides = identityConjuncts.stream()
+ .map(conjunct -> {
+ NullSafeEqual equal = (NullSafeEqual) conjunct;
+ Expression right = equal.right();
+ return right instanceof Slot ? ((Slot) right).getExprId()
+ : ((Slot) equal.left()).getExprId();
+ })
+ .collect(Collectors.toSet());
+ Assertions.assertEquals(2, deltaSides.size(),
+ "same-named identity keys must map to distinct delta key
slots");
+
Assertions.assertTrue(deltaSides.contains(deltaKeySlots.get(0).getExprId()));
+
Assertions.assertTrue(deltaSides.contains(deltaKeySlots.get(1).getExprId()));
+ }
+
private static final class AggRewriteResult {
private final PlanBundle bundle;
private final MTMV mtmv;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java
index 6f4e26b820c..0062b1f99ae 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeMTMVJoinTest.java
@@ -19,6 +19,7 @@ package org.apache.doris.mtmv.ivm;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.KeysType;
+import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.info.TableNameInfo;
import org.apache.doris.nereids.hint.DistributeHint;
@@ -59,6 +60,7 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.function.Executable;
+import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -1024,4 +1026,69 @@ class IvmNormalizeMTMVJoinTest extends IvmDeltaTestBase {
Assertions.assertEquals(JOIN_RIGHT_MATCH_COL,
slotOfIsNullEncoding(hashChildren.get(5)).getName(),
"last outer compose key is the outer right match flag");
}
+
+ @Test
+ void testFullKeysSinkMaterializesSameNamedUnprojectedKey() {
+ // SELECT l.id FROM l JOIN r ON l.id = r.id with ivm_use_full_keys:
the right id is
+ // an identity key that is not projected by the user output. It shares
its name with
+ // the projected left id, so a name-keyed "already projected" test at
the sink would
+ // drop it; the sink must materialize it under its own hidden key
column instead.
+ LogicalOlapScan left = buildMowScan(1, "l");
+ LogicalOlapScan right = buildMowScan(2, "r");
+ Slot leftId = left.getOutput().get(0);
+ Slot rightId = right.getOutput().get(0);
+ Assertions.assertEquals(leftId.getName(), rightId.getName(),
+ "test setup expects same-named key columns on both base
tables");
+
+ LogicalJoin<LogicalOlapScan, LogicalOlapScan> join = new LogicalJoin<>(
+ JoinType.INNER_JOIN, ImmutableList.of(new EqualTo(leftId,
rightId)),
+ left, right, JoinReorderContext.EMPTY);
+ LogicalProject<Plan> project = new
LogicalProject<>(ImmutableList.of(leftId), join);
+ LogicalResultSink<Plan> sink = new
LogicalResultSink<>(ImmutableList.of(leftId), project);
+
+ MTMV mtmv = buildMtmvFromPlan(ImmutableList.of(leftId));
+ mtmv.getIvmInfo().setUseFullKeys(true);
+ ConnectContext ctx = newConnectContext();
+ ctx.getStatementContext().setIvmRewriteContext(Optional.of(
+ new IvmRewriteContext(IvmRewriteContext.Mode.CREATE, mtmv,
false)));
+ JobContext jobContext = newJobContextForRoot(sink, ctx);
+ Plan normalized = new IvmNormalizeMTMV().rewriteRoot(sink, jobContext);
+ IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().get();
+
+ List<Slot> identityKeys = rewriteResult.getIdentityKeySlots();
+ Assertions.assertNotNull(identityKeys);
+ Assertions.assertTrue(identityKeys.stream()
+ .anyMatch(key ->
key.getExprId().equals(leftId.getExprId())),
+ "the projected left id must stay an identity key, but got: " +
identityKeys);
+
+ String embeddedId = IvmUtil.sanitizeIvmKeyName(rightId.getName());
+ Slot rightHiddenKey = identityKeys.stream()
+ .filter(key -> IvmUtil.isIvmHiddenColumn(key.getName())
+ && key.getName().endsWith("_" + embeddedId + "_COL__"))
+ .findFirst().orElse(null);
+ Assertions.assertNotNull(rightHiddenKey,
+ "same-named right id key must be materialized as a hidden key
column, got: "
+ + identityKeys);
+
+ // The hidden key column is materialized at the first project that
does not project
+ // the key: find that alias in the normalized tree and verify it wraps
the right id
+ // slot rather than the same-named left id.
+ List<Alias> hiddenAliases = new ArrayList<>();
+ normalized.foreach(node -> {
+ if (node instanceof LogicalProject) {
+ for (NamedExpression projection : ((LogicalProject<?>)
node).getProjects()) {
+ if (projection instanceof Alias &&
projection.getName().equals(rightHiddenKey.getName())) {
+ hiddenAliases.add((Alias) projection);
+ }
+ }
+ }
+ });
+ Assertions.assertFalse(hiddenAliases.isEmpty(),
+ "expected a hidden key projection named " +
rightHiddenKey.getName());
+ for (Alias hiddenAlias : hiddenAliases) {
+ Assertions.assertInstanceOf(Slot.class, hiddenAlias.child());
+ Assertions.assertEquals(rightId.getExprId(), ((Slot)
hiddenAlias.child()).getExprId(),
+ "the hidden key column must wrap the right id slot, not
the same-named left id");
+ }
+ }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMVTest.java
index 34898b049b8..a64a223960c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMVTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMVTest.java
@@ -1271,12 +1271,21 @@ class IvmNormalizeMTMVTest {
ImmutableList.of(join.getOutput().get(0),
join.getOutput().get(1)), join);
JobContext jobContext = newJobContextWithFullKeys(sink);
- new IvmNormalizeMTMV().rewriteRoot(sink, jobContext);
+ Plan result = new IvmNormalizeMTMV().rewriteRoot(sink, jobContext);
IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().orElseThrow();
List<String> keyNames = rewriteResult.getIdentityKeySlots().stream()
.map(Slot::getName).collect(Collectors.toList());
- // Left table keys (id, name) followed by right table keys (id, name),
deduped by output presence.
- Assertions.assertEquals(ImmutableList.of("id", "name", "id", "name"),
keyNames);
+ // Left keys (id, name) are the projected outputs; the same-named
right keys are not
+ // projected, so each must be materialized under its own hidden key
column instead of
+ // being folded onto the same-named left outputs (which would silently
drop the right
+ // dimensions from the full-keys layout).
+ String rightIdKey = Column.IVM_KEY_COL_PREFIX + "1_id_COL__";
+ String rightNameKey = Column.IVM_KEY_COL_PREFIX + "2_name_COL__";
+ Assertions.assertEquals(ImmutableList.of("id", "name", rightIdKey,
rightNameKey), keyNames);
+ List<String> outputNames = result.getOutput().stream()
+ .map(Slot::getName).collect(Collectors.toList());
+ Assertions.assertTrue(outputNames.contains(rightIdKey));
+ Assertions.assertTrue(outputNames.contains(rightNameKey));
}
@Test
diff --git
a/regression-test/data/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.out
b/regression-test/data/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.out
new file mode 100644
index 00000000000..23626b2cec3
--- /dev/null
+++ b/regression-test/data/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.out
@@ -0,0 +1,50 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !same_key_complete --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+
+-- !same_key_incremental --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+4 40 400 1
+
+-- !same_key_complete2 --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+4 40 400 1
+
+-- !same_key_fk_complete --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+
+-- !same_key_fk_incremental --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+4 40 400 1
+
+-- !same_key_fk_complete2 --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+4 40 400 1
+
+-- !same_key_up_complete --
+1 10 100 1
+2 20 200 1
+3 30 300 1
+
+-- !same_key_up_incremental --
+1 50 50 1
+2 20 200 1
+3 30 300 1
+
+-- !same_key_up_complete2 --
+1 50 50 1
+2 20 200 1
+3 30 300 1
+
diff --git
a/regression-test/data/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.out
b/regression-test/data/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.out
new file mode 100644
index 00000000000..bb339f239c6
--- /dev/null
+++
b/regression-test/data/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.out
@@ -0,0 +1,30 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !fk_hidden_desc --
+lid int Yes true \N
+__DORIS_IVM_KEY_1_id_COL__ int Yes true \N
+__DORIS_IVM_ROW_ID_COL__ largeint No true \N
+total bigint Yes false \N NONE
+cnt bigint No false \N NONE
+__DORIS_IVM_AGG_COUNT_COL__ bigint No false \N NONE
+__DORIS_IVM_AGG_0_COUNT_COL__ bigint No false \N NONE
+__DORIS_DELETE_SIGN__ tinyint No false 0 NONE
+__DORIS_VERSION_COL__ bigint No false 0 NONE
+__DORIS_SEQUENCE_COL__ bigint Yes false \N NONE
+
+-- !fk_hidden_complete --
+1 100 1
+2 200 1
+3 300 1
+
+-- !fk_hidden_incremental --
+1 100 1
+2 200 1
+3 300 1
+4 400 1
+
+-- !fk_hidden_complete2 --
+1 100 1
+2 200 1
+3 300 1
+4 400 1
+
diff --git
a/regression-test/suites/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.groovy
new file mode 100644
index 00000000000..a67617d31a8
--- /dev/null
+++
b/regression-test/suites/mtmv_p0/ivm/test_ivm_agg_join_same_name_group_key.groovy
@@ -0,0 +1,263 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_ivm_agg_join_same_name_group_key") {
+
+ // GROUP BY l.id, r.id over a join of two tables that both expose a column
"id": the
+ // same-named group keys must keep their identity through the aggregate
delta rewrite.
+ // A name-keyed lookup collapses both keys onto one slot and turns group
(1,20) into
+ // (20,20), diverging from COMPLETE.
+
+ def mvName = "ivm_same_key_mv"
+ def lTable = "ivm_same_key_l"
+ def rTable = "ivm_same_key_r"
+
+ sql """drop materialized view if exists ${mvName}"""
+ sql """drop table if exists ${lTable}"""
+ sql """drop table if exists ${rTable}"""
+
+ // Both tables carry a column literally named "id"; the join key is k.
+ sql """
+ CREATE TABLE ${lTable} (
+ id INT,
+ k INT,
+ v INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """
+ CREATE TABLE ${rTable} (
+ id INT,
+ k INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """INSERT INTO ${lTable} VALUES (1,1,100),(2,2,200),(3,3,300)"""
+ sql """INSERT INTO ${rTable} VALUES (10,1),(20,2),(30,3)"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT ${lTable}.id AS lid,
+ ${rTable}.id AS rid,
+ SUM(${lTable}.v) AS total,
+ COUNT(*) AS cnt
+ FROM ${lTable}
+ INNER JOIN ${rTable}
+ ON ${lTable}.k = ${rTable}.k
+ GROUP BY ${lTable}.id, ${rTable}.id
+ """
+
+ sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_same_key_complete """SELECT lid, rid, total, cnt FROM ${mvName}"""
+
+ // Incremental delta: a brand-new group (lid=4, rid=40). Under the
same-name collapse
+ // the lid value would be taken from the rid slot and the group would be
written as
+ // (40,40), so this assert is the regression guard.
+ sql """INSERT INTO ${lTable} VALUES (4,4,400)"""
+ sql """INSERT INTO ${rTable} VALUES (40,4)"""
+ sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_same_key_incremental """SELECT lid, rid, total, cnt FROM
${mvName}"""
+
+ // Cross-check against a fresh COMPLETE rebuild.
+ sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_same_key_complete2 """SELECT lid, rid, total, cnt FROM
${mvName}"""
+
+ sql """drop materialized view if exists ${mvName}"""
+ sql """drop table if exists ${lTable}"""
+ sql """drop table if exists ${rTable}"""
+
+ // =========================================================
+ // Part 2: same-named group keys with ivm_use_full_keys=true
+ // (MV key = row_id + group by keys)
+ // =========================================================
+ def fkMv = "ivm_same_key_fk_mv"
+ def fkL = "ivm_same_key_fk_l"
+ def fkR = "ivm_same_key_fk_r"
+
+ sql """drop materialized view if exists ${fkMv}"""
+ sql """drop table if exists ${fkL}"""
+ sql """drop table if exists ${fkR}"""
+
+ sql """
+ CREATE TABLE ${fkL} (
+ id INT,
+ k INT,
+ v INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """
+ CREATE TABLE ${fkR} (
+ id INT,
+ k INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """INSERT INTO ${fkL} VALUES (1,1,100),(2,2,200),(3,3,300)"""
+ sql """INSERT INTO ${fkR} VALUES (10,1),(20,2),(30,3)"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ${fkMv}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1', 'ivm_use_full_keys' = 'true')
+ AS
+ SELECT ${fkL}.id AS lid,
+ ${fkR}.id AS rid,
+ SUM(${fkL}.v) AS total,
+ COUNT(*) AS cnt
+ FROM ${fkL}
+ INNER JOIN ${fkR}
+ ON ${fkL}.k = ${fkR}.k
+ GROUP BY ${fkL}.id, ${fkR}.id
+ """
+
+ sql """REFRESH MATERIALIZED VIEW ${fkMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(fkMv)
+ order_qt_same_key_fk_complete """SELECT lid, rid, total, cnt FROM
${fkMv}"""
+
+ sql """INSERT INTO ${fkL} VALUES (4,4,400)"""
+ sql """INSERT INTO ${fkR} VALUES (40,4)"""
+ sql """REFRESH MATERIALIZED VIEW ${fkMv} INCREMENTAL"""
+ waitingMTMVTaskFinishedByMvName(fkMv)
+ order_qt_same_key_fk_incremental """SELECT lid, rid, total, cnt FROM
${fkMv}"""
+
+ // Cross-check the full-keys MV against a fresh COMPLETE rebuild.
+ sql """REFRESH MATERIALIZED VIEW ${fkMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(fkMv)
+ order_qt_same_key_fk_complete2 """SELECT lid, rid, total, cnt FROM
${fkMv}"""
+
+ sql """drop materialized view if exists ${fkMv}"""
+ sql """drop table if exists ${fkL}"""
+ sql """drop table if exists ${fkR}"""
+
+ // =========================================================
+ // Part 3: same-named group keys under UPDATE (delete + insert delta)
+ // =========================================================
+ def upMv = "ivm_same_key_up_mv"
+ def upL = "ivm_same_key_up_l"
+ def upR = "ivm_same_key_up_r"
+
+ sql """drop materialized view if exists ${upMv}"""
+ sql """drop table if exists ${upL}"""
+ sql """drop table if exists ${upR}"""
+
+ sql """
+ CREATE TABLE ${upL} (
+ id INT,
+ k INT,
+ v INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """
+ CREATE TABLE ${upR} (
+ id INT,
+ k INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """INSERT INTO ${upL} VALUES (1,1,100),(2,2,200),(3,3,300)"""
+ sql """INSERT INTO ${upR} VALUES (10,1),(20,2),(30,3)"""
+
+ sql """
+ CREATE MATERIALIZED VIEW ${upMv}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT ${upL}.id AS lid,
+ ${upR}.id AS rid,
+ SUM(${upL}.v) AS total,
+ COUNT(*) AS cnt
+ FROM ${upL}
+ INNER JOIN ${upR}
+ ON ${upL}.k = ${upR}.k
+ GROUP BY ${upL}.id, ${upR}.id
+ """
+
+ sql """REFRESH MATERIALIZED VIEW ${upMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(upMv)
+ order_qt_same_key_up_complete """SELECT lid, rid, total, cnt FROM
${upMv}"""
+
+ // MOW update on l: (1,1,100) -> (1,5,50) drops the join with r(10,1) and
forms a new
+ // group (1,50). The incremental delta deletes group (1,10) and inserts
group (1,50);
+ // a same-name collapse would keep (1,10) and/or write the new key wrongly.
+ sql """INSERT INTO ${upL} VALUES (1,5,50)"""
+ sql """INSERT INTO ${upR} VALUES (50,5)"""
+ sql """REFRESH MATERIALIZED VIEW ${upMv} INCREMENTAL"""
+ waitingMTMVTaskFinishedByMvName(upMv)
+ order_qt_same_key_up_incremental """SELECT lid, rid, total, cnt FROM
${upMv}"""
+
+ // Cross-check the UPDATE result against a fresh COMPLETE rebuild.
+ sql """REFRESH MATERIALIZED VIEW ${upMv} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(upMv)
+ order_qt_same_key_up_complete2 """SELECT lid, rid, total, cnt FROM
${upMv}"""
+
+ sql """drop materialized view if exists ${upMv}"""
+ sql """drop table if exists ${upL}"""
+ sql """drop table if exists ${upR}"""
+}
diff --git
a/regression-test/suites/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.groovy
new file mode 100644
index 00000000000..900ec27cbfc
--- /dev/null
+++
b/regression-test/suites/mtmv_p0/ivm/test_ivm_full_keys_same_name_unprojected_key.groovy
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_ivm_full_keys_same_name_unprojected_key") {
+
+ // GROUP BY l.id, r.id over a join where both tables expose "id", while
the user output
+ // selects only the left id (right id is an identity key that must be
materialized as a
+ // hidden key column). With ivm_use_full_keys the MV key is row_id + group
by keys;
+ // treating the same-named right id as already projected drops it from the
key set.
+
+ def mvName = "ivm_fk_hidden_mv"
+ def lTable = "ivm_fk_hidden_l"
+ def rTable = "ivm_fk_hidden_r"
+
+ sql """drop materialized view if exists ${mvName}"""
+ sql """drop table if exists ${lTable}"""
+ sql """drop table if exists ${rTable}"""
+
+ sql """
+ CREATE TABLE ${lTable} (
+ id INT,
+ k INT,
+ v INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """
+ CREATE TABLE ${rTable} (
+ id INT,
+ k INT
+ )
+ UNIQUE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW", "binlog.need_historical_value" = "true",
+ "enable_unique_key_merge_on_write" = "true"
+ )
+ """
+ sql """INSERT INTO ${lTable} VALUES (1,1,100),(2,2,200),(3,3,300)"""
+ sql """INSERT INTO ${rTable} VALUES (10,1),(20,2),(30,3)"""
+
+ // Only l.id is selected; r.id participates in GROUP BY but is not in the
output, so it
+ // must be carried by a hidden key column under ivm_use_full_keys.
+ sql """
+ CREATE MATERIALIZED VIEW ${mvName}
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 2
+ PROPERTIES ('replication_num' = '1', 'ivm_use_full_keys' = 'true')
+ AS
+ SELECT ${lTable}.id AS lid,
+ SUM(${lTable}.v) AS total,
+ COUNT(*) AS cnt
+ FROM ${lTable}
+ INNER JOIN ${rTable}
+ ON ${lTable}.k = ${rTable}.k
+ GROUP BY ${lTable}.id, ${rTable}.id
+ """
+
+ // The hidden key column for the unprojected r.id group key must exist in
the MV schema
+ // (hidden columns are hidden from DESC unless show_hidden_columns is on).
+ sql """set show_hidden_columns = true"""
+ qt_fk_hidden_desc """DESC ${mvName}"""
+ sql """set show_hidden_columns = false"""
+
+ sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_fk_hidden_complete """SELECT lid, total, cnt FROM ${mvName}"""
+
+ sql """INSERT INTO ${lTable} VALUES (4,4,400)"""
+ sql """INSERT INTO ${rTable} VALUES (40,4)"""
+ sql """REFRESH MATERIALIZED VIEW ${mvName} INCREMENTAL"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_fk_hidden_incremental """SELECT lid, total, cnt FROM ${mvName}"""
+
+ // Cross-check against a fresh COMPLETE rebuild.
+ sql """REFRESH MATERIALIZED VIEW ${mvName} COMPLETE"""
+ waitingMTMVTaskFinishedByMvName(mvName)
+ order_qt_fk_hidden_complete2 """SELECT lid, total, cnt FROM ${mvName}"""
+
+ sql """drop materialized view if exists ${mvName}"""
+ sql """drop table if exists ${lTable}"""
+ sql """drop table if exists ${rTable}"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
