This is an automated email from the ASF dual-hosted git repository.
yujun777 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e64f695c068 [fix](ivm) Initialize layout signatures on the first
refresh (#70785)
e64f695c068 is described below
commit e64f695c06823a13802550168b13b8acbc88448c
Author: yujun <[email protected]>
AuthorDate: Sat Nov 21 23:04:34 2026 +0800
[fix](ivm) Initialize layout signatures on the first refresh (#70785)
Unique-key IVM materialized views use a normalized plan signature to
protect the row identity and aggregate state layout across refreshes.
Incremental refresh checks the stored signature before it builds delta
writes.
### What problem does this PR solve?
Issue Number: None
Related PR: #62606 (introduced IVM layout signatures), #69890 (publishes
the signature after a complete refresh)
PR Type: Bugfix
Problem Summary:
A newly created deferred IVM can fail its first incremental refresh
because CREATE signed its SELECT plan and REFRESH signs an INSERT plan.
This PR establishes the baseline from the first successful write and
uses the analyzed table sink's column names and types in the signature.
```sql
SELECT u.channel, COUNT(*) AS row_count, SUM(u.amount) AS total_amount
FROM (
SELECT channel, amount FROM sales -- channel is VARCHAR(16)
UNION ALL
SELECT 'refund' AS channel, -amount FROM refunds
) u
GROUP BY u.channel
```
### Details
- **Change map:** 271 production lines cover signature input, refresh
publication, and DRY RUN. 354 test lines update FE assertions and
exercise the grouped UNION ALL, with 23 generated golden lines.
- **Frequency:** Reproducible before the first COMPLETE when binding the
refresh adds a write Project, including the grouped UNION ALL above.
- **Root cause:** The union combines the VARCHAR(16) column and the
VARCHAR(6) literal and produces STRING. CREATE applies its
physical-schema rule for the first visible column and stores it as
VARCHAR(65533), the maximum payload length with a two-byte storage
prefix. That schema rule runs before IVM chooses the hidden row ID as
the key. REFRESH binds an INSERT against the resulting schema. With
automatic truncation enabled, BindSink inserts substring(channel, 1,
65533) in a Project above the grouped query, so signing the CREATE and
REFRESH plans gives different results. NULL to TINYINT and ARRAY element
length coercion can also add write-side projections.
- **Why existing tests missed it:** The existing fallback test injects a
mismatch after a COMPLETE has already established a table-sink
signature. It does not exercise the first write of a deferred MV with a
write-side type conversion.
- **Fix:** Generate signatures from a normalized OLAP table sink and
retain the Project topology, including passthrough Projects. Record the
ordered bound target column names and types. A first successful
incremental or partition write establishes the baseline. Whole-MV
COMPLETE checks its batch signatures and can replace the baseline on
success. CREATE and DRY RUN use a result sink and skip the signature.
EXPLAIN REFRESH plans the table-sink path and leaves the stored
signature untouched. The unique-key normalizer owns root validation and
signature comparison. The update log records the matching normalized
tree, canonical signature, and SHA-256. The task result handler consumes
these diagnostics before the task enters history.
- **Other paths and branches:** This applies to the MOW IVM model.
Table-sink partition IDs and DML options vary by execution path and are
outside the signature. The signature format version remains 1 because
IVM has not been released. Comparing a subsequent partition-only rebuild
against the stored signature is a follow-up.
- **Behavior change:** A deferred IVM can run its first refresh with
write-side type coercion and does not require a prior COMPLETE. DRY RUN
can preview delta rows even when the execution plan would fail signature
validation.
---
.../main/java/org/apache/doris/catalog/MTMV.java | 54 ++++++----
.../org/apache/doris/catalog/OlapTableFactory.java | 10 --
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 45 +++++---
.../doris/mtmv/ivm/IvmIncrRefreshManager.java | 7 +-
.../doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java | 75 ++++++++------
.../apache/doris/mtmv/ivm/IvmPlanSignature.java | 18 ++--
.../doris/mtmv/ivm/IvmPlanSignatureGenerator.java | 37 ++-----
.../nereids/rules/analysis/IvmIncrRefreshMTMV.java | 12 +--
.../nereids/rules/analysis/IvmNormalizeMTMV.java | 13 +--
.../trees/plans/commands/info/CreateMTMVInfo.java | 14 +--
.../java/org/apache/doris/mtmv/AlterMTMVTest.java | 3 +-
.../java/org/apache/doris/mtmv/MTMVTaskTest.java | 46 +++++++--
.../doris/mtmv/ivm/IvmBaselineRebuildTest.java | 28 ++++-
.../doris/mtmv/ivm/IvmIncrRefreshManagerTest.java | 2 +-
.../mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java | 55 ++++++----
.../mtmv/ivm/IvmPlanSignatureGeneratorTest.java | 98 +++++-------------
.../rules/analysis/IvmIncrRefreshMTMVTest.java | 24 ++---
.../nereids/trees/plans/CreateMTMVCommandTest.java | 2 +-
.../ivm/test_ivm_sink_truncation_signature.out | 23 +++++
.../ivm/test_ivm_sink_truncation_signature.groovy | 115 +++++++++++++++++++++
20 files changed, 408 insertions(+), 273 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index 61e09b06988..7b3b4aa8a34 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -56,6 +56,8 @@ import org.apache.doris.mtmv.MTMVSnapshotIf;
import org.apache.doris.mtmv.MTMVStatus;
import org.apache.doris.mtmv.MTMVUtil;
import org.apache.doris.mtmv.ivm.IvmInfo;
+import org.apache.doris.mtmv.ivm.IvmPlanSignature;
+import org.apache.doris.mtmv.ivm.IvmRefreshMethod;
import org.apache.doris.mtmv.ivm.IvmRefreshSummary;
import org.apache.doris.mtmv.ivm.IvmUtil;
import org.apache.doris.mtmv.ivm.agg.IvmAggStateKey;
@@ -165,17 +167,6 @@ public class MTMV extends OlapTable {
this.refreshSnapshot = new MTMVRefreshSnapshot();
this.ivmInfo = new IvmInfo();
this.ivmInfo.setEnableIvm(params.enableIvm);
- if (params.enableIvm) {
- // The unique-key model pins the layout its refresh reads down by
a signature, so an MV of that
- // model without one is a bug in the create path. An append-only
detail MV publishes none: its
- // refresh reads no layout, and it has no sequence column for a
signature to order either.
- if (params.keysType != KeysType.DUP_KEYS) {
- if (params.ivmPlanSignature == null) {
- throw new IllegalArgumentException("IVM materialized view
requires a plan signature");
- }
- this.ivmInfo.setPlanSignature(params.ivmPlanSignature);
- }
- }
this.envInfo = new EnvInfo(-1L, -1L);
this.sessionVariables = params.sessionVariables;
mvRwLock = new ReentrantReadWriteLock(true);
@@ -375,6 +366,7 @@ public class MTMV extends OlapTable {
public boolean addTaskResult(AlterMTMV alterMTMV, boolean isReplay) {
MTMVTask task = alterMTMV.getTask();
+ IvmPlanSignature refreshedIvmPlanSignature =
task.takeRefreshedIvmPlanSignature();
MTMVRelation relation = alterMTMV.getRelation();
Map<String, MTMVRefreshPartitionSnapshot> partitionSnapshots =
alterMTMV.getPartitionSnapshots();
MTMVCache mtmvCacheWithGuard = null;
@@ -457,11 +449,8 @@ public class MTMV extends OlapTable {
this.status.setSchemaChangeDetail(null);
this.status.setRefreshState(MTMVRefreshState.SUCCESS);
this.relation = relation;
- if (!isReplay && ivmInfo.isEnableIvm()) {
- String refreshedIvmPlanSignature =
task.getRefreshedIvmPlanSignature();
- if (refreshedIvmPlanSignature != null) {
- ivmInfo.setPlanSignature(refreshedIvmPlanSignature);
- }
+ if (!isReplay && ivmInfo.isEnableIvm() &&
refreshedIvmPlanSignature != null) {
+ updateIvmPlanSignature(refreshedIvmPlanSignature,
task.getTaskId(), IvmRefreshMethod.COMPLETE);
}
} else {
this.status.setRefreshState(MTMVRefreshState.FAIL);
@@ -875,9 +864,9 @@ public class MTMV extends OlapTable {
* <p>{@code isIvm()} is on the query path, and what it reads is fixed for
the MV's lifetime:
* {@code enableIvm} is decided at CREATE and no ALTER path changes it.
The rest of the payload is
* touched only by the refresh of this MV -- the sequence prefix is read
when a refresh plans and
- * advanced when its transaction commits, and the plan signature is
published by the same task's
- * result -- and one MV's tasks run one at a time, under its job's write
lock (see
- * {@code MTMVTask#runTask}). So no reader here races a writer.
+ * advanced when its transaction commits, and the refresh publishes the
layout signature after
+ * writing data. One MV's tasks run one at a time under its job's write
lock (see
+ * {@code MTMVTask#runTask}).
*
* <p>Never null: the field is created with the MV, and {@link
#gsonPostProcess} covers an image
* that carries it as null.
@@ -1480,6 +1469,33 @@ public class MTMV extends OlapTable {
return null;
}
+ /**
+ * Pins the layout on the first committed write. Publish it before any
subsequent partition batch
+ * or delta attempt, since successful writes survive a later failure in
the same refresh task.
+ */
+ public void initializeIvmPlanSignature(IvmPlanSignature signature, long
expectedSchemaChangeVersion,
+ long taskId, IvmRefreshMethod method) {
+ EditLogItem editLogItem;
+ writeMvLock();
+ try {
+ if (schemaChangeVersion != expectedSchemaChangeVersion ||
ivmInfo.getPlanSignature() != null) {
+ return;
+ }
+ updateIvmPlanSignature(signature, taskId, method);
+ editLogItem = submitIvmInfoChange();
+ } finally {
+ writeMvUnlock();
+ }
+ editLogItem.await();
+ }
+
+ private void updateIvmPlanSignature(IvmPlanSignature signature, long
taskId, IvmRefreshMethod method) {
+ ivmInfo.setPlanSignature(signature.getSha256());
+ LOG.info("Updated IVM layout signature, mv={}, taskId={}, method={},
plan={}, canonicalString={}, sha256={}",
+ getName(), taskId, method, signature.getPlanTree(),
signature.getCanonicalString(),
+ signature.getSha256());
+ }
+
private EditLogItem submitIvmInfoChange() {
// The caller has already mutated ivmInfo under the MV write lock.
Submit its snapshot directly;
// replay later applies the payload through alterIvmInfo().
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableFactory.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableFactory.java
index 125381d8ef5..3d68d3b59a2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableFactory.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTableFactory.java
@@ -53,7 +53,6 @@ public class OlapTableFactory {
public MTMVRefreshInfo refreshInfo;
public String querySql;
public boolean enableIvm;
- public String ivmPlanSignature;
public Map<String, String> mvProperties;
public MTMVPartitionInfo mvPartitionInfo;
public MTMVRelation relation;
@@ -165,14 +164,6 @@ public class OlapTableFactory {
return this;
}
- public OlapTableFactory withIvmPlanSignature(String ivmPlanSignature) {
- Preconditions.checkState(params instanceof MTMVParams, "Invalid
argument for "
- + params.getClass().getSimpleName());
- MTMVParams mtmvParams = (MTMVParams) params;
- mtmvParams.ivmPlanSignature = ivmPlanSignature;
- return this;
- }
-
private OlapTableFactory withRefreshInfo(MTMVRefreshInfo refreshInfo) {
Preconditions.checkState(params instanceof MTMVParams, "Invalid
argument for "
+ params.getClass().getSimpleName());
@@ -213,7 +204,6 @@ public class OlapTableFactory {
CreateMTMVInfo createMTMVInfo = (CreateMTMVInfo) createTableInfo;
return withRefreshInfo(createMTMVInfo.getRefreshInfo())
.withEnableIvm(createMTMVInfo.isEnableIvm())
- .withIvmPlanSignature(createMTMVInfo.getIvmPlanSignature())
.withQuerySql(createMTMVInfo.getQuerySql())
.withMvProperties(createMTMVInfo.getMvProperties())
.withMvPartitionInfo(createMTMVInfo.getMvPartitionInfo())
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
index d82fa07a00a..5460388ff8f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java
@@ -346,8 +346,8 @@ public class MTMVTask extends AbstractTask {
// Only the worker thread writes it, and it is read as the worker reports
the failure.
private transient String failureStage =
IvmRefreshFailureStage.OTHER.name();
private long mtmvSchemaChangeVersion;
- // Published only after a signature-mismatch fallback succeeds and its
task result is accepted.
- private transient String refreshedIvmPlanSignature;
+ // Passed to the task result handler and cleared before the task enters
history.
+ private transient volatile IvmPlanSignature refreshedIvmPlanSignature;
private Map<MvccTableInfo, MvccSnapshot> snapshots = Maps.newHashMap();
@@ -1004,8 +1004,10 @@ public class MTMVTask extends AbstractTask {
// recorded only if the refresh commits; see captureLatestEpochs.
Map<String, Long> capturedEpochs =
captureLatestEpochs(incrementalScope);
IvmIncrRefreshResult ivmResult;
+ AtomicReference<IvmPlanSignature> signatureRef = new
AtomicReference<>();
try {
ivmResult = executeWithRetry(() -> {
+ signatureRef.set(null);
ConnectContext ivmConnectContext =
MTMVPlanUtil.createMTMVContext(mtmv,
MTMVPlanUtil.DISABLE_RULES_WHEN_RUN_MTMV_TASK);
try {
@@ -1014,7 +1016,7 @@ public class MTMVTask extends AbstractTask {
ivmConnectContext,
getRefreshAuditStmt(RefreshMode.INCREMENTAL,
Sets.newHashSet(incrementalScope)),
this::recordQueryId,
- this::registerExecutor);
+ createRefreshConsumer(signatureRef));
mtmv.validateIvmRefreshStart(mtmvSchemaChangeVersion);
return
ivmIncrRefreshManager.doRefresh(ivmIncrRefreshContext);
} finally {
@@ -1027,6 +1029,11 @@ public class MTMVTask extends AbstractTask {
}
if (ivmResult.isSuccess()) {
recordCommitted(IvmRefreshMethod.INCREMENTAL);
+ IvmPlanSignature signature = signatureRef.get();
+ if (signature != null) {
+ mtmv.initializeIvmPlanSignature(signature,
mtmvSchemaChangeVersion, getTaskId(),
+ IvmRefreshMethod.INCREMENTAL);
+ }
this.partitionSnapshots.putAll(capturedSnapshots);
recordRefreshCompleted(incrementalScope);
commitCapturedEpochs(capturedEpochs);
@@ -1379,8 +1386,7 @@ public class MTMVTask extends AbstractTask {
boolean refreshAllPartitions =
Sets.newHashSet(partitions).equals(mtmv.getPartitionNames());
// A whole-MV COMPLETE resets the stream baselines as it rebuilds the
rows. Publish the
// matching layout signature with that successful baseline, including
a capacity change.
- // A partition refresh cannot publish a signature for partitions it
did not rebuild.
- boolean capturePlanSignature = refreshMode == RefreshMode.COMPLETE &&
mtmv.isIvm()
+ boolean replacePlanSignature = refreshMode == RefreshMode.COMPLETE &&
mtmv.isIvm()
&& !mtmv.isIvmAppendOnlyDupTable();
IvmPlanSignature refreshedPlanSignature = null;
for (int i = 0; i < execNum; i++) {
@@ -1409,7 +1415,7 @@ public class MTMVTask extends AbstractTask {
try {
IvmPlanSignature batchPlanSignature =
refreshPartitionsWithRetry(
execPartitionNames, tableWithPartKey, rewriteContext,
refreshMode);
- if (capturePlanSignature) {
+ if (replacePlanSignature) {
batchPlanSignature =
Objects.requireNonNull(batchPlanSignature,
"IVM COMPLETE refresh did not produce a plan
signature");
if (refreshedPlanSignature == null) {
@@ -1418,6 +1424,9 @@ public class MTMVTask extends AbstractTask {
throw new JobException("IVM COMPLETE refresh generated
inconsistent plan signatures, mv="
+ mtmv.getName());
}
+ } else if (batchPlanSignature != null) {
+ mtmv.initializeIvmPlanSignature(batchPlanSignature,
mtmvSchemaChangeVersion, getTaskId(),
+ IvmRefreshMethod.PARTITION_REBUILD);
}
} catch (Exception e) {
LOG.error("Execution failed after retries, mvName: {}, taskId:
{}",
@@ -1451,8 +1460,12 @@ public class MTMVTask extends AbstractTask {
commitCapturedEpochs(batchCapturedEpochs);
}
}
- if (capturePlanSignature) {
- refreshedIvmPlanSignature = refreshedPlanSignature.getSha256();
+ if (replacePlanSignature) {
+ refreshedIvmPlanSignature = refreshedPlanSignature;
+ // The cancel thread may have recorded the result before this
worker finished the last batch.
+ if (getStatus() == TaskStatus.CANCELED) {
+ refreshedIvmPlanSignature = null;
+ }
}
LOG.info("MTMVTask refresh used snapshot: {}, mvDbName: {}, mvName:
{}, taskId: {}", partitionSnapshots,
mtmv.getDatabase().getFullName(), mtmv.getName(), getTaskId());
@@ -1724,19 +1737,15 @@ public class MTMVTask extends AbstractTask {
return null;
}
return Objects.requireNonNull(signatureRef.get(),
- "IVM COMPLETE refresh did not produce a plan signature");
+ "IVM refresh did not produce a plan signature");
}
/**
- * Builds the executor consumer for a COMPLETE refresh: registers the
executing
- * statement for cancellation, and when the command finishes (consumer
invoked with
- * {@code null}), captures the IVM plan signature into {@code
signatureRef} from the
- * still-registered executor before it is cleared.
+ * Registers the refresh executor for cancellation and captures the plan
signature on completion,
+ * before the null callback clears the executor.
*/
private Consumer<StmtExecutor>
createRefreshConsumer(AtomicReference<IvmPlanSignature> signatureRef) {
return executor -> {
- // executor == null 且 this.executor 仍注册:executeCommand 的 finally
回调,
- // 命令已执行完成,在 registerExecutor(null) 清空字段前先摘取 IVM 签名。
if (executor == null && this.executor != null &&
this.executor.planner() != null) {
((NereidsPlanner) this.executor.planner())
.getCascadesContext()
@@ -2188,8 +2197,10 @@ public class MTMVTask extends AbstractTask {
return ivmCapturedEpochs == null ? Collections.emptyMap() :
Maps.newHashMap(ivmCapturedEpochs);
}
- public String getRefreshedIvmPlanSignature() {
- return refreshedIvmPlanSignature;
+ public IvmPlanSignature takeRefreshedIvmPlanSignature() {
+ IvmPlanSignature signature = refreshedIvmPlanSignature;
+ refreshedIvmPlanSignature = null;
+ return signature;
}
public IvmRefreshMethod getIvmRefreshMethod() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManager.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManager.java
index 034527873b1..0cecd466856 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManager.java
@@ -26,6 +26,7 @@ import org.apache.doris.mtmv.BaseTableInfo;
import org.apache.doris.mtmv.MTMVPartitionUtil;
import org.apache.doris.mtmv.MTMVPlanUtil;
import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.analyzer.UnboundResultSink;
import org.apache.doris.nereids.analyzer.UnboundTableSink;
import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.nereids.trees.plans.Plan;
@@ -123,12 +124,10 @@ public class IvmIncrRefreshManager {
}
/**
- * Parse the MV's SELECT query into a logical plan (sink stripped). Used
by REFRESH ...
- * INCREMENTAL WITH DRY RUN, which plans the raw query with an IVM rewrite
context so the
- * analyzer rules turn it into the delta query.
+ * Build the result-producing query for REFRESH ... INCREMENTAL WITH DRY
RUN.
*/
public LogicalPlan buildQueryPlan(MTMV mtmv) {
- return parseInsertQueryPlan(mtmv);
+ return new UnboundResultSink<>(parseInsertQueryPlan(mtmv));
}
@VisibleForTesting
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
index d0283c0ec6c..5e0d8412fa0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMV.java
@@ -71,6 +71,7 @@ import org.apache.doris.nereids.types.TinyIntType;
import org.apache.doris.nereids.types.VarcharType;
import org.apache.doris.nereids.util.TypeCoercionUtils;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
@@ -82,7 +83,6 @@ import java.util.IdentityHashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.stream.Collectors;
@@ -215,15 +215,53 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
private int unionIdxCounter;
private int baseTableRowIdRenameCounter;
+ /**
+ * Normalizes a unique-key MV statement after {@link IvmNormalizeMTMV}
selects its model.
+ * Computes the layout signature for write plans and checks the existing
baseline for incremental refresh.
+ */
+ public Plan normalizeRoot(Plan plan, JobContext jobContext) {
+ Plan result = normalize(plan, jobContext);
+ if (result instanceof LogicalResultSink) {
+ return result;
+ }
+ if (!(result instanceof LogicalOlapTableSink)) {
+ throw new IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
+ "IVM MOW normalization requires a table or result sink,
but found "
+ + result.getClass().getSimpleName());
+ }
+
+ IvmPlanSignature signature = new
IvmPlanSignatureGenerator().generate(result);
+ if (ivmRewriteContext.getMode() == IvmRewriteContext.Mode.INCREMENTAL)
{
+ validatePlanSignature(ivmRewriteContext.getMtmv(), signature);
+ }
+ rewriteResult.setPlanSignature(signature);
+ return result;
+ }
+
+ private void validatePlanSignature(MTMV mtmv, IvmPlanSignature
currentSignature) {
+ String storedSignature = mtmv.getIvmInfo().getPlanSignature();
+ if (storedSignature == null ||
storedSignature.equals(currentSignature.getSha256())) {
+ return;
+ }
+ String detail = "IVM layout signature mismatch for mv=" +
mtmv.getName()
+ + ", storedSignature=" + storedSignature
+ + ", currentSignature=" + currentSignature.getSha256()
+ + ", currentCanonical=" + currentSignature.getCanonicalString()
+ + ", currentPlan=" + currentSignature.getPlanTree()
+ + ". Run one REFRESH MATERIALIZED VIEW " + mtmv.getName()
+ + " INCREMENTAL FALLBACK or REFRESH MATERIALIZED VIEW " +
mtmv.getName()
+ + " COMPLETE to rebuild the IVM layout baseline and publish
its matching signature.";
+ throw new IvmException(IvmFailureReason.PLAN_SIGNATURE_MISMATCH,
detail);
+ }
+
/**
* Rewrites {@code plan} into the hidden layout a unique-key IVM MV is
maintained through.
*
* <p>The table model has already been settled and recorded by {@link
IvmNormalizeMTMV}, which is
* the only production caller: what is left here is the layout itself,
which is why this rewriter
- * knows nothing about any other model. The caller publishes the returned
plan as the statement's
- * normalized plan, as it does for the detail model too; this rewriter
publishes the signature that
- * pins the layout down, which is what its model adds to the plan and no
other model has.
+ * knows nothing about any other model.
*/
+ @VisibleForTesting
public Plan normalize(Plan plan, JobContext jobContext) {
StatementContext statementContext =
jobContext.getCascadesContext().getStatementContext();
ivmRewriteContext =
statementContext.getIvmRewriteContext().orElse(null);
@@ -243,38 +281,9 @@ public class IvmNormalizeUniqKeysMTMV extends
DefaultPlanRewriter<IvmNormalizeUn
this.useFullKeys = resolveUseFullKeys();
Plan result = plan.accept(this, NormalizeContext.ROOT);
validateAggStateColumnsPersisted(result);
- // The plan is not published here: the caller publishes it, and does
so for both table models.
- // What this model adds to a normalized plan is the layout its refresh
reads, which is published
- // as the signature that pins the layout down.
- IvmPlanSignature planSignature = new
IvmPlanSignatureGenerator().generate(result);
- rewriteResult.setPlanSignature(planSignature);
- IvmRewriteContext.Mode mode = ivmRewriteContext.getMode();
- if (mode == IvmRewriteContext.Mode.INCREMENTAL) {
- // Incremental refresh relies on the stored IVM layout: a drift in
the normalized
- // plan (row-id generation path) would produce an unmatchable
delta. Check the
- // signature as soon as normalization completes so every
incremental path fails
- // fast instead of attempting a delta rewrite against a stale
layout baseline.
- validatePlanSignature(ivmRewriteContext.getMtmv(), planSignature,
result);
- }
return result;
}
- private void validatePlanSignature(MTMV mtmv, IvmPlanSignature
currentSignature, Plan currentPlan) {
- String storedSignature = mtmv.getIvmInfo().getPlanSignature();
- if (Objects.equals(storedSignature, currentSignature.getSha256())) {
- return;
- }
- String detail = "IVM layout signature mismatch for mv=" +
mtmv.getName()
- + ", storedSignature=" + storedSignature
- + ", currentSignature=" + currentSignature.getSha256()
- + ", currentCanonical=" + currentSignature.getCanonicalString()
- + ", currentPlan=" + currentPlan.treeString()
- + ". Run one REFRESH MATERIALIZED VIEW " + mtmv.getName()
- + " INCREMENTAL FALLBACK or REFRESH MATERIALIZED VIEW " +
mtmv.getName()
- + " COMPLETE to rebuild the IVM layout baseline and publish
its matching signature.";
- throw new IvmException(IvmFailureReason.PLAN_SIGNATURE_MISMATCH,
detail);
- }
-
/**
* The MIN/MAX top-K capacity of this MV, resolved once per statement.
*
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignature.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignature.java
index 32964c47add..5aeaf692612 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignature.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignature.java
@@ -24,16 +24,15 @@ import java.util.Objects;
*
* <p>The canonical string records only normalized-plan properties that may
change
* IVM row-id or hidden-column layout, such as hidden output expressions, scan
- * identity, join child order, and union arm order. The persisted signature
stores
- * only the SHA-256 digest; the canonical string is kept in memory for
diagnostics
- * and tests.
+ * identity, join child order, union arm order, and table sink column names
and types.
+ * The persisted signature stores only the SHA-256 digest. The canonical
string and plan
+ * tree are kept in memory for diagnostics during planning and refresh result
publication.
*
* <p>Example:
* <pre>
* canonicalString =
* IVM_LAYOUT_SIGNATURE_V1
- * ROOT[hiddenOutput=[SLOT[name=__DORIS_IVM_ROW_ID_COL__]],
- * plan=PROJECT[hiddenOutputs=[...],child=SCAN[table=ctl.db.t]]]
+ *
ROOT[plan=TABLE_SINK[columns=[...],child=PROJECT[hiddenOutputs=[...],child=SCAN[table=ctl.db.t]]]]
* sha256 = 5a3f...e92c
* </pre>
*/
@@ -44,15 +43,22 @@ public class IvmPlanSignature {
/** SHA-256 digest of {@link #canonicalString}; this is the compact value
persisted on the MV. */
private final String sha256;
- public IvmPlanSignature(String canonicalString, String sha256) {
+ private final String planTree;
+
+ public IvmPlanSignature(String canonicalString, String sha256, String
planTree) {
this.canonicalString = Objects.requireNonNull(canonicalString,
"canonicalString can not be null");
this.sha256 = Objects.requireNonNull(sha256, "sha256 can not be null");
+ this.planTree = Objects.requireNonNull(planTree, "planTree can not be
null");
}
public String getCanonicalString() {
return canonicalString;
}
+ public String getPlanTree() {
+ return planTree;
+ }
+
public String getSha256() {
return sha256;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGenerator.java
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGenerator.java
index bad0c5359e7..b79fc3cb35d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGenerator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGenerator.java
@@ -84,7 +84,7 @@ public class IvmPlanSignatureGenerator {
if (!debugSalt.isEmpty()) {
canonical = canonical + "\nDEBUG_SIGNATURE_SALT=" + debugSalt;
}
- return new IvmPlanSignature(canonical, sha256(canonical));
+ return new IvmPlanSignature(canonical, sha256(canonical),
normalizedPlan.treeString());
}
private CanonicalNode canonicalPlan(Plan plan) {
@@ -92,36 +92,11 @@ public class IvmPlanSignatureGenerator {
}
private CanonicalNode canonicalProject(LogicalProject<?> project) {
- if (isPassthroughProject(project)) {
- return canonicalPlan(project.child());
- }
return CanonicalNode.node("PROJECT")
.field("hiddenOutputs",
canonicalHiddenNamedExpressions(project.getProjects()))
.field("child", canonicalPlan(project.child()));
}
- private boolean isPassthroughProject(LogicalProject<?> project) {
- return
project.getProjects().stream().allMatch(this::isSignatureNeutralProjection);
- }
-
- private boolean isSignatureNeutralProjection(NamedExpression expression) {
- return isPassthroughExpression(expression) ||
isIgnorableSinkHiddenProjection(expression);
- }
-
- private boolean isPassthroughExpression(NamedExpression expression) {
- if (expression instanceof SlotReference) {
- return true;
- }
- if (expression instanceof Alias && ((Alias) expression).child()
instanceof SlotReference) {
- return expression.getName().equals(((SlotReference) ((Alias)
expression).child()).getName());
- }
- return false;
- }
-
- private boolean isIgnorableSinkHiddenProjection(NamedExpression
expression) {
- return IvmUtil.isCommonHiddenSlot(expression.getName());
- }
-
private CanonicalNode canonicalAggregate(LogicalAggregate<?> agg) {
return CanonicalNode.node("AGG")
.field("hiddenOutputs",
canonicalHiddenNamedExpressions(agg.getOutputExpressions()))
@@ -417,7 +392,15 @@ public class IvmPlanSignatureGenerator {
@Override
public CanonicalNode visitLogicalOlapTableSink(LogicalOlapTableSink<?
extends Plan> sink, Void context) {
- return sink.child().accept(this, context);
+ CanonicalList columns = CanonicalList.list();
+ for (Slot slot : sink.getTargetTableSlots()) {
+ columns.add(CanonicalNode.node("COLUMN")
+ .field("name", slot.getName())
+ .field("type", slot.getDataType().toSql()));
+ }
+ return CanonicalNode.node("TABLE_SINK")
+ .field("columns", columns)
+ .field("child", canonicalPlan(sink.child()));
}
@Override
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMV.java
index e8b0b4af132..8278d76c501 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMV.java
@@ -71,17 +71,9 @@ public class IvmIncrRefreshMTMV implements CustomRewriter {
private Plan rewriteIncrementalPlan(Plan plan, IvmRewriteResult
rewriteResult,
IvmRewriteContext rewriteContext, ConnectContext connectContext) {
- if (!(plan instanceof LogicalOlapTableSink)) {
- if (!rewriteContext.isDryRun()) {
- throw new
IvmException(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
- "IVM incremental refresh requires LogicalOlapTableSink
root, but found "
- + plan.getClass().getSimpleName());
- }
- // Dry run: the root is the raw MV query (no sink wrapper).
Rewrite it in place,
- // then cap rows with an optional LogicalLimit and wrap with a
result sink so the
- // plan is executable as a plain query.
+ if (rewriteContext.isDryRun()) {
Plan rewritten = newDeltaRewriter().generateIncrRefreshPlan(
- plan, rewriteResult, rewriteContext, connectContext);
+ plan.child(0), rewriteResult, rewriteContext,
connectContext);
Optional<IvmDryRunLimit> dryRunLimit =
rewriteContext.getDryRunLimit();
if (dryRunLimit.isPresent()) {
IvmDryRunLimit limit = dryRunLimit.get();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
index 5f3cefa7369..0ce366723ff 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/IvmNormalizeMTMV.java
@@ -21,7 +21,6 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.MTMV;
import org.apache.doris.mtmv.ivm.IvmNormalizeDupKeysMTMV;
import org.apache.doris.mtmv.ivm.IvmNormalizeUniqKeysMTMV;
-import org.apache.doris.mtmv.ivm.IvmPlanSignature;
import org.apache.doris.mtmv.ivm.IvmRewriteContext;
import org.apache.doris.mtmv.ivm.IvmRewriteResult;
import org.apache.doris.nereids.StatementContext;
@@ -78,22 +77,16 @@ public class IvmNormalizeMTMV implements CustomRewriter {
// pin down.
result = new IvmNormalizeDupKeysMTMV().normalize(plan,
statementContext.getExcludedTriggerTables());
} else {
- result = new IvmNormalizeUniqKeysMTMV().normalize(plan,
jobContext);
+ result = new IvmNormalizeUniqKeysMTMV().normalizeRoot(plan,
jobContext);
}
// Whichever normalizer built it, the normalized plan is published
here, and only here, so
// that a statement downstream of this rewrite reads the plan of the
model that was chosen.
rewriteResult.setNormalizeRewritten(true);
rewriteResult.setNormalizedPlan(result);
if (rewriteContext.getMode() == IvmRewriteContext.Mode.CREATE) {
- // The normalized plan is what an incremental refresh is built
against, so CREATE logs it
- // whole, together with the signature the unique-key model pins it
down by. A detail MV has
- // no signature: it holds the query's own rows, and its refresh
reads no layout.
- IvmPlanSignature planSignature = rewriteResult.getPlanSignature();
- LOG.info("IVM normalized plan, mtmvName={}, keysType={},
detailModel={}, inputRoot={}, plan={}, "
- + "canonicalString={}, signature={}",
+ LOG.info("IVM normalized plan, mtmvName={}, keysType={},
detailModel={}, inputRoot={}, plan={}",
rewriteContext.getMtmvName(), keysType, detailModel,
plan.getClass().getSimpleName(),
- result.treeString(), planSignature == null ? null :
planSignature.getCanonicalString(),
- planSignature == null ? null : planSignature.getSha256());
+ result.treeString());
}
return result;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
index 5c7115e2136..98bce5a233e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
@@ -109,7 +109,6 @@ public class CreateMTMVInfo extends CreateTableInfo {
* distribution and table-information passes lay the MV out differently
per table model.
*/
private boolean ivmAppendOnlyDupModel;
- private String ivmPlanSignature;
/**
* constructor for create MTMV
@@ -373,14 +372,10 @@ public class CreateMTMVInfo extends CreateTableInfo {
&&
mtmvAnalyzeQueryInfo.getIvmRewriteResult().isAppendOnlyDupTable();
if (isEnableIvm() && !ivmAppendOnlyDupModel) {
// The unique-key model orders what its delta writes against what
the MV already holds, so its
- // sequence column type is part of the MV's properties, and the
layout it reads is pinned down by
- // a signature. An append-only detail MV has neither.
+ // sequence column type is part of the MV's properties.
sequenceDataType =
calculateSequenceDataType(mtmvAnalyzeQueryInfo.getIvmNormalizedPlan());
this.properties.put(PropertyAnalyzer.PROPERTIES_FUNCTION_COLUMN +
"."
+ PropertyAnalyzer.PROPERTIES_SEQUENCE_TYPE,
sequenceDataType.toCatalogDataType().toSql());
- this.ivmPlanSignature =
mtmvAnalyzeQueryInfo.getIvmRewriteResult().getPlanSignature().getSha256();
- } else {
- this.ivmPlanSignature = null;
}
}
@@ -576,10 +571,6 @@ public class CreateMTMVInfo extends CreateTableInfo {
return mvPartitionInfo;
}
- public String getIvmPlanSignature() {
- return ivmPlanSignature;
- }
-
public Map<String, String> getSessionVariables() {
return sessionVariables;
}
@@ -596,7 +587,6 @@ public class CreateMTMVInfo extends CreateTableInfo {
private final MTMVPartitionType mvPartitionType;
private final Expression mvPartitionExpression;
private final boolean enableIvm;
- private final String ivmPlanSignature;
private AnalyzeQueryState(CreateMTMVInfo info) {
this.properties = info.properties == null ? null :
Maps.newHashMap(info.properties);
@@ -610,7 +600,6 @@ public class CreateMTMVInfo extends CreateTableInfo {
this.mvPartitionType =
info.mvPartitionDefinition.getPartitionType();
this.mvPartitionExpression =
info.mvPartitionDefinition.getFunctionCallExpression();
this.enableIvm = info.enableIvm;
- this.ivmPlanSignature = info.ivmPlanSignature;
}
private static AnalyzeQueryState capture(CreateMTMVInfo info) {
@@ -629,7 +618,6 @@ public class CreateMTMVInfo extends CreateTableInfo {
info.mvPartitionDefinition.setPartitionType(mvPartitionType);
info.mvPartitionDefinition.setFunctionCallExpression(mvPartitionExpression);
info.enableIvm = enableIvm;
- info.ivmPlanSignature = ivmPlanSignature;
}
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java
index 42b6796061f..9e1e3d5271d 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/AlterMTMVTest.java
@@ -414,9 +414,8 @@ public class AlterMTMVTest extends TestWithFeService {
.getDb("alter_ivm_test").get()
.getTableOrMetaException("ivm_alter_mv");
- // Incremental MTMV persists plan signature at create time.
IvmInfo initialInfo = mtmv.getIvmInfo();
- Assertions.assertNotNull(initialInfo.getPlanSignature());
+ Assertions.assertNull(initialInfo.getPlanSignature());
IvmInfo newInfo = new IvmInfo(initialInfo);
newInfo.setPlanSignature("sig-1");
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
index c6d093c702b..df71c12cf8a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVTaskTest.java
@@ -2013,15 +2013,40 @@ public class MTMVTaskTest {
}
}
+ @Test
+ public void testSuccessfulIncrementalRefreshPublishesFirstSignature()
throws Exception {
+ Mockito.when(mtmv.isIvm()).thenReturn(true);
+ Mockito.when(mtmv.getName()).thenReturn("test_mv");
+ MTMVTask task = new MTMVTask(mtmv, relation, new
MTMVTaskContext(MTMVTaskTriggerMode.MANUAL));
+ MTMVRefreshContext refreshContext = mockIvmIncrRefreshContext();
+ IvmPlanSignature signature = new IvmPlanSignature("layout",
"signature", "plan tree");
+ StmtExecutor executor = executorWithPlanSignature(signature);
+
+ try (MockedConstruction<IvmIncrRefreshManager> ignored =
Mockito.mockConstruction(IvmIncrRefreshManager.class,
+ (mock, context) ->
Mockito.when(mock.doRefresh(Mockito.any())).thenAnswer(invocation -> {
+ IvmIncrRefreshContext refresh = invocation.getArgument(0);
+ refresh.getExecutorConsumer().accept(executor);
+ refresh.getExecutorConsumer().accept(null);
+ return IvmIncrRefreshResult.success();
+ }))) {
+ IvmIncrRefreshResult result = Deencapsulation.invoke(task,
"executeSingleIvmAttempt",
+ refreshContext, Collections.emptySet());
+ Assertions.assertTrue(result.isSuccess());
+ }
+
Mockito.verify(mtmv).initializeIvmPlanSignature(Mockito.same(signature),
Mockito.anyLong(),
+ Mockito.eq(task.getTaskId()),
Mockito.eq(IvmRefreshMethod.INCREMENTAL));
+ }
+
@Test
public void testSignatureMismatchCompleteCapturesConsistentPlanSignature()
throws Exception {
MTMVTask task = new MTMVTask(mtmv, relation, new
MTMVTaskContext(MTMVTaskTriggerMode.MANUAL));
Deencapsulation.setField(task, "ivmFallbackReason",
IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name());
- IvmPlanSignature signature = new IvmPlanSignature("layout_s2",
"signature_s2");
+ IvmPlanSignature signature = new IvmPlanSignature("layout_s2",
"signature_s2", "plan tree");
executeCompleteRefresh(task, signature, signature);
- Assertions.assertEquals("signature_s2",
task.getRefreshedIvmPlanSignature());
+ Assertions.assertSame(signature, task.takeRefreshedIvmPlanSignature());
+ Assertions.assertNull(task.takeRefreshedIvmPlanSignature());
}
@Test
@@ -2030,32 +2055,32 @@ public class MTMVTaskTest {
Deencapsulation.setField(task, "ivmFallbackReason",
IvmFailureReason.PLAN_SIGNATURE_MISMATCH.name());
JobException exception = Assertions.assertThrows(JobException.class,
() -> executeCompleteRefresh(task,
- new IvmPlanSignature("layout_s2", "signature_s2"),
- new IvmPlanSignature("layout_s3", "signature_s3")));
+ new IvmPlanSignature("layout_s2", "signature_s2", "plan tree"),
+ new IvmPlanSignature("layout_s3", "signature_s3", "plan
tree")));
Assertions.assertTrue(exception.getMessage().contains("inconsistent
plan signatures"));
- Assertions.assertNull(task.getRefreshedIvmPlanSignature());
+ Assertions.assertNull(task.takeRefreshedIvmPlanSignature());
}
@Test
public void testOtherCompleteFallbackCapturesPlanSignature() throws
Exception {
MTMVTask task = new MTMVTask(mtmv, relation, new
MTMVTaskContext(MTMVTaskTriggerMode.MANUAL));
Deencapsulation.setField(task, "ivmFallbackReason",
IvmFailureReason.CHANGELOG_BROKEN.name());
- IvmPlanSignature signature = new IvmPlanSignature("layout_s2",
"signature_s2");
+ IvmPlanSignature signature = new IvmPlanSignature("layout_s2",
"signature_s2", "plan tree");
executeCompleteRefresh(task, signature, signature);
- Assertions.assertEquals("signature_s2",
task.getRefreshedIvmPlanSignature());
+ Assertions.assertSame(signature, task.takeRefreshedIvmPlanSignature());
}
@Test
public void testCapacityRebuildCapturesPlanSignatureWithoutFallback()
throws Exception {
Mockito.when(mtmv.getStatus()).thenReturn(new
MTMVStatus(MTMVState.SCHEMA_CHANGE, "capacity changed"));
MTMVTask task = new MTMVTask(mtmv, relation, new
MTMVTaskContext(MTMVTaskTriggerMode.MANUAL));
- IvmPlanSignature signature = new IvmPlanSignature("layout_k4",
"signature_k4");
+ IvmPlanSignature signature = new IvmPlanSignature("layout_k4",
"signature_k4", "plan tree");
executeCompleteRefresh(task, signature, signature);
Assertions.assertNull(Deencapsulation.getField(task,
"ivmFallbackReason"));
- Assertions.assertEquals("signature_k4",
task.getRefreshedIvmPlanSignature());
+ Assertions.assertSame(signature, task.takeRefreshedIvmPlanSignature());
}
@Test
@@ -2138,7 +2163,8 @@ public class MTMVTaskTest {
@Test
public void testRefreshedIvmPlanSignatureIsNotSerialized() {
MTMVTask task = new MTMVTask();
- Deencapsulation.setField(task, "refreshedIvmPlanSignature",
"new_signature");
+ Deencapsulation.setField(task, "refreshedIvmPlanSignature",
+ new IvmPlanSignature("layout", "new_signature", "plan tree"));
Assertions.assertFalse(GsonUtils.GSON.toJson(task).contains("new_signature"));
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
index 0ce6ee5b229..b6a4b14ffc2 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmBaselineRebuildTest.java
@@ -1285,10 +1285,12 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
int historySize = mtmv.getHistoryTasks().size();
AlterMTMV result = taskResult(mtmv, TaskStatus.FAILED, taskVersion);
- Deencapsulation.setField(result.getTask(),
"refreshedIvmPlanSignature", "new_signature");
+ Deencapsulation.setField(result.getTask(), "refreshedIvmPlanSignature",
+ new IvmPlanSignature("layout", "new_signature", "plan tree"));
String planSignature = mtmv.getIvmInfo().getPlanSignature();
Assertions.assertFalse(mtmv.addTaskResult(result, false));
+
Assertions.assertNull(result.getTask().takeRefreshedIvmPlanSignature());
Assertions.assertEquals(historySize, mtmv.getHistoryTasks().size());
Assertions.assertEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
Assertions.assertEquals(planSignature,
mtmv.getIvmInfo().getPlanSignature());
@@ -1341,13 +1343,33 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
Assertions.assertEquals(planSignature,
mtmv.getIvmInfo().getPlanSignature());
}
+ @Test
+ public void testPartitionRefreshInitializesButCannotReplacePlanSignature()
throws Exception {
+ String db = "ivm_partition_refresh_initial_signature";
+ createPartitionedIvmTableAndMv(db);
+ MTMV mtmv = getMtmv(db);
+ long version = mtmv.getSchemaChangeVersion();
+
+ Assertions.assertNull(mtmv.getIvmInfo().getPlanSignature());
+ IvmPlanSignature first = new IvmPlanSignature("first layout", "first
signature", "first plan");
+ mtmv.initializeIvmPlanSignature(first, version + 1, 1,
IvmRefreshMethod.PARTITION_REBUILD);
+ Assertions.assertNull(mtmv.getIvmInfo().getPlanSignature());
+
+ mtmv.initializeIvmPlanSignature(first, version, 2,
IvmRefreshMethod.PARTITION_REBUILD);
+ IvmPlanSignature second = new IvmPlanSignature("second layout",
"second signature", "second plan");
+ mtmv.initializeIvmPlanSignature(second, version, 3,
IvmRefreshMethod.PARTITION_REBUILD);
+
+ Assertions.assertEquals(first.getSha256(),
mtmv.getIvmInfo().getPlanSignature());
+ }
+
@Test
public void testSuccessfulSignatureFallbackPublishesNewPlanSignature()
throws Exception {
String db = "ivm_successful_signature_fallback";
createPartitionedIvmTableAndMv(db);
MTMV mtmv = getMtmv(db);
AlterMTMV result = taskResult(mtmv, TaskStatus.SUCCESS,
mtmv.getSchemaChangeVersion());
- Deencapsulation.setField(result.getTask(),
"refreshedIvmPlanSignature", "new_signature");
+ Deencapsulation.setField(result.getTask(), "refreshedIvmPlanSignature",
+ new IvmPlanSignature("layout", "new_signature", "plan tree"));
boolean compatibilityMode = Config.enable_check_compatibility_mode;
try {
Config.enable_check_compatibility_mode = true;
@@ -1358,6 +1380,8 @@ public class IvmBaselineRebuildTest extends
TestWithFeService {
Assertions.assertEquals("new_signature",
mtmv.getIvmInfo().getPlanSignature());
Assertions.assertEquals("new_signature",
result.getIvmInfo().getPlanSignature());
+
Assertions.assertTrue(mtmv.getHistoryTasks().contains(result.getTask()));
+
Assertions.assertNull(result.getTask().takeRefreshedIvmPlanSignature());
Assertions.assertNotEquals(MTMVState.SCHEMA_CHANGE,
mtmv.getStatus().getState());
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManagerTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManagerTest.java
index ef318686683..7d0a9ec2c8c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManagerTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmIncrRefreshManagerTest.java
@@ -211,7 +211,7 @@ public class IvmIncrRefreshManagerTest {
public void testManagerReturnsPlanSignatureMismatchFallback() throws
Exception {
MTMV mtmv = mockMtmv();
TestIvmIncrRefreshManager manager = new
TestIvmIncrRefreshManager(newContext(mtmv), Collections.emptyList());
- manager.planSignatureMismatch = new IvmPlanSignature("current plan",
"current-signature");
+ manager.planSignatureMismatch = new IvmPlanSignature("current plan",
"current-signature", "plan tree");
IvmIncrRefreshResult result = manager.doRefresh(manager.context);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java
index 8068d72be9f..5fd15e1edb7 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmNormalizeUniqKeysMTMVTest.java
@@ -128,24 +128,27 @@ class IvmNormalizeUniqKeysMTMVTest {
}
@Test
- void testIvmRewriteContextEnablesNormalizeWithoutSessionVariable() {
- JobContext jobContext = newJobContextForRoot(scan, false,
Collections.emptySet(),
+ void testCreateResultSinkNormalizesWithNoSignature() {
+ Plan sink = new
LogicalResultSink<>(ImmutableList.of(scan.getOutput().get(0)), scan);
+ JobContext jobContext = newJobContextForRoot(sink, false,
Collections.emptySet(),
Optional.of(IvmRewriteContext.create("test_mtmv")));
- Plan result = new IvmNormalizeUniqKeysMTMV().normalize(scan,
jobContext);
- Assertions.assertInstanceOf(LogicalProject.class, result);
+ Plan result = new IvmNormalizeUniqKeysMTMV().normalizeRoot(sink,
jobContext);
+
+ Assertions.assertInstanceOf(LogicalResultSink.class, result);
IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().orElseThrow();
- Assertions.assertNotNull(rewriteResult.getPlanSignature());
+ Assertions.assertNull(rewriteResult.getPlanSignature());
}
@Test
void testIncrementalSignatureMismatchThrowsAtNormalize() {
MTMV mtmv = mockIvmMtmv("stored-signature");
- JobContext jobContext = newJobContextForRoot(scan, true,
Collections.emptySet(),
+ Plan sink = signatureTestSink(mtmv);
+ JobContext jobContext = newJobContextForRoot(sink, true,
Collections.emptySet(),
Optional.of(IvmRewriteContext.incremental(mtmv)));
IvmException exception = Assertions.assertThrows(IvmException.class,
- () -> new IvmNormalizeUniqKeysMTMV().normalize(scan,
jobContext));
+ () -> new IvmNormalizeUniqKeysMTMV().normalizeRoot(sink,
jobContext));
Assertions.assertEquals(IvmFailureReason.PLAN_SIGNATURE_MISMATCH,
exception.getFailureReason());
Assertions.assertTrue(exception.getMessage().contains("storedSignature=stored-signature"));
@@ -153,20 +156,35 @@ class IvmNormalizeUniqKeysMTMVTest {
}
@Test
- void testIncrementalSignatureMatchPassesNormalize() {
- // Normalize once (CREATE mode) to learn the computed signature for
this plan.
- JobContext probe = newJobContextForRoot(scan, true);
- new IvmNormalizeUniqKeysMTMV().normalize(scan, probe);
- String signature =
probe.getCascadesContext().getIvmRewriteResult().orElseThrow()
- .getPlanSignature().getSha256();
-
- MTMV mtmv = mockIvmMtmv(signature);
- JobContext jobContext = newJobContextForRoot(scan, true,
Collections.emptySet(),
+ void testIncrementalWithoutSignaturePassesNormalize() {
+ MTMV mtmv = mockIvmMtmv(null);
+ Plan sink = signatureTestSink(mtmv);
+ JobContext jobContext = newJobContextForRoot(sink, true,
Collections.emptySet(),
Optional.of(IvmRewriteContext.incremental(mtmv)));
- Plan result = new IvmNormalizeUniqKeysMTMV().normalize(scan,
jobContext);
+ Plan result = new IvmNormalizeUniqKeysMTMV().normalizeRoot(sink,
jobContext);
- Assertions.assertInstanceOf(LogicalProject.class, result);
+ Assertions.assertInstanceOf(LogicalOlapTableSink.class, result);
+ IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().orElseThrow();
+ Assertions.assertNotNull(rewriteResult.getPlanSignature());
+ Assertions.assertNull(mtmv.getIvmInfo().getPlanSignature());
+ }
+
+ @Test
+ void testMowNormalizeRejectsRootWithoutSink() {
+ JobContext jobContext = newJobContextForRoot(scan, true,
Collections.emptySet(),
+ Optional.of(IvmRewriteContext.incremental(mockIvmMtmv(null))));
+
+ IvmException exception = Assertions.assertThrows(IvmException.class,
+ () -> new IvmNormalizeUniqKeysMTMV().normalizeRoot(scan,
jobContext));
+ Assertions.assertEquals(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
exception.getFailureReason());
+ Assertions.assertTrue(exception.getMessage().contains("requires a
table or result sink"));
+ }
+
+ private Plan signatureTestSink(MTMV mtmv) {
+ return new LogicalOlapTableSink<>(new Database(), mtmv,
ImmutableList.of(), ImmutableList.of(),
+ ImmutableList.copyOf(scan.getOutput()), false,
TPartialUpdateNewRowPolicy.APPEND,
+ DMLCommandType.INSERT, scan);
}
private static MTMV mockIvmMtmv(String signature) {
@@ -226,7 +244,6 @@ class IvmNormalizeUniqKeysMTMVTest {
Assertions.assertFalse(rowId.nullable());
IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().orElseThrow();
Assertions.assertTrue(rewriteResult.isDeterministic(rowId));
- Assertions.assertNotNull(rewriteResult.getPlanSignature());
}
@Test
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGeneratorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGeneratorTest.java
index 203f026d60f..bc15c659ff9 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGeneratorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/mtmv/ivm/IvmPlanSignatureGeneratorTest.java
@@ -54,6 +54,7 @@ import org.apache.doris.qe.ConnectContext;
import org.apache.doris.thrift.TPartialUpdateNewRowPolicy;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -104,29 +105,34 @@ class IvmPlanSignatureGeneratorTest extends
IvmDeltaTestBase {
}
@Test
- void testOlapTableSinkDoesNotChangeSignature() {
+ void testOlapTableSinkSignatureUsesTargetColumnNamesAndTypes() {
LogicalOlapScan scan = buildMowScan(1, "t");
- LogicalProject<?> project = buildRowIdProject(scan,
- ImmutableList.of(scan.getOutput().get(0),
scan.getOutput().get(1)));
- IvmPlanSignature withoutSink = signatureForNormalizedPlan(project);
+ LogicalProject<?> project = buildRowIdProject(scan,
scan.getOutput().subList(0, 2));
+ SlotReference intId = new SlotReference("id", IntegerType.INSTANCE);
+ SlotReference bigintId = new SlotReference("id", BigIntType.INSTANCE);
+ SlotReference renamedId = new SlotReference("renamed_id",
IntegerType.INSTANCE);
+
+ IvmPlanSignature signature = tableSinkSignature(scan, project,
ImmutableList.of(1L), intId);
+ Assertions.assertEquals(signature.getSha256(),
+ tableSinkSignature(scan, project, ImmutableList.of(2L),
intId).getSha256());
+ Assertions.assertNotEquals(signature.getSha256(),
+ tableSinkSignature(scan, project, ImmutableList.of(1L),
bigintId).getSha256());
+ Assertions.assertNotEquals(signature.getSha256(),
+ tableSinkSignature(scan, project, ImmutableList.of(1L),
renamedId).getSha256());
+ }
+ private IvmPlanSignature tableSinkSignature(LogicalOlapScan scan, Plan
child, List<Long> partitionIds,
+ Slot targetColumn) {
LogicalOlapTableSink<Plan> sink = new LogicalOlapTableSink<>(
- new Database(),
- scan.getTable(),
- scan.getTable().getBaseSchema(),
- ImmutableList.of(),
- ImmutableList.copyOf(project.getOutput()),
- false,
- TPartialUpdateNewRowPolicy.APPEND,
- DMLCommandType.NONE,
- project);
- IvmPlanSignature withSink = signatureForNormalizedPlan(sink);
-
- Assertions.assertEquals(withoutSink.getSha256(), withSink.getSha256());
+ new Database(), scan.getTable(),
scan.getTable().getBaseSchema(), partitionIds,
+ ImmutableList.copyOf(child.getOutput()), false,
TPartialUpdateNewRowPolicy.APPEND,
+ DMLCommandType.INSERT, child);
+ return signatureForNormalizedPlan(sink.withChildAndUpdateOutput(child,
ImmutableList.of(),
+ ImmutableMap.of(), ImmutableList.of(targetColumn)));
}
@Test
- void testPassthroughProjectsDoNotChangeSignature() {
+ void testPassthroughProjectsParticipateInSignature() {
LogicalOlapScan scan = buildMowScan(1, "t");
LogicalProject<?> normalizedProject = buildRowIdProject(scan,
ImmutableList.of(scan.getOutput().get(0),
scan.getOutput().get(1)));
@@ -145,57 +151,7 @@ class IvmPlanSignatureGeneratorTest extends
IvmDeltaTestBase {
.build(), slotPassthroughProject);
IvmPlanSignature passthroughSignature =
signatureForNormalizedPlan(aliasPassthroughProject);
- Assertions.assertEquals(baseSignature.getSha256(),
passthroughSignature.getSha256());
- }
-
- @Test
- void testSinkAdapterProjectDoesNotChangeSignature() {
- LogicalOlapScan scan = buildMowScan(1, "t");
- LogicalProject<?> normalizedProject = buildRowIdProject(scan,
- ImmutableList.of(scan.getOutput().get(0),
scan.getOutput().get(1)));
- IvmPlanSignature baseSignature =
signatureForNormalizedPlan(normalizedProject);
-
- LogicalProject<?> sinkAdapterProject = new
LogicalProject<>(ImmutableList.<NamedExpression>builder()
- .add(new Alias(new IntegerLiteral(0), Column.DELETE_SIGN))
- .add(new Alias(normalizedProject.getOutput().get(0),
Column.IVM_ROW_ID_COL))
- .add(new Alias(new IntegerLiteral(0), Column.VERSION_COL))
- .add(new Alias(new IntegerLiteral(0), Column.COMMIT_TSO_COL))
- .add(new Alias(normalizedProject.getOutput().get(1),
normalizedProject.getOutput().get(1).getName()))
- .add(new Alias(normalizedProject.getOutput().get(2),
normalizedProject.getOutput().get(2).getName()))
- .build(), normalizedProject);
-
- IvmPlanSignature sinkAdapterSignature =
signatureForNormalizedPlan(sinkAdapterProject);
- Assertions.assertEquals(baseSignature.getSha256(),
sinkAdapterSignature.getSha256());
- }
-
- @Test
- void testSinkWithAdapterProjectDoesNotChangeSignature() {
- LogicalOlapScan scan = buildMowScan(1, "t");
- LogicalProject<?> normalizedProject = buildRowIdProject(scan,
- ImmutableList.of(scan.getOutput().get(0),
scan.getOutput().get(1)));
- IvmPlanSignature baseSignature =
signatureForNormalizedPlan(normalizedProject);
-
- LogicalProject<?> sinkAdapterProject = new
LogicalProject<>(ImmutableList.<NamedExpression>builder()
- .add(new Alias(new IntegerLiteral(0), Column.DELETE_SIGN))
- .add(new Alias(normalizedProject.getOutput().get(0),
Column.IVM_ROW_ID_COL))
- .add(new Alias(new IntegerLiteral(0), Column.VERSION_COL))
- .add(new Alias(new IntegerLiteral(0), Column.COMMIT_TSO_COL))
- .add(new Alias(normalizedProject.getOutput().get(1),
normalizedProject.getOutput().get(1).getName()))
- .add(new Alias(normalizedProject.getOutput().get(2),
normalizedProject.getOutput().get(2).getName()))
- .build(), normalizedProject);
- LogicalOlapTableSink<Plan> sink = new LogicalOlapTableSink<>(
- new Database(),
- scan.getTable(),
- scan.getTable().getBaseSchema(),
- ImmutableList.of(),
- ImmutableList.copyOf(sinkAdapterProject.getOutput()),
- false,
- TPartialUpdateNewRowPolicy.APPEND,
- DMLCommandType.NONE,
- sinkAdapterProject);
-
- IvmPlanSignature sinkSignature = signatureForNormalizedPlan(sink);
- Assertions.assertEquals(baseSignature.getSha256(),
sinkSignature.getSha256());
+ Assertions.assertNotEquals(baseSignature.getSha256(),
passthroughSignature.getSha256());
}
@Test
@@ -391,7 +347,7 @@ class IvmPlanSignatureGeneratorTest extends
IvmDeltaTestBase {
@Test
void testAggregateSignatureUsesNormalizedHiddenOutputsWithoutAggMeta() {
PlanBundle bundle = normalizeAggPlan(buildGroupedAgg(buildMowScan(1,
"t")));
- IvmPlanSignature signature = bundle.rewriteResult.getPlanSignature();
+ IvmPlanSignature signature =
signatureForNormalizedPlan(bundle.normalizedPlan);
Assertions.assertFalse(signature.getCanonicalString().contains("AGG_META"));
Assertions.assertFalse(signature.getCanonicalString().contains("aggType="));
@@ -469,9 +425,7 @@ class IvmPlanSignatureGeneratorTest extends
IvmDeltaTestBase {
private IvmPlanSignature signatureForPlan(Plan root) {
ConnectContext ctx = newConnectContext();
JobContext jobContext = newJobContextForRoot(root, ctx);
- new IvmNormalizeUniqKeysMTMV().normalize(root, jobContext);
- IvmRewriteResult rewriteResult =
jobContext.getCascadesContext().getIvmRewriteResult().get();
- return rewriteResult.getPlanSignature();
+ return signatureForNormalizedPlan(new
IvmNormalizeUniqKeysMTMV().normalize(root, jobContext));
}
private IvmPlanSignature signatureForNormalizedPlan(Plan normalizedPlan) {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMVTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMVTest.java
index fff4cf2864d..ab3784430ea 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMVTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/IvmIncrRefreshMTMVTest.java
@@ -98,19 +98,6 @@ class IvmIncrRefreshMTMVTest {
Assertions.assertEquals(0, rule.rewriter.callCount);
}
- @Test
- void testNonSinkRootThrows() {
- RecordingRule rule = new RecordingRule(scan);
-
- IvmException exception = Assertions.assertThrows(IvmException.class,
- () -> rule.rewriteRoot(scan, newJobContext(scan,
- IvmRewriteContext.incremental(mtmv),
newRewriteResult(SIGNATURE))));
-
- Assertions.assertEquals(IvmFailureReason.PLAN_PATTERN_UNSUPPORTED,
exception.getFailureReason());
-
Assertions.assertTrue(exception.getMessage().contains("LogicalOlapTableSink"));
- Assertions.assertEquals(0, rule.rewriter.callCount);
- }
-
@Test
void testDryRunRewritesQueryRootAndCapsWithLimit() {
LogicalProject<Plan> deltaPlan = new LogicalProject<>(
@@ -119,7 +106,8 @@ class IvmIncrRefreshMTMVTest {
RecordingRule rule = new RecordingRule(deltaPlan);
IvmDryRunLimit dryRunLimit = new IvmDryRunLimit(3, 10);
- Plan result = rule.rewriteRoot(scan, newJobContext(scan,
+ Plan root = new
LogicalResultSink<>(ImmutableList.copyOf(scan.getOutput()), scan);
+ Plan result = rule.rewriteRoot(root, newJobContext(root,
IvmRewriteContext.incrementalDryRun(mtmv,
Optional.of(dryRunLimit)),
newRewriteResult(SIGNATURE)));
@@ -144,7 +132,8 @@ class IvmIncrRefreshMTMVTest {
scan.getOutput().get(0),
scan.getOutput().get(0).getName())), scan);
RecordingRule rule = new RecordingRule(deltaPlan);
- Plan result = rule.rewriteRoot(scan, newJobContext(scan,
+ Plan root = new
LogicalResultSink<>(ImmutableList.copyOf(scan.getOutput()), scan);
+ Plan result = rule.rewriteRoot(root, newJobContext(root,
IvmRewriteContext.incrementalDryRun(mtmv,
Optional.of(dryRunLimit)),
newRewriteResult(SIGNATURE)));
@@ -162,7 +151,8 @@ class IvmIncrRefreshMTMVTest {
scan.getOutput().get(0),
scan.getOutput().get(0).getName())), scan);
RecordingRule rule = new RecordingRule(deltaPlan);
- Plan result = rule.rewriteRoot(scan, newJobContext(scan,
+ Plan root = new
LogicalResultSink<>(ImmutableList.copyOf(scan.getOutput()), scan);
+ Plan result = rule.rewriteRoot(root, newJobContext(root,
IvmRewriteContext.incrementalDryRun(mtmv, Optional.empty()),
newRewriteResult(SIGNATURE)));
@@ -311,7 +301,7 @@ class IvmIncrRefreshMTMVTest {
private IvmRewriteResult newRewriteResult(String signature) {
IvmRewriteResult rewriteResult = new IvmRewriteResult();
- rewriteResult.setPlanSignature(new IvmPlanSignature("canonical",
signature));
+ rewriteResult.setPlanSignature(new IvmPlanSignature("canonical",
signature, "plan tree"));
return rewriteResult;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
index df810ff920c..93e6d491b15 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
@@ -496,7 +496,7 @@ public class CreateMTMVCommandTest extends
TestWithFeService {
Assertions.assertTrue(mtmv.isIvm());
Assertions.assertFalse(mtmv.isIvmAppendOnlyDupTable());
Assertions.assertNotNull(mtmv.getColumn(Column.SEQUENCE_COL));
- Assertions.assertNotNull(mtmv.getIvmInfo().getPlanSignature());
+ Assertions.assertNull(mtmv.getIvmInfo().getPlanSignature());
}
@Test
diff --git
a/regression-test/data/mtmv_p0/ivm/test_ivm_sink_truncation_signature.out
b/regression-test/data/mtmv_p0/ivm/test_ivm_sink_truncation_signature.out
new file mode 100644
index 00000000000..1099ab0844b
--- /dev/null
+++ b/regression-test/data/mtmv_p0/ivm/test_ivm_sink_truncation_signature.out
@@ -0,0 +1,23 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !first_refresh_method --
+SUCCESS PARTITION_REBUILD
+
+-- !after_first_incremental --
+app 1 80.00
+refund 2 -30.00
+store 1 50.00
+web 3 320.00
+
+-- !order_ivm_sink_truncation_dry_run --
+-116140974810391529965885828402163972140 refund 3 -60.00 3
3 [regression-fake-sequence] 0
+22103889325343121324737495639256262515 web 4 470.00 4 3
[regression-fake-sequence] 0
+
+-- !refresh_method --
+SUCCESS INCREMENTAL
+
+-- !after_incremental --
+app 1 80.00
+refund 3 -60.00
+store 1 50.00
+web 4 470.00
+
diff --git
a/regression-test/suites/mtmv_p0/ivm/test_ivm_sink_truncation_signature.groovy
b/regression-test/suites/mtmv_p0/ivm/test_ivm_sink_truncation_signature.groovy
new file mode 100644
index 00000000000..c4e5d62fac8
--- /dev/null
+++
b/regression-test/suites/mtmv_p0/ivm/test_ivm_sink_truncation_signature.groovy
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_ivm_sink_truncation_signature") {
+ sql "SET enable_materialized_view_rewrite = false"
+ sql "SET enable_dml_materialized_view_rewrite = false"
+ sql "SET enable_insert_value_auto_cast = true"
+ sql "SET enable_strict_cast = false"
+
+ sql "DROP MATERIALIZED VIEW IF EXISTS ivm_sink_truncation_mv"
+ sql "DROP TABLE IF EXISTS ivm_sink_truncation_sales"
+ sql "DROP TABLE IF EXISTS ivm_sink_truncation_refunds"
+
+ sql """
+ CREATE TABLE ivm_sink_truncation_sales (
+ sale_id BIGINT NOT NULL,
+ channel VARCHAR(16),
+ amount DECIMAL(12,2)
+ )
+ UNIQUE KEY(sale_id)
+ DISTRIBUTED BY HASH(sale_id) BUCKETS 1
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'enable_unique_key_merge_on_write' = 'true',
+ 'changelog.enable' = 'true',
+ 'changelog.need_historical_value' = 'true'
+ )
+ """
+
+ sql """
+ CREATE TABLE ivm_sink_truncation_refunds (
+ refund_id BIGINT NOT NULL,
+ amount DECIMAL(12,2)
+ )
+ UNIQUE KEY(refund_id)
+ DISTRIBUTED BY HASH(refund_id) BUCKETS 1
+ PROPERTIES (
+ 'replication_num' = '1',
+ 'enable_unique_key_merge_on_write' = 'true',
+ 'changelog.enable' = 'true',
+ 'changelog.need_historical_value' = 'true'
+ )
+ """
+
+ sql """
+ INSERT INTO ivm_sink_truncation_sales VALUES
+ (101, 'web', 120.00),
+ (102, 'app', 80.00),
+ (103, 'web', 200.00),
+ (104, 'store', 50.00),
+ (105, 'web', NULL)
+ """
+ sql "INSERT INTO ivm_sink_truncation_refunds VALUES (901, 10.00), (902,
20.00)"
+
+ sql """
+ CREATE MATERIALIZED VIEW ivm_sink_truncation_mv
+ BUILD DEFERRED REFRESH INCREMENTAL ON MANUAL
+ DISTRIBUTED BY RANDOM BUCKETS 1
+ PROPERTIES ('replication_num' = '1')
+ AS
+ SELECT u.channel, COUNT(*) AS row_count, SUM(u.amount) AS total_amount
+ FROM (
+ SELECT channel, amount FROM ivm_sink_truncation_sales
+ UNION ALL
+ SELECT 'refund' AS channel, -amount AS amount FROM
ivm_sink_truncation_refunds
+ ) u
+ GROUP BY u.channel
+ """
+
+ // Preview the first refresh, then let the actual write establish the
signature.
+ // The write plan still truncates the STRING union output to the target
VARCHAR length.
+ explain {
+ sql "ANALYZED PLAN REFRESH MATERIALIZED VIEW ivm_sink_truncation_mv
INCREMENTAL WITH ALL STREAMS"
+ contains "LogicalOlapTableSink"
+ contains "substring(channel#"
+ }
+
+ def refreshTaskQuery = """
+ SELECT Status, IvmRefreshMethod
+ FROM tasks('type' = 'mv')
+ WHERE MvDatabaseName = '${context.dbName}' AND MvName =
'ivm_sink_truncation_mv'
+ ORDER BY CreateTime DESC, CAST(TaskId AS BIGINT) DESC
+ LIMIT 1
+ """
+
+ sql "REFRESH MATERIALIZED VIEW ivm_sink_truncation_mv INCREMENTAL"
+ waitingMTMVTaskFinishedByMvName("ivm_sink_truncation_mv")
+ qt_first_refresh_method refreshTaskQuery
+ order_qt_after_first_incremental "SELECT channel, row_count, total_amount
FROM ivm_sink_truncation_mv"
+
+ sql "INSERT INTO ivm_sink_truncation_sales VALUES (106, 'web', 150.00)"
+ sql "INSERT INTO ivm_sink_truncation_refunds VALUES (903, 30.00)"
+ // A dry run has no table write projection, even after a write signature
exists.
+ ivm_hidden_qt("order_ivm_sink_truncation_dry_run", """
+ REFRESH MATERIALIZED VIEW ivm_sink_truncation_mv INCREMENTAL WITH DRY
RUN
+ """)
+ sql "REFRESH MATERIALIZED VIEW ivm_sink_truncation_mv INCREMENTAL"
+ waitingMTMVTaskFinishedByMvName("ivm_sink_truncation_mv")
+ qt_refresh_method refreshTaskQuery
+ order_qt_after_incremental "SELECT channel, row_count, total_amount FROM
ivm_sink_truncation_mv"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]