morningman opened a new pull request, #69785:
URL: https://github.com/apache/doris/pull/69785
### What problem does this PR solve?
Issue Number: None
Related PR: #69772
Problem Summary:
**In short.** An IVM materialized view over a duplicate-key base table
silently misses rows once the table's `changelog.ttl_seconds` has expired its
stream offset: the refresh succeeds, the rows written between the offset and
the cutoff are never read, and a base-table query of the MV's shape that
transparent rewrite answers from the MV returns the same wrong result. This PR
makes every read of an IVM refresh reject an expired offset, whatever its scan
type, so the refresh fails with `CHANGELOG_BROKEN` or, when it may fall back,
rebuilds the MV with COMPLETE (#69772). It also carries the two follow-ups
yujun777 suggested on #69772.
**Background.** An IVM MV reads each base table through an internal table
stream. A duplicate-key table is read `APPEND_ONLY`:
`OlapTableStreamWrapper.getStreamScanType()` returns `APPEND_ONLY` for
`DUP_KEYS`, whatever type the stream declares.
`NormalizeOlapTableStreamScan.applyChangelogTtl` decides what an expired start
means: a `MIN_DELTA` read is rejected (#69772 turns that into
`CHANGELOG_BROKEN` and COMPLETE), any other read is clipped to the first TSO
the changelog retains. The clipping is right for a user stream, which reads the
changelog it retains.
**The problem, and what it cost.**
- For an IVM refresh the clipping loses data without a trace. The rows
written between the offset and the cutoff are never read, the refresh succeeds,
the offset moves past them, and no later incremental refresh brings them back.
- It hits every IVM MV over a duplicate-key table, the aggregate ones
(unique-key MV) and the append-only detail ones alike, whenever the MV is not
refreshed within the TTL of a write.
**How this PR fixes it.**
1. `applyChangelogTtl` rejects an expired start for every read of an IVM
statement (the statement carries an `IvmRewriteContext`), whatever the scan
type. The condition is the statement, not the type the stream declares: a user
stream on a duplicate-key table also declares `MIN_DELTA` by default, and it
keeps its clipping. An IVM statement never reads a user stream: an MV query
that scans a stream is rejected when the MV is created. The snapshot read of a
duplicate-key table is bounded by the offset on the base table itself and reads
no changelog, so it is unchanged.
2. The follow-ups from the review of #69772, in their own commit and without
a behavior change:
- the rebuild an incremental attempt runs before its delta now turns a
failure into a fallback when its reason's only repair is COMPLETE
(`IvmFailureReason.requiresCompleteRefresh()`), and passes that reason through,
instead of naming `CHANGELOG_BROKEN`;
- the `CHANGELOG_BROKEN` branch of `handleIvmFallbackResult`, which
returned the same result as the `requiresCompleteRefresh()` branch after it, is
removed; why only COMPLETE repairs an expired changelog now sits on
`IvmFailureReason.CHANGELOG_BROKEN`.
**Results** (local shared-nothing cluster; a duplicate-key table, a row
written after the MV's offset, then `changelog.ttl_seconds` set to 1 and two
seconds of waiting):
| Refresh after the offset expired | Before | After |
| --- | --- | --- |
| `REFRESH ... INCREMENTAL` (strict) | SUCCESS, the row missing from the MV
| FAILED, `IvmFallbackReason = CHANGELOG_BROKEN`, the MV keeps its rows |
| `REFRESH ... AUTO` | SUCCESS with nothing left to read; the row stays
missing | SUCCESS through COMPLETE, `IvmFallbackReason = CHANGELOG_BROKEN`, the
MV matches the base table |
| the next refresh, with the TTL back to a day | incremental; the row
missing for good | incremental |
| a user stream on the same table | reads from the first retained TSO | the
same |
Both an aggregate MV and an append-only detail MV over the table behave as
above.
**Classes, and how they call each other.**
- `IvmIncrRefreshMTMV` → `IvmDeltaRewriteState.createDeltaScan` →
`LogicalOlapTableStreamScan` →
`NormalizeOlapTableStreamScan.makeTableStreamScan` →
`makeIncrementalScanFromChangelog` → `applyChangelogTtl` →
`ChangelogUtils.effectiveStartTso`, which now rejects for an IVM statement.
- The rejection reaches `IvmIncrRefreshManager.doRefresh` →
`IvmFailureClassifier.classifyExecutionFailure` → `CHANGELOG_BROKEN` →
`MTMVTask.handleIvmFallbackResult` → strict: failure with the reason; otherwise
`FALLBACK_TO_COMPLETE`.
- The rebuild before the delta: `MTMVTask.executeIvmAttempt` →
`executePartitionBasedRefresh` → on a failure whose reason
`requiresCompleteRefresh()`, `handleIvmFallbackResult` with that reason.
Not changed: user streams, direct incremental queries
(`BindRelation.applyChangelogTtl`), the snapshot and reset reads, and anything
persisted. A partition with no stream offset is a separate problem, fixed by
#69778.
### Release note
Fix an IVM materialized view over a duplicate-key base table silently
missing rows after the table's `changelog.ttl_seconds` expired the stream
offset. A refresh that allows fallback now rebuilds the MV with COMPLETE; a
strict INCREMENTAL refresh fails with reason `CHANGELOG_BROKEN`.
### Check List (For Author)
- Test: Regression test / Unit Test
- Regression test: `mtmv_p0/ivm/test_ivm_changelog_offset_expired_dup`
(new, nonConcurrent): an aggregate MV and an append-only detail MV over one
duplicate-key table, strict and AUTO refreshes after the offset expired and a
refresh after the TTL is restored, each MV compared with the base table; and a
user stream on the table, which still reads past its expired offset. Local
shared-nothing cluster: first on a build without this change, where both strict
refreshes succeeded without the row and the MVs never caught up; with the
change the suite passes, and passes again against the generated `.out`. With
both commits, the whole `mtmv_p0/ivm` directory passes (114 suites,
`test_ivm_changelog_offset_expired` of #69772 included). Not run on a cloud
cluster.
- Unit Test: `ExplainTableStreamPlanTest` (two new cases: a user stream
read `APPEND_ONLY` clips its expired offset to the first retained TSO; the same
read in an IVM statement is rejected) and `MTMVTaskTest` (a new case: a rebuild
failure that COMPLETE does not repair still fails an attempt that may fall
back), run with the IVM unit tests (`org.apache.doris.mtmv.ivm`,
`CreateMTMVCommandTest`, `ShowCreateMTMVTest`, `DropMaterializedViewTest`,
`ChangelogUtilsTest`): 630 tests pass.
- Behavior changed: Yes. An IVM refresh over a duplicate-key table whose
stream offset expired falls back to COMPLETE, or fails if it is strict, instead
of succeeding without the rows. This includes a partition that was only idle
for longer than the TTL before a new write, which an IVM MV over a unique-key
table already handles this way; keeping the offsets of caught-up partitions
alive is a separate change.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
