morningman opened a new pull request, #69831:
URL: https://github.com/apache/doris/pull/69831
### What problem does this PR solve?
Issue Number: None
Related PR: #69772, #69785
Problem Summary:
**In short:** an IVM materialized view rebuilds itself completely whenever
one of its base table partitions is written after being idle for longer than
`changelog.ttl_seconds` (one day by default), although nothing the refresh
needs has expired. This PR has the Master record, for such idle partitions, a
keep-alive TSO up to which the partition is known to hold no change, and the
refresh checks the TTL against it. Both shared-nothing and cloud mode are
covered; MetaService is not changed.
#### Background
An IVM MV reads each base table through an internal table stream, which
keeps a consumption offset (a TSO) per partition. A refresh reads the
partition's changelog after the offset and moves the offset to what it read. A
table's changelog is kept for `changelog.ttl_seconds`; a read whose offset is
older than `now - TTL` may have lost changes, so an IVM read refuses it
(`CHANGELOG_OFFSET_EXPIRED`), and since #69772 an AUTO refresh answers that as
`CHANGELOG_BROKEN` and falls back to COMPLETE (a strict INCREMENTAL refresh
fails). Since #69785 the same holds for an IVM MV over a duplicate-key table,
whose expired offset used to be clipped silently.
#### The problem, and what it cost
- The offset of a partition moves only when a refresh consumes the
partition, and a refresh only consumes partitions with new data. A partition
nobody writes keeps the offset of its last change.
- The TTL check only compares that offset with the cutoff; it does not know
whether anything was written in between. So once a partition has been idle for
longer than the TTL, the first write to it makes the next refresh refuse the
offset, although the only changes after it are the ones just written, all
retained.
- The cost is a COMPLETE refresh of the whole MV (or a failed strict
refresh) for a dimension table updated weekly, a backfill into an older
partition, or a quiet weekend.
#### How this PR fixes it
- **Keep-alive TSO K.** For a partition at its end, the Master records K,
meaning the partition holds no change in (offset, K]. The read of a refresh
moves its start to after K before the TTL check
(`NormalizeOlapTableStreamScan.applyChangelogTtl`). The rows read are the same,
and only retained changelog is read. The offset itself does not move, so the
invariants built on "offset == partition TSO means caught up" stay as they are.
- **When.** `IvmStreamKeepAliveDaemon`, a Master daemon, runs a round every
`ivm_stream_keepalive_interval_second` (600) over the IVM streams, found from
the MV side. A partition qualifies when the base table applies a TTL, the
stream has a real offset for it past the initial snapshot, nothing is committed
to it after the offset, and the newer of the offset and K has less than half
the TTL left.
- **Shared-nothing mode.** K is a TSO taken while the base table read lock
is held; a commit takes its TSO inside the base table write lock, so every
commit the partition has not had yet gets a larger one. A partition with a
committed transaction that is not visible yet does not qualify.
- **Cloud mode.** The offsets live in MetaService. K is the TSO service's
committed watermark, taken *before* the offsets are read through
`get_table_stream_offset` (which waits for pending transactions). Every commit
at or below the watermark has finished before the read, and a commit that lands
later gets a larger TSO, so a partition whose offset is at its end in the read
holds no change up to the watermark, for good. The read goes in batches of at
most 10000 partitions.
- **K is bound to its offset.** K is recorded in FE metadata with the offset
it was taken against, and a read uses it only when it starts from that offset
(in cloud mode, the offset the read state carries). An offset that moved, or
one MetaService reports differently, for example after a restore, leaves K out.
- **Persistence.** A new GSON field of `OlapTableStream`; the entries are
logged in a new optional list of `OP_TABLE_STREAM_CLEANUP` and replayed the
same way; K of a dropped partition is pruned with its offsets.
#### Results
The new regression suite `mtmv_p0/ivm/test_ivm_stream_keepalive`
(shared-nothing, run locally): an aggregate MV over a merge-on-write table and
an append-only detail MV over a duplicate-key table, both base tables with
`changelog.ttl_seconds = 60` and a five-second check. Each row is what
`tasks()` reports for the refresh (`Status`, `RefreshMode`,
`IvmFallbackReason`), for both MVs, and every MV matches its query over the
base table.
| Scenario | Keep-alive off (the behavior before this PR) | Keep-alive on |
| --- | --- | --- |
| Idle for 70 seconds, longer than the TTL, then written; `REFRESH ... AUTO`
| `SUCCESS COMPLETE CHANGELOG_BROKEN`: the whole MV is rebuilt | `SUCCESS`,
incremental, no fallback |
| A write lands while a keep-alive round is held (debug point) between what
it has read and what it would record | -- | nothing is recorded for the
partition; the refresh after it is incremental and reads the write |
The FE log of that run shows the keep-alive TSO renewed every thirty
seconds, half the TTL, while the partitions were idle, and nothing recorded for
the held stream once released.
#### Classes, and how they call each other
- `IvmStreamKeepAliveDaemon` (new): the Master daemon. It checks every
second whether `ivm_stream_keepalive_interval_second` has passed since its last
round, so that a change of the interval applies at once, and finds the IVM
streams through `IvmUtil.getIvmStreams` (new, which `mv_infos()` now uses too).
- `TableStreamManager.keepAliveIdlePartitions` /
`keepAliveIdleCloudPartitions` (new): one round in each mode; the
shared-nothing one takes the locks in the order `cleanupStalePartitionOffsets`
takes them.
- `OlapTableStream`: the new `partitionKeepAlive` map (GSON `pka`) of
`PartitionKeepAlive(offset, keepAliveTso)`, `unprotectedKeepAlive`,
`effectiveOffset`; pruned with the offsets of a dropped partition.
- `TableStreamCleanupInfo`: the new optional `keepAliveEntries` of
`OP_TABLE_STREAM_CLEANUP`.
- `OlapTableStreamWrapper.getKeepAliveTso` and
`NormalizeOlapTableStreamScan.applyChangelogTtl`: the read side.
```
IvmStreamKeepAliveDaemon (Master, every ivm_stream_keepalive_interval_second)
|-- shared-nothing: TableStreamManager.keepAliveIdlePartitions(stream)
| stream read lock -> base table read lock -> TSOService.getTSO() =
K -> stream write lock
| -> OlapTableStream.unprotectedKeepAlive({partition: offset}, K)
| ->
EditLog.logTableStreamCleanup(TableStreamCleanupInfo.ofKeepAlive(...))
`-- cloud: TableStreamManager.keepAliveIdleCloudPartitions(streams)
TSOService.getStatusSnapshot().getCommittedTso() = K
-> CloudTableStreamReadStateHelper.getReadStates(batch)
(MetaService get_table_stream_offset)
-> stream write lock -> unprotectedKeepAlive ->
logTableStreamCleanup
replay: OP_TABLE_STREAM_CLEANUP ->
TableStreamManager.replayTableStreamCleanup -> replayKeepAlive
read: IvmDeltaRewriteState / BindRelation -> new
OlapTableStreamWrapper(...) (captures (offset, K))
[cloud: CloudTableStreamReadStateHook -> installCloudReadStates]
(offset from MetaService)
NormalizeOlapTableStreamScan.applyChangelogTtl
-> OlapTableStreamWrapper.getKeepAliveTso(p) (K only when the
read starts from its offset)
-> start = max(start, K) ->
ChangelogUtils.effectiveStartTso(start, cutoff, reject)
```
Not changed: MetaService, the protos, the offsets themselves, user streams,
and the direct incremental query (`BindRelation.applyChangelogTtl`), which has
no offset.
### Release note
An IVM materialized view no longer falls back to a COMPLETE refresh when a
base table partition is written after being idle for longer than
`changelog.ttl_seconds`. New FE configs `enable_ivm_stream_keepalive` (default
true) and `ivm_stream_keepalive_interval_second` (default 600), both mutable.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- `test_ivm_stream_keepalive` (new, nonConcurrent): the cases in Results.
Run locally in shared-nothing mode, together with the whole `mtmv_p0/ivm`
directory (116 suites, all pass). Not run locally in cloud mode; `mtmv_p0` is
part of the cloud pipeline, where the same suite takes the cloud branch.
- `TableStreamManagerKeepAliveTest` (new, 16 cases): the conditions of
both modes (a change after the offset, a committed transaction that is not
visible, the initial snapshot, the margin, no TTL or one that never expires an
offset; in cloud mode a state that is not at its end or has no real offset, a
watermark not past the offset, no calibrated TSO, a failed read, reads in
batches), K taken against another offset not counting, the wrapper using K only
for a read from its offset in both modes, replay, pruning, and state written
before K.
- `ExplainTableStreamPlanTest`: an IVM read of an expired offset with a
fresh K is admitted and starts after K; one whose K expired too, or was taken
against another offset, is rejected.
- The IVM unit tests, every `catalog/stream` test class and the other
classes using the changed code (47 classes, 677 cases) pass; checkstyle passes
for fe-core and fe-common.
- Behavior changed:
- [ ] No.
- [x] Yes. A refresh reads an idle partition incrementally where it used
to fall back to COMPLETE (or fail, if strict).
- Does this need documentation?
- [ ] No.
- [x] Yes. The two FE configs; documentation will follow.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
