morningman opened a new pull request, #69868:
URL: https://github.com/apache/doris/pull/69868
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** The BEs clean up the changelog of a table by TTL, but
`/metrics` cannot tell whether that cleanup runs, fails, or reclaims anything.
This PR counts each cleanup on the BE as it finishes, in three new series: how
many succeeded or failed, how many bytes of expired changelog they dropped, and
when one last succeeded.
**Background.** A table with `changelog.enable` keeps its changelog in a
hidden changelog index, and `changelog.ttl_seconds` bounds how long the
changelog is kept. The BEs remove what is older than that in a cumulative
compaction of a tablet of that index (a changelog tablet): rowsets whose commit
TSO is at or before the TTL cutoff are left out of the merge, so the output
replaces them without their rows. Two submitters start these compactions, both
as `CHANGELOG_COMPACTION`: the changelog TTL scanner, for tablets it finds
holding expired rowsets, and the regular compaction producer; either compaction
picks expired rowsets first.
**The problem, and what it cost.**
- No alert can watch the cleanup. The bvars `changelog_ttl_discovered`,
`changelog_ttl_submitted` and `changelog_ttl_failed` are only on the brpc
`/vars` page, and they count what the TTL scanner found and submitted, not what
the compactions did: a cleanup that fails after it was submitted counts
nowhere, and nothing counts the bytes removed.
- The only changelog series in `/metrics`,
`compaction_task_state_total{type="changelog"}`, is the number of changelog
compactions running and pending. A cleanup that keeps failing lets the
changelog grow without bound, and nothing on the BE says why.
**How this PR fixes it.**
- Three series, counted as a compaction that drops expired changelog ends,
and registered at startup so they exist from 0:
| Series | Type | Labels | Value |
| --- | --- | --- | --- |
| `doris_be_changelog_ttl_cleanup_total` | counter | `result`=success/failed
| the compactions that drop expired changelog rowsets, counted as each ends |
| `doris_be_changelog_ttl_reclaimed_bytes_total` | counter | | the total
disk size of the expired rowsets the successful ones dropped |
| `doris_be_changelog_ttl_last_success_timestamp_seconds` | gauge | | when
one last succeeded, in seconds since the epoch |
- A compaction is a cleanup when one of its inputs is expired changelog: at
or below the visible version its TTL snapshot was taken at, with a commit TSO
at or before the cutoff. That is exactly what `filter_changelog_ttl_rowsets`
leaves out of the merge, and the two now ask one predicate,
`Compaction::is_expired_changelog_rowset`, which
`pick_expired_changelog_rowset` uses too. It is decided from the inputs rather
than from what the merge filtered, so that a cleanup that fails before its
merge, a cloud compaction that cannot refresh its tablet meta for example, is
counted as failed.
- Counted where the execution of the compaction ends:
`CumulativeCompaction::execute_compact` in shared-nothing mode,
`CloudCumulativeCompaction::record_compaction_success` and
`record_compaction_failure` in cloud mode. A compaction that does not get past
its preparation (the tablet is under another cumulative compaction, the meta
service refuses the job) has not started a cleanup and is not counted.
- In cloud mode the BE that runs the compaction counts it, so a query sums
over the BEs; the shared storage is not measured. The dropped rowsets become
stale rowsets, whose files the stale rowset sweep or the recycler deletes
later; the bytes are counted when the rowsets leave the tablet.
- The bvars stay as they are.
**Results.** Before this PR, the `/metrics` of a BE has none of these
series. After it, on a local shared-nothing cluster, a table with
`changelog.ttl_seconds` = 1 and two rows inserted (regression
`test_changelog_ttl_cleanup_metrics`):
| | `cleanup_total{result="success"}` | `cleanup_total{result="failed"}` |
`reclaimed_bytes_total` | `last_success_timestamp_seconds` |
| --- | --- | --- | --- | --- |
| BE started | 0 | 0 | 0 | 0 |
| once the changelog rows have been cleaned up | 1 | 0 | 1112 | the time of
the cleanup |
The 1112 bytes are what the BE logs for that changelog compaction:
`input_rowsets_total_size=1.09 KB`, `output_rowset_total_size=0`,
`filtered_row_num=2`.
**Classes, and how they call each other.**
- `DorisMetrics`: the three series.
- `Compaction`: `is_expired_changelog_rowset` (new), shared by
`pick_expired_changelog_rowset` and `filter_changelog_ttl_rowsets`;
`record_changelog_ttl_cleanup` (new) counts a cleanup from the inputs.
- `CumulativeCompaction`, `CloudCumulativeCompaction`: call it as the
execution ends.
```
CumulativeCompaction::execute_compact shared-nothing
└─ CompactionMixin::execute_compact ─► merge: filter_changelog_ttl_rowsets
─┐
└─ record_changelog_ttl_cleanup(ok)
│ is_expired_changelog_rowset
CloudCumulativeCompaction::execute_compact cloud
│
└─ CloudCompactionMixin::execute_compact ─► merge: same filter
─────────────┤
└─ record_compaction_success ─► record_changelog_ttl_cleanup(true)
│
record_compaction_failure ─► record_changelog_ttl_cleanup(false)
│
└─ DorisMetrics::changelog_ttl_*
◄─────────┘ (from the inputs)
```
Nothing else changes: no config, no thrift or proto change, and what the
compactions pick and drop stays the same.
### Release note
New BE metrics `doris_be_changelog_ttl_cleanup_total{result}`,
`doris_be_changelog_ttl_reclaimed_bytes_total` and
`doris_be_changelog_ttl_last_success_timestamp_seconds`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit test: `CumulativeCompactionTest.TestRecordChangelogTtlCleanup`
checks what counts as a cleanup (not a compaction whose inputs hold no expired
changelog, nor one that took no TTL snapshot; a rowset above the visible
version of the snapshot, or an empty one, is not expired changelog) and that
the reclaimed bytes are those of the rowsets `filter_changelog_ttl_rowsets`
drops;
`CompactionMetricsTest.TestChangelogTtlCleanupCountedAsCompactionFinishes` runs
`CumulativeCompaction::execute_compact` up to its merge and checks a success
and a failure;
`CloudCompactionTest.changelog_ttl_cleanup_counted_as_compaction_finishes` does
the same through `record_compaction_success` and `record_compaction_failure`.
With the success hooks removed, the two that go through them fail. Together
with the other compaction, changelog TTL and `DorisMetrics` suites, 114 cases
pass and one is skipped outside Release.
Regression test: `changelog_p0/test_changelog_ttl_cleanup_metrics`
(nonConcurrent: it waits for the TTL to expire), on a local shared-nothing
cluster; not run in cloud mode.
Manual test: `promtool check metrics` on the `/metrics` of the BE
reports no parsing error; the only finding on the new series is "no help text",
which every BE series gets (the BE writes no HELP lines).
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. The doris-website change comes with the documentation of the
changelog and IVM monitoring.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
