morningman opened a new pull request, #69848:
URL: https://github.com/apache/doris/pull/69848
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
**In short.** Table streams and IVM materialized views export no metrics,
and the signals an operator needs from them (is an MV fresh, is a stream about
to fall behind its changelog TTL) are too costly to compute while `/metrics` is
scraped. This PR adds a daemon of the Master FE that samples into an immutable
snapshot, renders the snapshot in `/metrics`, and puts the first series on it:
the changelog kept by the tables, and the health of the sampling itself. The
state of the streams and of the IVM materialized views comes with follow-up PRs
on this framework.
**Background.** A table with `changelog.enable` keeps a changelog, which
table streams and IVM materialized views consume. Their state shows only in
system tables (`table_streams`, `table_stream_consumption`, `mv_infos()`),
which have to be queried one by one, and `SHOW DATA` lists the changelog size
of each table as `ChangelogSize`.
**The problem, and what it cost.**
- No alert can watch the changelog: nothing in `/metrics` says how much
changelog the tables keep, or whether the streams and IVM materialized views
keep up with it.
- The coming signals need per-partition state: offsets, visible versions,
and in cloud mode a MetaService call. `MetricRepo.getMetric` renders the whole
`/metrics` response under one lock, so computing them there would make every
scrape pay for a walk over the metadata.
- A gauge that every FE exports and a non-Master FE sets to 0, the way the
routine load lag does, drags a `min` or `max` across the FEs to 0.
**How this PR fixes it.**
- `ChangelogIvmMonitor`, a daemon of the Master FE: every
`changelog_ivm_monitor_interval_seconds` it samples, builds an immutable
`ChangelogIvmMonitorSnapshot`, and replaces the previous one at once. A round
that fails keeps the previous snapshot and counts the failure.
- `MetricVisitor.visitChangelogIvmMonitor`, beside `visitCloudTableStats`,
renders the series from the latest snapshot. Only the Master exports them:
another FE exports none of them rather than 0, and a new Master exports none of
the sampled series until its first round completes. The JSON and core outputs
of `/metrics` render nothing.
- The first series:
| Series | Type | Value |
| --- | --- | --- |
| `doris_fe_changelog_retained_bytes{scope="single_replica"}` | gauge | the
changelog kept by all the tables, counting one replica of each tablet |
| `doris_fe_changelog_retained_bytes{scope="all_replicas"}` | gauge | the
same over all the replicas, which is what `SHOW DATA` lists as `ChangelogSize`;
not exported in cloud mode, where a tablet keeps one copy in the shared storage
|
| `doris_fe_changelog_ivm_monitor_last_sample_timestamp_seconds` | gauge |
when the latest round completed |
| `doris_fe_changelog_ivm_monitor_sample_duration_seconds` | gauge | how
long the latest round took |
| `doris_fe_changelog_ivm_monitor_sample_failures_total` | counter | the
rounds that failed since this FE became the Master |
The last three let an alert tell stale data (`time() -
doris_fe_changelog_ivm_monitor_last_sample_timestamp_seconds > 120`, or the
series absent) from a healthy state.
- Two FE configs, mutable, Master-only and EXPERIMENTAL like the other table
stream configs: `enable_changelog_ivm_monitor` (true), which takes effect with
`enable_feature_changelog` only, so a cluster without the changelog feature
exports nothing new; and `changelog_ivm_monitor_interval_seconds` (15), which
the daemon checks every second, so a change applies at once.
**Results.**
| Case | Before | After |
| --- | --- | --- |
| `/metrics` of the Master, `enable_feature_changelog` on | no changelog
series | the five series above |
| `/metrics` of any other FE | — | none of them |
| `enable_feature_changelog` or `enable_changelog_ivm_monitor` off | — |
none of them, and no sampling |
| `all_replicas` against the `ChangelogSize` total of `SHOW DATA PROPERTIES
("entire_warehouse" = "true")` | — | equal (1340323 bytes on both, local
cluster) |
**Classes, and how they call each other.**
- `ChangelogIvmMonitor` (`org.apache.doris.changelog`): the daemon.
`buildSnapshot` sums `OlapTable.getChangelogSize(true)` and
`getChangelogSize(false)` over the tables of the internal catalog, from the
statistics that `TabletStatMgr` or `CloudTabletStatMgr` refreshes.
- `ChangelogIvmMonitorSnapshot` (`org.apache.doris.changelog`): what one
round sampled; never changes once built.
- `MetricVisitor.visitChangelogIvmMonitor`: implemented by
`PrometheusMetricVisitor`, empty in `JsonMetricVisitor` and
`SimpleCoreMetricVisitor`; called by `MetricRepo.getMetric` after
`visitCloudTableStats`.
- `Env`: holds the monitor and starts it with the other daemons of the
Master.
```
Env.startMasterOnlyDaemonThreads
└─ ChangelogIvmMonitor.start() checks every second; a round each
changelog_ivm_monitor_interval_seconds
└─ buildSnapshot() sums
OlapTable.getChangelogSize(true / false)
└─ volatile ChangelogIvmMonitorSnapshot replaced as a whole
▲
MetricRepo.getMetric ─► PrometheusMetricVisitor.visitChangelogIvmMonitor()
Master only
```
Nothing else changes: no persisted metadata, no thrift or proto change, and
the existing series are untouched.
### Release note
With `enable_feature_changelog` on, the Master FE exports
`doris_fe_changelog_retained_bytes{scope}` and the health of its sampling,
`doris_fe_changelog_ivm_monitor_last_sample_timestamp_seconds`,
`doris_fe_changelog_ivm_monitor_sample_duration_seconds` and
`doris_fe_changelog_ivm_monitor_sample_failures_total`. New FE configs
`enable_changelog_ivm_monitor` and `changelog_ivm_monitor_interval_seconds`.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit test: `ChangelogIvmMonitorTest` checks that a round sums the
changelog that `SHOW DATA` reads from the table statistics, that a failed round
keeps the previous snapshot and counts the failure, and that rounds follow the
interval, a shorter interval applies at the next check, and a disabled monitor
samples nothing and drops its snapshot. `ChangelogIvmMonitorMetricsTest` checks
the exact text of the series, also through `MetricRepo.getMetric`; only the
failure counter before the first round; nothing on a Follower, or with the
monitor or the changelog feature disabled; `single_replica` only in cloud mode;
nothing in the JSON and core outputs.
Regression test: `changelog_p0/test_changelog_ivm_monitor_metrics`, run
on a local shared-nothing cluster (not in cloud mode). On the `/metrics` of the
Master, `all_replicas` (`single_replica` in cloud mode) equals the
`ChangelogSize` total of `SHOW DATA PROPERTIES ("entire_warehouse" = "true")`;
the health series are present and fresh; no other FE exports them; nothing once
the monitor is disabled.
Manual: `promtool check metrics` on the `/metrics` of the Master reports
no parsing error and no lint finding on the new series.
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
With `enable_feature_changelog` on, the `/metrics` of the Master FE has
the new series, and a daemon of the Master sums the table statistics every 15
seconds. Nothing changes otherwise.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
Together with the rest of the changelog and IVM monitoring, in a
follow-up doc PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
