unbridled-41 opened a new issue, #5478:
URL: https://github.com/apache/rocketmq-dashboard/issues/5478
### Studio Version
branch: `rocketmq-studio` @ `6a68042f` (the revision verified; the branch is
based on it)
deployed as: built from source (unit test)
### Runtime Environment
Not environment-specific: the metric collector, verified with unit tests.
### Connected RocketMQ Cluster
An Apache instance where a broker stays in the NameServer topology while its
runtime stats cannot be read (down, restarting, or a missing master address).
### Describe the Bug
When a broker's runtime stats cannot be read at all, the collector records
only `broker.availability` and returns, so `broker.disk.usage_ratio`,
`broker.jvm.heap.usage_ratio` and `broker.send_queue.usage_ratio` produce no
sample. The collection scope declares all five keys regardless, and the alert
reconciler skips reconciliation only when an unavailable sample carries **the
rule's own metric key** — so an active rule on one of the three metrics
advances with `clear` and emits RESOLVED on the very pass that failed to read
the value; it flaps back when the broker recovers.
The parse-miss branches already record unavailable samples for these metrics
(the rationale accepted in #4521); the three early-return/exception branches
did not.
### Steps to Reproduce
1. Have an enabled CLUSTER rule on e.g. `broker.disk.usage_ratio` FIRING for
broker X.
2. Make X unreachable while it is still listed in the NameServer topology,
so `examineBrokerClusterInfo` succeeds and `fetchBrokerRuntimeStats` throws.
3. Observe the next collection pass resolve the incident.
### What Did You Expect to See?
A collection failure records an unavailable sample for each affected metric
and leaves the active alert alone, as `docs/studio-native-alerting-design.md`
describes and the parse-miss branches already do.
### What Did You See Instead?
The active alert is resolved by the collection that failed.
### Evidence
- `cluster/metrics/collectors/ApacheRocketMqClusterMetricsCollector.java`
(the three failure branches), `cluster/metrics/CollectorScheduler.java` (the
declared scope), `ops/alert/NativeAlertProcessor.java` (the coverage guard and
`advance(clear)`), `ops/alert/AlertStateMachine.java` ("A regular metric must
not clear an alert merely because collection failed").
- `cd server && mvn -o test
-Dtest=ApacheRocketMqClusterMetricsCollectorTest` — the added case fails before
the change (no `broker.disk.usage_ratio` sample) and passes after (`Tests run:
5`); the wider `cluster.metrics.**` + `ops.alert.**` sweep runs 473 tests with
0 failures.
### Impact
A false RESOLVED (and its notification) for an active infrastructure alert,
plus a flap on recovery.
### Acceptance Criteria
- The failure branches record an unavailable sample for each runtime metric
with the same labels.
- A regression test pins it.
Related: #4519 (the same defect, fixed for the parse-miss branches by #4521).
**Corresponding PR:** #5449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
