unbridled-41 opened a new pull request, #5420:
URL: https://github.com/apache/rocketmq-dashboard/pull/5420
## Problem / Evidence
The bundled `RocketMQDLQResendHigh` rule
(`server/src/main/resources/alerts/rocketmq-dlq-resend-high.yaml:9`, generated
by `server/scripts/gen_alert_rule_yaml.py:76`) annotates itself as:
> "More than 10 dead-letter queue resends occurred in 5 minutes."
but its expression is a per-second rate:
```
rate(rocketmq_dlq_resend_count[5m]) > 10
```
`rate()` returns the average increase per second, so the rule only fires
above ~10 resends/second — roughly 3000 resends inside the 5-minute window it
documents. A DLQ that is quietly re-driving dead letters at tens per minute,
which is already abnormal, never trips it.
The sibling rule expressing the identical "more than N in 5 minutes"
contract, `rocketmq-consumer-rebalance.yaml`, uses
`increase(rocketmq_consumer_rebalance_times[5m]) > 10`, and
`rocketmq-threadpool-reject.yaml` uses an `increase()` window too. The DLQ rule
is the outlier.
Reachable surface: `AlertRuleAssetService.loadDefaultRules()` feeds
`AlertService.exportPrometheusRulesYaml()` (the default rule set exported when
no custom business rules exist), and the raw asset is served by `GET
/api/alert-rules/assets/{name}` / `/{name}/export`.
## Root cause / Fix
The annotation describes a count over a window; the expression thresholds a
per-second rate. Fix: use the windowed increase, matching the annotation and
the sibling rules:
```
increase(rocketmq_dlq_resend_count[5m]) > 10
```
Changed in the generator, the checked-in asset, and the dev-mode mirror in
`web/src/mock/alertRuleAssets.ts`.
## Priority & scoring
PRIORITY 58 = 影响 18 (a shipped default rule silently never fires at the
documented sensitivity, leaving DLQ replay storms unalerted) + 波及 8 (one asset
+ generator + mock, exported as the default set) + 可复现 20 (deterministic
semantics of rate vs increase) + 维护价值 12.
FIX_CONFIDENCE 90: one expression change, regression-tested against the
shipped asset; no other rule fields touched.
## Tests
```
cd server && mvn -o test -Dtest=AlertRuleAssetServiceTest
```
- Red before the fix: `expected: <increase(rocketmq_dlq_resend_count[5m]) >
10> but was: <rate(rocketmq_dlq_resend_count[5m]) > 10>`
- Green after: `Tests run: 12, Failures: 0, Errors: 0, Skipped: 0`
- Regenerating all 23 assets reproduces the checked-in directory byte for
byte (`diff -rq` clean).
- `mvn -o checkstyle:check` passes.
## Risk
Low. Only the `expr` of one bundled asset changes; alert name, group,
severity, duration and annotations are untouched, so rule fingerprints/state
keys are unaffected. An operator who already exported the previous expression
keeps it until they re-export.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
