The GitHub Actions job "CI" on rocketmq-dashboard.git/fix/alert-pending-anchor 
has failed.
Run started by GitHub user unbridled-41 (triggered by unbridled-41).

Head commit for run:
d9e241cbcf725fd093000599c5e30d07f5ea487a / unbridled-41 
<[email protected]>
fix(alert): end a pending episode whose metric is no longer collected

A PENDING state is an anchor for the "must hold for duration / N samples"
contract, but nothing ever ends it when the metric stops arriving:

- the reconcile loop was the only thing that could, and it iterates
  `stateRepository.findActive(...)`, which selected only FIRING and ACKED;
- even for a state it did see, it persisted only a RESOLVED transition.

So a rule with duration=5m that hit once (firstPendingAt=T0) and whose metric 
then
disappeared from every collection kept that anchor: the first sample that 
returned
five minutes later fired the alert with `now >= firstPendingAt + duration`, i.e.
after two isolated readings instead of five continuous minutes - and the runtime
column showed PENDING for a metric nobody was collecting any more. Prometheus's
`for:` (which the export maps this field to) restarts its timer when the series
stops matching.

Include PENDING in `findActive`, and persist the state whenever the status moved
even when the transition is not RESOLVED; a real resolution still emits the
lifecycle event, and a failed collection still keeps the episode (the
unavailable-sample skip is untouched). Residual, stated honestly: an episode 
whose
collection pass never runs at all, or that only ever sees UNAVAILABLE samples, 
is
not re-anchored by this change - that needs a last-sample timestamp the state
schema does not have.

Tests (server/): mvn -o -B -ntp test 
-Dtest='org.apache.rocketmq.studio.ops.alert.**'
-> Tests run: 328, Failures: 0, Errors: 0.
- new 
NativeAlertProcessorTest#aPendingEpisodeEndsWhenItsFingerprintIsNoLongerCollectedTest
  fails before this change ("Wanted but not invoked: save") and passes after it.
- new 
MybatisPlusAlertStateRepositoryTest#findActiveQueriesPendingEpisodesTooTest
  fails before it (the query bound two statuses) and passes after.
- mvn -o -B -ntp checkstyle:check passes.

Report URL: 
https://github.com/apache/rocketmq-dashboard/actions/runs/37990910557

With regards,
GitHub Actions via GitBox

Reply via email to