Thanks a lot Brian. This solved my issue. On Wednesday, July 19, 2023 at 4:47:52 AM UTC-4 Brian Candler wrote:
> What version of alertmanager are you using? FWIW, I was unable to > reproduce this with alertmanager 0.25.0 > > route: > receiver: r0 > routes: > - receiver: r1 > matchers: [ 'instance =~ ".+"' ] > continue: true > - receiver: r2 > matchers: [ "instance = 'bar'" ] > continue: true > > If I raise an alert with instance="foo", it only sends to r1. > > However, the web-based routing tree editor *is* confused by the example > above. So unfortunately, I think this is just a symptom of the web tool > being different from the real thing. > > On Wednesday, 19 July 2023 at 08:36:06 UTC+1 Brian Candler wrote: > >> It looks like you're using the wrong sort of quotes in your "matchers". >> >> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' >> ] # this works >> - matchers: [ realm = "middleEarth-moria.middle-earth.com" >> ] # this works >> - matchers: [ realm = 'middleEarth-moria.middle-earth.com' >> ] # THIS SILENTLY MATCHES EVERYTHING! >> - matchers: [ "realm = 'middleEarth-moria.middle-earth.com'" >> ] # SO DOES THIS! >> >> For me, it works correctly, at least in the web tool, if you rewrite to >> the following: >> >> route: >> routes: >> - matchers: [ 'owner = middleEarth' ] >> receiver: 'middleEarth-alerts' >> routes: >> - matchers: [ 'alertname = MordorThreatens' ] >> receiver: 'middleEarth-alerts-prod-critical' >> routes: >> - matchers: [ 'realm =~ "middleEarth-.*.middle-earth.com"' ] >> receiver: 'middleEarth-alerts-prod-critical' >> continue: true >> - matchers: [ 'realm = middleEarth-rohan.middle-earth.com' ] >> receiver: 'rohan-alerts' >> continue: true >> - matchers: [ 'realm = middleEarth-rivendell.middle-earth.com' >> ] >> receiver: 'rivendell-alerts' >> continue: true >> - matchers: [ 'realm = middleEarth-shire.middle-earth.com' ] >> receiver: 'shire-alerts' >> continue: true >> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' ] >> receiver: 'moria-alerts' >> continue: true >> >> (Note that I've also quoted the entire matcher expression, to be sure >> that it doesn't get broken up by YAML parsing) >> >> I don't know what's going on under the hood, and perhaps alertmanager >> ought to give some sort of error if it sees a rule it doesn't understand, >> rather than silently passing the test. >> >> On Tuesday, 18 July 2023 at 20:28:32 UTC+1 realElonMusk wrote: >> >>> Hello Brian, >>> >>> Thank you for your response and for suggesting the use of the tool. >>> After using the tool with my labels, I observed that all routes under >>> *alertname >>> = MordorThreatens* are highlighted. This result is not what I expected >>> based on my understanding of the routing configuration. >>> >>> My expectation was that only the route with *realm = >>> 'middleEarth-shire.middle-earth.com >>> <http://middleEarth-shire.middle-earth.com>'* should be highlighted >>> given the label configuration I am testing with. Could you help me >>> understand why all routes under *alertname = MordorThreatens* are >>> highlighted instead? >>> >>> Here are the labels I used for testing: >>> >>> *{owner="middleEarth", alertname="MordorThreatens", >>> realm="middleEarth-shire.middle-earth.com >>> <http://middleEarth-shire.middle-earth.com>"}* >>> >>> Any further insights would be greatly appreciated. >>> >>> On Tuesday, July 18, 2023 at 2:25:59 PM UTC-4 Brian Candler wrote: >>> >>>> Quite possibly *none* of your sub-matchers are matching, and it's >>>> falling back to the default receiver 'middleEarth-alerts-prod-critical' >>>> which is at the same level as routes: >>>> >>>> route: >>>> routes: >>>> - matchers: [ owner = middleEarth ] >>>> receiver: 'middleEarth-alerts' >>>> routes: >>>> - matchers: [ alertname = MordorThreatens ] >>>> * receiver: 'middleEarth-alerts-prod-critical' << this is >>>> used if *none* of the routes below match* >>>> routes: ... >>>> >>>> But without seeing your actual alert labels and conditions I can't give >>>> any more help. >>>> >>>> There is an alert route testing tool you can use online, and you can >>>> paste in your *real* labels and matchers: >>>> https://prometheus.io/webtools/alerting/routing-tree-editor/ >>>> >>>> On Tuesday, 18 July 2023 at 18:42:13 UTC+1 realElonMusk wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm experiencing an issue with the *continue: true* option in my >>>>> Prometheus alert routing setup. Here's the configuration: >>>>> >>>>> route: >>>>> routes: >>>>> - matchers: [ owner = middleEarth ] >>>>> receiver: 'middleEarth-alerts' >>>>> routes: >>>>> - matchers: [ alertname = MordorThreatens ] >>>>> receiver: 'middleEarth-alerts-prod-critical' >>>>> routes: >>>>> - matchers: [ realm =~ 'middleEarth-.*.middle-earth.com' ] >>>>> receiver: 'middleEarth-alerts-prod-critical' >>>>> continue: true >>>>> - matchers: [ realm = 'middleEarth-rohan.middle-earth.com' >>>>> ] >>>>> receiver: 'rohan-alerts' >>>>> continue: true >>>>> - matchers: [ realm = ' >>>>> middleEarth-rivendell.middle-earth.com' ] >>>>> receiver: 'rivendell-alerts' >>>>> continue: true >>>>> - matchers: [ realm = 'middleEarth-shire.middle-earth.com' >>>>> ] >>>>> receiver: 'shire-alerts' >>>>> continue: true >>>>> - matchers: [ realm = 'middleEarth-moria.middle-earth.com' >>>>> ] >>>>> receiver: 'moria-alerts' >>>>> continue: true >>>>> >>>>> In this setup, when an alert with realm=' >>>>> middleEarth-shire.middle-earth.com' is triggered, it successfully >>>>> matches against the first route and is routed to the >>>>> 'middleEarth-alerts-prod-critical' receiver as expected. However, the >>>>> routing doesn't continue to the next matchers. Specifically, the alert is >>>>> never matched against realm='middleEarth-shire.middle-earth.com' to >>>>> be sent to the 'shire-alerts' receiver, even though *continue: true* >>>>> is set. >>>>> >>>>> I've verified the labels and they seem to be correct. Why does the >>>>> alert routing not continue after the first match? Any insights on how to >>>>> resolve this issue would be greatly appreciated. >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c442bffd-618c-419f-a48c-85c2cf0dc860n%40googlegroups.com.
