Hey, Brian, sorry I wasn't clear. Using the the right quotes in the matchers solved the issue. We're using 0.23.0 in production and this is where the issue was first noticed. I'm yet to verify this in our production environment. However, the issue with the original configuration (without the right quotes) is reproducible on 0.23.0 and 0.25.0. I'm using 0.25.0 on my local environment.
On Thursday, July 20, 2023 at 2:46:51 AM UTC-4 Brian Candler wrote: > Could you say what version of alertmanager you're using? > > If it's current (0.25.0) and the issue is reproducible, then I think it's > worth raising as a ticket (if only because the behaviour is confusing). > But I'd be interested to know what it is that makes it reproducible in your > case, but not by my simplified version. > > On Thursday, 20 July 2023 at 01:59:03 UTC+1 realElonMusk wrote: > >> Thanks a lot Brian. >> This solved my issue. >> >> On Wednesday, July 19, 2023 at 4:47:52 AM UTC-4 Brian Candler wrote: >> >>> What version of alertmanager are you using? FWIW, I was unable to >>> reproduce this with alertmanager 0.25.0 >>> >>> route: >>> receiver: r0 >>> routes: >>> - receiver: r1 >>> matchers: [ 'instance =~ ".+"' ] >>> continue: true >>> - receiver: r2 >>> matchers: [ "instance = 'bar'" ] >>> continue: true >>> >>> If I raise an alert with instance="foo", it only sends to r1. >>> >>> However, the web-based routing tree editor *is* confused by the example >>> above. So unfortunately, I think this is just a symptom of the web tool >>> being different from the real thing. >>> >>> On Wednesday, 19 July 2023 at 08:36:06 UTC+1 Brian Candler wrote: >>> >>>> It looks like you're using the wrong sort of quotes in your "matchers". >>>> >>>> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' >>>> ] # this works >>>> - matchers: [ realm = "middleEarth-moria.middle-earth.com" >>>> ] # this works >>>> - matchers: [ realm = 'middleEarth-moria.middle-earth.com' >>>> ] # THIS SILENTLY MATCHES EVERYTHING! >>>> - matchers: [ "realm = 'middleEarth-moria.middle-earth.com'" >>>> ] # SO DOES THIS! >>>> >>>> For me, it works correctly, at least in the web tool, if you rewrite to >>>> the following: >>>> >>>> route: >>>> routes: >>>> - matchers: [ 'owner = middleEarth' ] >>>> receiver: 'middleEarth-alerts' >>>> routes: >>>> - matchers: [ 'alertname = MordorThreatens' ] >>>> receiver: 'middleEarth-alerts-prod-critical' >>>> routes: >>>> - matchers: [ 'realm =~ "middleEarth-.*.middle-earth.com"' >>>> ] >>>> receiver: 'middleEarth-alerts-prod-critical' >>>> continue: true >>>> - matchers: [ 'realm = middleEarth-rohan.middle-earth.com' >>>> ] >>>> receiver: 'rohan-alerts' >>>> continue: true >>>> - matchers: [ 'realm = >>>> middleEarth-rivendell.middle-earth.com' ] >>>> receiver: 'rivendell-alerts' >>>> continue: true >>>> - matchers: [ 'realm = middleEarth-shire.middle-earth.com' >>>> ] >>>> receiver: 'shire-alerts' >>>> continue: true >>>> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' >>>> ] >>>> receiver: 'moria-alerts' >>>> continue: true >>>> >>>> (Note that I've also quoted the entire matcher expression, to be sure >>>> that it doesn't get broken up by YAML parsing) >>>> >>>> I don't know what's going on under the hood, and perhaps alertmanager >>>> ought to give some sort of error if it sees a rule it doesn't understand, >>>> rather than silently passing the test. >>>> >>>> On Tuesday, 18 July 2023 at 20:28:32 UTC+1 realElonMusk wrote: >>>> >>>>> Hello Brian, >>>>> >>>>> Thank you for your response and for suggesting the use of the tool. >>>>> After using the tool with my labels, I observed that all routes under >>>>> *alertname >>>>> = MordorThreatens* are highlighted. This result is not what I >>>>> expected based on my understanding of the routing configuration. >>>>> >>>>> My expectation was that only the route with *realm = >>>>> 'middleEarth-shire.middle-earth.com >>>>> <http://middleEarth-shire.middle-earth.com>'* should be highlighted >>>>> given the label configuration I am testing with. Could you help me >>>>> understand why all routes under *alertname = MordorThreatens* are >>>>> highlighted instead? >>>>> >>>>> Here are the labels I used for testing: >>>>> >>>>> *{owner="middleEarth", alertname="MordorThreatens", >>>>> realm="middleEarth-shire.middle-earth.com >>>>> <http://middleEarth-shire.middle-earth.com>"}* >>>>> >>>>> Any further insights would be greatly appreciated. >>>>> >>>>> On Tuesday, July 18, 2023 at 2:25:59 PM UTC-4 Brian Candler wrote: >>>>> >>>>>> Quite possibly *none* of your sub-matchers are matching, and it's >>>>>> falling back to the default receiver 'middleEarth-alerts-prod-critical' >>>>>> which is at the same level as routes: >>>>>> >>>>>> route: >>>>>> routes: >>>>>> - matchers: [ owner = middleEarth ] >>>>>> receiver: 'middleEarth-alerts' >>>>>> routes: >>>>>> - matchers: [ alertname = MordorThreatens ] >>>>>> * receiver: 'middleEarth-alerts-prod-critical' << this is >>>>>> used if *none* of the routes below match* >>>>>> routes: ... >>>>>> >>>>>> But without seeing your actual alert labels and conditions I can't >>>>>> give any more help. >>>>>> >>>>>> There is an alert route testing tool you can use online, and you can >>>>>> paste in your *real* labels and matchers: >>>>>> https://prometheus.io/webtools/alerting/routing-tree-editor/ >>>>>> >>>>>> On Tuesday, 18 July 2023 at 18:42:13 UTC+1 realElonMusk wrote: >>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> I'm experiencing an issue with the *continue: true* option in my >>>>>>> Prometheus alert routing setup. Here's the configuration: >>>>>>> >>>>>>> route: >>>>>>> routes: >>>>>>> - matchers: [ owner = middleEarth ] >>>>>>> receiver: 'middleEarth-alerts' >>>>>>> routes: >>>>>>> - matchers: [ alertname = MordorThreatens ] >>>>>>> receiver: 'middleEarth-alerts-prod-critical' >>>>>>> routes: >>>>>>> - matchers: [ realm =~ 'middleEarth-.*.middle-earth.com' >>>>>>> ] >>>>>>> receiver: 'middleEarth-alerts-prod-critical' >>>>>>> continue: true >>>>>>> - matchers: [ realm = ' >>>>>>> middleEarth-rohan.middle-earth.com' ] >>>>>>> receiver: 'rohan-alerts' >>>>>>> continue: true >>>>>>> - matchers: [ realm = ' >>>>>>> middleEarth-rivendell.middle-earth.com' ] >>>>>>> receiver: 'rivendell-alerts' >>>>>>> continue: true >>>>>>> - matchers: [ realm = ' >>>>>>> middleEarth-shire.middle-earth.com' ] >>>>>>> receiver: 'shire-alerts' >>>>>>> continue: true >>>>>>> - matchers: [ realm = ' >>>>>>> middleEarth-moria.middle-earth.com' ] >>>>>>> receiver: 'moria-alerts' >>>>>>> continue: true >>>>>>> >>>>>>> In this setup, when an alert with realm=' >>>>>>> middleEarth-shire.middle-earth.com' is triggered, it successfully >>>>>>> matches against the first route and is routed to the >>>>>>> 'middleEarth-alerts-prod-critical' receiver as expected. However, the >>>>>>> routing doesn't continue to the next matchers. Specifically, the alert >>>>>>> is >>>>>>> never matched against realm='middleEarth-shire.middle-earth.com' to >>>>>>> be sent to the 'shire-alerts' receiver, even though *continue: true* >>>>>>> is set. >>>>>>> >>>>>>> I've verified the labels and they seem to be correct. Why does the >>>>>>> alert routing not continue after the first match? Any insights on how >>>>>>> to >>>>>>> resolve this issue would be greatly appreciated. >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/07aed660-a181-46e9-981e-1d172ea34e9dn%40googlegroups.com.
