What version of alertmanager are you using? FWIW, I was unable to reproduce
this with alertmanager 0.25.0
route:
receiver: r0
routes:
- receiver: r1
matchers: [ 'instance =~ ".+"' ]
continue: true
- receiver: r2
matchers: [ "instance = 'bar'" ]
continue: true
If I raise an alert with instance="foo", it only sends to r1.
However, the web-based routing tree editor *is* confused by the example
above. So unfortunately, I think this is just a symptom of the web tool
being different from the real thing.
On Wednesday, 19 July 2023 at 08:36:06 UTC+1 Brian Candler wrote:
> It looks like you're using the wrong sort of quotes in your "matchers".
>
> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' ]
> # this works
> - matchers: [ realm = "middleEarth-moria.middle-earth.com" ]
> # this works
> - matchers: [ realm = 'middleEarth-moria.middle-earth.com' ]
> # THIS SILENTLY MATCHES EVERYTHING!
> - matchers: [ "realm = 'middleEarth-moria.middle-earth.com'"
> ] # SO DOES THIS!
>
> For me, it works correctly, at least in the web tool, if you rewrite to
> the following:
>
> route:
> routes:
> - matchers: [ 'owner = middleEarth' ]
> receiver: 'middleEarth-alerts'
> routes:
> - matchers: [ 'alertname = MordorThreatens' ]
> receiver: 'middleEarth-alerts-prod-critical'
> routes:
> - matchers: [ 'realm =~ "middleEarth-.*.middle-earth.com"' ]
> receiver: 'middleEarth-alerts-prod-critical'
> continue: true
> - matchers: [ 'realm = middleEarth-rohan.middle-earth.com' ]
> receiver: 'rohan-alerts'
> continue: true
> - matchers: [ 'realm = middleEarth-rivendell.middle-earth.com'
> ]
> receiver: 'rivendell-alerts'
> continue: true
> - matchers: [ 'realm = middleEarth-shire.middle-earth.com' ]
> receiver: 'shire-alerts'
> continue: true
> - matchers: [ 'realm = middleEarth-moria.middle-earth.com' ]
> receiver: 'moria-alerts'
> continue: true
>
> (Note that I've also quoted the entire matcher expression, to be sure that
> it doesn't get broken up by YAML parsing)
>
> I don't know what's going on under the hood, and perhaps alertmanager
> ought to give some sort of error if it sees a rule it doesn't understand,
> rather than silently passing the test.
>
> On Tuesday, 18 July 2023 at 20:28:32 UTC+1 realElonMusk wrote:
>
>> Hello Brian,
>>
>> Thank you for your response and for suggesting the use of the tool. After
>> using the tool with my labels, I observed that all routes under *alertname
>> = MordorThreatens* are highlighted. This result is not what I expected
>> based on my understanding of the routing configuration.
>>
>> My expectation was that only the route with *realm =
>> 'middleEarth-shire.middle-earth.com
>> <http://middleEarth-shire.middle-earth.com>'* should be highlighted
>> given the label configuration I am testing with. Could you help me
>> understand why all routes under *alertname = MordorThreatens* are
>> highlighted instead?
>>
>> Here are the labels I used for testing:
>>
>> *{owner="middleEarth", alertname="MordorThreatens",
>> realm="middleEarth-shire.middle-earth.com
>> <http://middleEarth-shire.middle-earth.com>"}*
>>
>> Any further insights would be greatly appreciated.
>>
>> On Tuesday, July 18, 2023 at 2:25:59 PM UTC-4 Brian Candler wrote:
>>
>>> Quite possibly *none* of your sub-matchers are matching, and it's
>>> falling back to the default receiver 'middleEarth-alerts-prod-critical'
>>> which is at the same level as routes:
>>>
>>> route:
>>> routes:
>>> - matchers: [ owner = middleEarth ]
>>> receiver: 'middleEarth-alerts'
>>> routes:
>>> - matchers: [ alertname = MordorThreatens ]
>>> * receiver: 'middleEarth-alerts-prod-critical' << this is
>>> used if *none* of the routes below match*
>>> routes: ...
>>>
>>> But without seeing your actual alert labels and conditions I can't give
>>> any more help.
>>>
>>> There is an alert route testing tool you can use online, and you can
>>> paste in your *real* labels and matchers:
>>> https://prometheus.io/webtools/alerting/routing-tree-editor/
>>>
>>> On Tuesday, 18 July 2023 at 18:42:13 UTC+1 realElonMusk wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm experiencing an issue with the *continue: true* option in my
>>>> Prometheus alert routing setup. Here's the configuration:
>>>>
>>>> route:
>>>> routes:
>>>> - matchers: [ owner = middleEarth ]
>>>> receiver: 'middleEarth-alerts'
>>>> routes:
>>>> - matchers: [ alertname = MordorThreatens ]
>>>> receiver: 'middleEarth-alerts-prod-critical'
>>>> routes:
>>>> - matchers: [ realm =~ 'middleEarth-.*.middle-earth.com' ]
>>>> receiver: 'middleEarth-alerts-prod-critical'
>>>> continue: true
>>>> - matchers: [ realm = 'middleEarth-rohan.middle-earth.com'
>>>> ]
>>>> receiver: 'rohan-alerts'
>>>> continue: true
>>>> - matchers: [ realm = '
>>>> middleEarth-rivendell.middle-earth.com' ]
>>>> receiver: 'rivendell-alerts'
>>>> continue: true
>>>> - matchers: [ realm = 'middleEarth-shire.middle-earth.com'
>>>> ]
>>>> receiver: 'shire-alerts'
>>>> continue: true
>>>> - matchers: [ realm = 'middleEarth-moria.middle-earth.com'
>>>> ]
>>>> receiver: 'moria-alerts'
>>>> continue: true
>>>>
>>>> In this setup, when an alert with realm='
>>>> middleEarth-shire.middle-earth.com' is triggered, it successfully
>>>> matches against the first route and is routed to the
>>>> 'middleEarth-alerts-prod-critical' receiver as expected. However, the
>>>> routing doesn't continue to the next matchers. Specifically, the alert is
>>>> never matched against realm='middleEarth-shire.middle-earth.com' to be
>>>> sent to the 'shire-alerts' receiver, even though *continue: true* is
>>>> set.
>>>>
>>>> I've verified the labels and they seem to be correct. Why does the
>>>> alert routing not continue after the first match? Any insights on how to
>>>> resolve this issue would be greatly appreciated.
>>>>
>>>> Thank you.
>>>>
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9314030d-02b5-419e-868f-19dd5646a885n%40googlegroups.com.
