Hi,
On 11/21/20 2:41 PM, Aleksandar Ilic wrote:
I was wondering if there is any way to set an alert only to be triggered
at a specific time of day.
As I saw for alertmanager there is PR open on GitHub but wondering if
there is any workaround for this or any other way.
What David suggested would be the simplest method, I think. It has a
drawback though: The alert will resolve once the working hours pass.
We use the following pattern instead:
* Create a pseudo-alert InhibitOutOfWorkingHours which fires except
during the relevant working hours. We want this to be localtime-aware,
which is why we don't use hour() but a textfile collector-provided
metric called localtime_hour, etc.
* We use alert relabeling to automatically add the label
inhibited_by=<alertname> to all Inhibit.* alerts (i.e.
{alertname="InhibitOutOfWorkingHours",inhibited_by="InhibitOutOfWorkingHours"})
* We set up a generic inhibition rule which takes source alerts with
alertname="Inhibit.+" and inhibits target alerts with equal inhibited_by
labels.
* The alert(s) which should be inhibited during certain timeframes are
modified to have the appropriate label (e.g.
inhibited_by="InhibitOutOfWorkingHours")
Might sound complicated, but it works fine so far and is supported by
our configuration management logic.
Kind regards,
Christian
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/663b62ec-fdde-193b-7183-4fce4a40b003%40hoffmann-christian.info.
