It looks like a good idea.But I'm using alertManager to receive all the 
alerts, including calling the alertManager API directly to create the 
alerts, and I can't forbid the client not to call for a certain period of 
time, so I want to set it on alertManager to be silent  

在2020年11月24日星期二 UTC+8 上午5:24:35<Christian Hoffmann> 写道:

> Hi,
>
> On 11/21/20 2:41 PM, Aleksandar Ilic wrote:
> > I was wondering if there is any way to set an alert only to be triggered 
> > at a specific time of day.
> > 
> > As I saw for alertmanager there is PR open on GitHub but wondering if 
> > there is any workaround for this or any other way.
>
> What David suggested would be the simplest method, I think. It has a 
> drawback though: The alert will resolve once the working hours pass.
>
> We use the following pattern instead:
>
> * Create a pseudo-alert InhibitOutOfWorkingHours which fires except 
> during the relevant working hours. We want this to be localtime-aware, 
> which is why we don't use hour() but a textfile collector-provided 
> metric called localtime_hour, etc.
>
> * We use alert relabeling to automatically add the label 
> inhibited_by=<alertname> to all Inhibit.* alerts (i.e. 
>
> {alertname="InhibitOutOfWorkingHours",inhibited_by="InhibitOutOfWorkingHours"})
>
> * We set up a generic inhibition rule which takes source alerts with 
> alertname="Inhibit.+" and inhibits target alerts with equal inhibited_by 
> labels.
>
> * The alert(s) which should be inhibited during certain timeframes are 
> modified to have the appropriate label (e.g. 
> inhibited_by="InhibitOutOfWorkingHours")
>
>
> Might sound complicated, but it works fine so far and is supported by 
> our configuration management logic.
>
> Kind regards,
> Christian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/91967dd5-dea9-4596-9170-7d20428eb1a8n%40googlegroups.com.

Reply via email to