Re: [prometheus-developers] How. to deal with outdated series in alerts.

[Matthias Rampke]

Restart the application that produces metrics.

Generally, the client libraries will remember metrics even if they are not
being incremented anymore (it cannot know that they won't be again).
Restarting clears the "seen" label set in each process.

Side note: this can also bite you the other way – if an endpoint is
unexpectedly never being requested after a restart (say, it wasn't hooked
up correctly in the code), this alert will not detect it, because generic
instrumentation of HTTP calls cannot know which valid paths there *should*
be.

In some cases, we alert like this because we want to check if a specific
business action has occurred. Instead of relying on the automatic HTTP
metrics, we can separately instrument the code with another metric that we
control, and where we can "increment by zero" for all possible label
combinations on startup.

/MR

On Sun, Nov 15, 2020, 17:00 Mounya A  wrote:

> Hello all,
>I have a question - how do we deal with  labels that are no longer
> there (intentionally stopped) in alerts. Will there be any threshold time
> to consider it as stale or unwanted.
>I have configured an alert when rate(requests[1m]) == 0. It is
> firing alerts for labels, that didn't show up in the past 7 days. I have
> intentionally stopped and don't want to alert in such conditions. How to
> deal with this.
>
>   Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gZyYhJ0pUiFsqyPqKZGeKxf3%2Bf1r1Os3KuYBkwigbViag%40mail.gmail.com.

Re: [prometheus-developers] Alertmanager HA

[Matthias Rampke]

The listen address is the one that alertmanager binds to. Generally, this
can be 0.0.0.0 (all interfaces), I believe that is the default and thus
sometimes omitted.

It will try to guess which address it can be reached at by other AMs (which
address it should advertise to them). In some circumstances this doesn't
work right. A Kubernetes pod probably isn't one of those circumstances, but
specifying the address explicitly doesn't hurt either.

So in short, I would expect everything to work out of the box in a standard
Kubernetes pod, but you can be more explicit if you want to be.

/MR


On Sat, Nov 14, 2020, 20:41 Dudi Cohen  wrote:

> Hi all, I would like to configure Alertmanager for high availability in
> k8s.
> Can anyone please explain the difference
> between `cluster.advertise-address` and `cluster.listen-address`? I've seen
> either one used in different examples, sometimes with `0.0.0.0` and
> sometimes with `$(POD_IP)` so i'm also not sure what the address should be.
> The documentation is a bit unclear regarding the differences:
> "The cluster.advertise-address flag is required if the instance doesn't
> have an IP address that is part of RFC 6890
>  with a default route."
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-developers/4b78b5fd-4e04-4276-a8d0-6388d8398413n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gbycNqMd%2BKAVabWrATUpWbYrYv_THH6otdgCtsz%2Bed8dA%40mail.gmail.com.

[prometheus-developers] How. to deal with outdated series in alerts.

[Mounya A]

Hello all, 
   I have a question - how do we deal with  labels that are no longer 
there (intentionally stopped) in alerts. Will there be any threshold time 
to consider it as stale or unwanted. 
   I have configured an alert when rate(requests[1m]) == 0. It is 
firing alerts for labels, that didn't show up in the past 7 days. I have 
intentionally stopped and don't want to alert in such conditions. How to 
deal with this. 

  Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-developers/d7c7ed77-0f69-4a5a-b36a-40c95c669546n%40googlegroups.com.