Hi Steve, On 3/16/20 9:09 PM, Steve wrote: > I have been playing with the up metric to see if it is reliable. > > So far, I conclude it is not. Do you have the same results?
I guess I would have the same (technical) results, but my conclusion would be that "up" works as expected and is reliable. ;) > This is what I have done: > > > - I create the exporter. It is a pod. > > - At the scrape interval T, Prometheus server discovers it. > > - Target is visible and state is UP (UP=1) > > - Now I delete the pod. > > - At the next scrape interval T+1, Prometheus server detects the > exporter is down UP is (UP=0) > > - So far so good…but... > > - At the next scrape interval T+2, Prometheus server removes the target > and I don’t see have any info about the existence of the target (UP > metric for target disappears) This is expected. When your service discovery removes a target, Prometheus will also stop knowing and scraping it. You can build alerts for such a situation if this is unexpected in your case (e.g. "up offset 10m unless up" would send an alert for all targets which had been there 10 minutes ago, but are no longer there). However, in most cases, such a behavior is wanted. If you remove a target from a static file service discovery, this is usually by intention (e.g. because a server has been removed). If a target gets removed from the kubernetes service discovery, it might be because the container got destroyed and redeployed elsewhere. In both cases you would usually want your monitoring to stop. So, if this behavior is a problem in your case, try describing why and maybe there is a solution for that. :) Kind regards, Christian -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6ca32822-c369-9d88-8378-3588f986cba5%40hoffmann-christian.info.
