Hi Christian Thanks for the suggestion. I stand corrected. Regards Steve
On Mon, Mar 16, 2020 at 4:21 PM Christian Hoffmann < [email protected]> wrote: > Hi Steve, > > On 3/16/20 9:09 PM, Steve wrote: > > I have been playing with the up metric to see if it is reliable. > > > > So far, I conclude it is not. Do you have the same results? > > I guess I would have the same (technical) results, but my conclusion > would be that "up" works as expected and is reliable. ;) > > > This is what I have done: > > > > > > - I create the exporter. It is a pod. > > > > - At the scrape interval T, Prometheus server discovers it. > > > > - Target is visible and state is UP (UP=1) > > > > - Now I delete the pod. > > > > - At the next scrape interval T+1, Prometheus server detects the > > exporter is down UP is (UP=0) > > > > - So far so good…but... > > > > - At the next scrape interval T+2, Prometheus server removes the target > > and I don’t see have any info about the existence of the target (UP > > metric for target disappears) > > This is expected. When your service discovery removes a target, > Prometheus will also stop knowing and scraping it. > > You can build alerts for such a situation if this is unexpected in your > case (e.g. "up offset 10m unless up" would send an alert for all targets > which had been there 10 minutes ago, but are no longer there). > > However, in most cases, such a behavior is wanted. If you remove a > target from a static file service discovery, this is usually by > intention (e.g. because a server has been removed). > If a target gets removed from the kubernetes service discovery, it might > be because the container got destroyed and redeployed elsewhere. In both > cases you would usually want your monitoring to stop. > > So, if this behavior is a problem in your case, try describing why and > maybe there is a solution for that. :) > > Kind regards, > Christian > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAL7AJRsgotSHPZ5%2Be%3D_j7mJN6pcn_L0hL0w6M5piC7gWZNASVw%40mail.gmail.com.
