I realize that if you have an instrumented application which is auto-scaled in Kubernetes and a pod is terminated before the next scrape interval, you can loose the values which were exposed by that instance. I also understand that in this situation, if you really want to avoid the possibility of loosing these metrics, the application could be modified to instead push metrics to a Push Gateway. I realize in general we should rely on prometheus metrics for trending and not depend on exact values. Unfortunately, in some high request/traffic environment, many increments of a counter over multiple pods can become significant, thus losing those values can be a problem.
I figure it would be reasonable to push the metrics at a defined interval and again prior to shutdown (given the pod is sent a clean shutdown signal and the application has logic to intercept this) although I wanted to know how common this is amongst the rest of the prometheus community? Is there a more appropriate suggested course of action in this case to avoid loosing values of an instrumented app in Kubernetes? Thanks for your help -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/77cd9e93-cf58-4e53-bf24-b103e781085en%40googlegroups.com.
