On 31.03.21 15:33, Al wrote: > Unfortunately, in some high request/traffic environment, many increments of > a counter over multiple pods can become significant, thus losing those > values can be a problem.
The way Prometheus calculates the rate of a counter extrapolates to cover the missed traffic. In fact, the details were optimized so that a rolling restart of all your service instances neither underreports nor overreports the traffic. High traffic environment are actually _less_ of a problem here because the statistical error gets smaller with many requests served. You would have a problem if the missed counter increments were in some way systematically different from the counter increments caught be scrapes. If that's not the case, statistics will save your day. > I figure it would be reasonable to push the metrics at a defined interval > and again prior to shutdown (given the pod is sent a clean shutdown signal > and the application has logic to intercept this) although I wanted to know > how common this is amongst the rest of the prometheus community? Is there > a more appropriate suggested course of action in this case to avoid loosing > values of an instrumented app in Kubernetes? Pushing the metrics to the Pushgateway for such a scenario is not what the Pushgateway was built for. It will simply melt under the load, not to speak about many other conceptional problems. Naively, one might now think that a push-based monitoring system is better suited to collect metrics in a controlled fashion upon a clean shutdown. However, you'll need to execute this with exceptional care to not run into a lot of other problems, especially in high traffic scenarios. (That's in part where the (not entirely true) “push doesn't scale” mantra is coming from.) So in summary, I'd say in high traffic scenarios, the most robust solution for most is the way Prometheus is already doing it. -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] [email protected] -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20210407174532.GV2627%40jahnn.
