I meant, "Rates coming from two different pods" -> "Counters coming from two different pods"
On Sat, Jan 16, 2021 at 4:48 PM Julius Volz <[email protected]> wrote: > Hi Manish, > > Rates coming from two different pods will be separate time series, as you > said. If you apply the rate() function to them, then the rate of increase > will be calculated separately for each of the series. You can then choose > to sum up those multiple rates to get a total rate of increase over all > pods (e.g. "sum(rate(my_counter[5m]))"). > > When pods are coming and going, there are some sources of uncertainty > introduced: Between Prometheus' last scrape of a pod and that pod being > terminated, the pod may get some additional counter increments. These extra > increments will be "lost" to Prometheus forever and there's no way to > recover them, because they died with your pod. Similarly, when a new pod > first gets scraped, it may already have seen some counter increments before > that first scrape. But these effects are usually small unless the rate of > pod terminations + creations comes close to the magnitude of your scrape > interval. Usually you should scrape often enough to get most counter > increments. > > Further, when running the rate() function over a series, Prometheus will > apply a heuristic to detect whether a series starts or ends within the > provided rate window (see > https://github.com/prometheus/prometheus/blob/e483cff61ff1b68e8925db1059393c4d36af9df5/promql/functions.go#L107-L110 > for more details). If rate() thinks that a series starts *before* the > window / continues *after* the window, rate() extrapolates the calculated > slope toward the respective window boundary, becauses it assumes that > the counter behavior will be roughly similar during that extrapolated time. > Otherwise, if rate() thinks that a series starts or ends under the window, > it does not do this extrapolation, and thus returns a lower value. This > behavior tries to do the right thing on average, but will never be fully > precise. > > Regards, > Julius > > > On Fri, Jan 15, 2021 at 7:02 PM Manish G <[email protected]> > wrote: > >> Hi All, >> >> rate function handles restarts if metrics is of counter type, as >> documentation tells. >> >> But how is it of help in something like kubernetes environment. For eg, >> we have application running on several pods and metrics scraped by >> prometheus has pod id as one of the label. So if a pod goes away and >> replaced by a new one, then we have 2 time series here, both treated >> separately. >> >> So I am not sure how does rate function is of help here? >> I hope I am able to convey my point. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAOFd6ig%2BjkVGOkXse4JuP%3DTFhszOHGOyMw-LQgYgNF8dhiEn8w%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAOFd6ig%2BjkVGOkXse4JuP%3DTFhszOHGOyMw-LQgYgNF8dhiEn8w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5zYEvBnxvGj8PQVMre_hyv1KHMq1A6kefBJasMqEx7zRQ%40mail.gmail.com.
