While searching for some patterns I noticed this thread. Looked on this counter warm-up solution - quite clever, thanks for sharing!
We are working on different solution, utilizing created timestamps feature. It's being designed here <https://docs.google.com/document/d/1kakDVn8aP1JerimLeazuMfy2jaB14W2kZDHMRokPcv4/edit> and client implementation is in progress here <https://github.com/prometheus/client_golang/pull/1313>. Feedback welcome! On Thursday, October 6, 2022 at 6:04:45 PM UTC+1 Bruno Martinez wrote: > Hi Julius, > > Indeed this is about catching the initial increase of counters. > If, for example, we count the http request of a server, and we got 30 > requests just after start-up, before the first scrape by Prometheus. > Exporting the counter first to 0 (before exporting 30) ensures these 30 > requests are included in the result of increase() over a period that > includes the start-up time. > > Concerning your second point, the library supports defining a 'warm-up' > period during which each new counter (new combination of labels) will be > exported to 0 (whoever makes the GET request on /metrics). We use this for > the case of multiple Prometheus. In our case we have between 2 and 4 > instances and we use thanos for query. > > So you don't need to know how many Prometheus, but make sure the warm-up > phase is big enough to cover the scrape period (in our case all Prometheus > have the same scrape period of 30s). > > I agree this adds some dependency between the exporter and Prometheus. > > Regards > Bruno > > Le jeu. 6 oct. 2022 à 12:17, Julius Volz <[email protected]> a écrit : > >> (Ah, I guess this is about "catching" any single initial counter >> increases for slow-moving counters as well, so that rate outputs are not 0 >> on those) >> >> On Thu, Oct 6, 2022, 18:15 Julius Volz <[email protected]> wrote: >> >>> Hi Bruno, >>> >>> Could you outline how returning 0 (instead of 1) for counters for their >>> first scrape helps with the existence issues of a time series? The sample >>> value of a series should have no effect on the series' existence. >>> >>> And as >>> https://github.com/goto-opensource/smart-prometheus-client#counters-warm-up >>> notes, it's important to point out here again that this is not really >>> compatible with Prometheus' collection model where producers of metrics >>> should not need to know how many Prometheus servers are scraping them, and >>> GET requests to /metrics should be idempotent for that reason. >>> >>> Regards, >>> Julius >>> >>> On Thu, Oct 6, 2022 at 5:12 PM Bruno Martinez <[email protected]> >>> wrote: >>> >>>> Hello Prometheus Community! >>>> >>>> There have been many discussions and issues related to the problem of >>>> new timeseries and the increase() (example: here >>>> <https://github.com/prometheus/prometheus/issues/1673>, or here >>>> <https://github.com/prometheus/client_ruby/issues/141>) and even the >>>> official documentation of Prometheus mentions to "avoid missing metrics >>>> <https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics>". >>>> >>>> >>>> >>>> The workaround consist in manually initializing our metrics to 0. This >>>> can become complex with vectors when we don't know in advance the >>>> combination of label values, moreover we need to be sure the first scrape >>>> from Prometheus include this 0 value. >>>> >>>> At Goto.com we implemented a solution >>>> <https://github.com/goto-opensource/smart-prometheus-client> to this >>>> problem that has been running in production for more than 1 year and we >>>> decided to open it to the community. >>>> >>>> It's golang library extending the official client that ensure the first >>>> collection(s) of counters always return 0 instead of their actual value. >>>> This works for counters/histogram/summaries and the corresponding vectors. >>>> >>>> This is done in the background without anything to add in your >>>> application code. >>>> >>>> >>>> We hope this will help the community, try it! >>>> >>>> feedback are welcome. >>>> >>>> >>>> https://github.com/goto-opensource/smart-prometheus-client >>>> >>>> Have a nice day! >>>> Bruno >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/63982f2c-32f6-4d91-b694-cd08226203e1n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/63982f2c-32f6-4d91-b694-cd08226203e1n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Julius Volz >>> PromLabs - promlabs.com >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2dc97c99-fcf5-4b19-a374-d6495b863132n%40googlegroups.com.
