Scrape interval is not an average. It should be very consistent. The timing is controlled from the start of the scrape, not the end. A typical healthy Prometheus setup will be on the exact same millisecond each time.
For example, you can see the scrape interval is on the exact same millisecond by looking at the `up` metric for a task. Prometheus assigns the timestamp of the metrics based on the start of the scrape, not the completion. https://prometheus.demo.do.prometheus.io/graph?g0.range_input=1h&g0.expr=up%7Bjob%3D%22prometheus%22%7D%5B5m%5D&g0.tab=1 What you're probably talking about is the splay between tasks. The Prometheus server attempts to spread out scrapes over the scrape interval by applying a consistent hash to the list of targets. On Sun, Feb 16, 2020 at 3:53 PM Stuart Clark <[email protected]> wrote: > Both are expected. > > The scrape interval is an average, so gaps between scrapes might be a bit > longer or shorter each time. Different scrapes with the same scrape > interval will happen at different times to prevent everything happening at > once, overwhelming the server and network. > > Ideally most metrics should be counters rather than gauges which are a lot > more resistant to slight differences in timings between scrapes and across > jobs & targets. > > On 16 February 2020 14:42:32 GMT, Arun Raj <[email protected]> wrote: >> >> hi All, >> >> We are running prometheus server in our testing environment which >> scraps 3 different exporters.( node exporter, kafka exporter and custom >> exporter). >> >> 1. All the three exporters are pulling its metrics in different timing >> even though the scrap interval are same. Is it expected behaviour ? >> 2. Our application's very time sensitive, we would like to scrap for >> every 60 seconds. But we could see the collection time difference are like >> 61 seconds or 62 seconds at sometimes. Is there any way to make it >> consistent ? Or Is it expected behaviour ? >> >> Regards, >> Arun Raj. R >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/9387C3B3-BF29-45E3-81A7-0CD5C9F5296C%40Jahingo.com > <https://groups.google.com/d/msgid/prometheus-users/9387C3B3-BF29-45E3-81A7-0CD5C9F5296C%40Jahingo.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrsLBSLndSG2EMRiOjBEp7qoMBmO8OW8ybV5s_bAhm-YQ%40mail.gmail.com.
