Sounds reasonable, scraping around 2 minutes and storage shouldn't be a problem. I wonder about huge ingestion expectations on Prometheus when deployed as a standalone instance. I will test this and figure out if I see an impact.
I agree that I deviated a bit from my original question. But you helped me understand the concept and reasoning. Thanks a ton Brain Candler, really appreciate your efforts! Cheers, Teja On Friday, January 21, 2022 at 9:17:33 AM UTC+1 Brian Candler wrote: > > Can I set a scraping interval of a job to 20 minutes? At the moment, one > can't adjust query delta look-back per scrape job. > > It's not recommended to scrape less frequently than once every 2 minutes. > With the default 5-minute lookback, this gives a degree of robustness > against losing a single scrape. > > In theory, you could set the lookback to say 50 minutes and then scrape > every 20 minutes. Like I say, it's not recommended, and as you've > observed, this is a global setting. > > Is there a particular reason why scraping every 2 minutes can't be done? > Don't worry about TSDB storage. Prometheus does delta-compression, so if > repeated scrapes of the same exporter give the same value, the difference > between them is zero and it uses hardly any storage at all (just the > timestamp deltas). > > If the problem is that your scrape task is expensive to run, then run it > from a cronjob and put the output somewhere where it can be scraped (e.g. > node_exporter textfile collector). This is a good idea anyway for > expensive metrics, as it avoids DoS problems if multiple clients are > scraping the same exporter. > > I can't really answer your other questions about staleness markers. My > understanding is that staleness markers are not exposed to users > <https://www.robustperception.io/staleness-and-promql> (even though > internally they're a special kind of NaN); so if you query a timeseries > which is stale, I would expect that vector result would not include that > timeseries - it would be just as if the timeseries did not exist at that > point in time. In other words, I'd expect that count(foo) would give the > number of timeseries for metric "foo" which are not stale. But that's just > my expectation; you should test it if it matters to you. It's completely > different from the question you originally raised about stopping and > restarting prometheus. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/57ec9e82-d408-4172-a017-e8a58de82058n%40googlegroups.com.

