Sounds reasonable, scraping around 2 minutes and storage shouldn't be a 
problem.
I wonder about huge ingestion expectations on Prometheus when deployed as a 
standalone instance. I will test this and figure out if I see an impact.

I agree that I deviated a bit from my original question. But you helped me 
understand the concept and reasoning.

Thanks a ton Brain Candler, really appreciate your efforts!

Cheers,
Teja

On Friday, January 21, 2022 at 9:17:33 AM UTC+1 Brian Candler wrote:

> > Can I set a scraping interval of a job to 20 minutes? At the moment, one 
> can't adjust query delta look-back per scrape job.
>
> It's not recommended to scrape less frequently than once every 2 minutes.  
> With the default 5-minute lookback, this gives a degree of robustness 
> against losing a single scrape.
>
> In theory, you could set the lookback to say 50 minutes and then scrape 
> every 20 minutes.  Like I say, it's not recommended, and as you've 
> observed, this is a global setting.
>
> Is there a particular reason why scraping every 2 minutes can't be done?  
> Don't worry about TSDB storage.  Prometheus does delta-compression, so if 
> repeated scrapes of the same exporter give the same value, the difference 
> between them is zero and it uses hardly any storage at all (just the 
> timestamp deltas).
>
> If the problem is that your scrape task is expensive to run, then run it 
> from a cronjob and put the output somewhere where it can be scraped (e.g. 
> node_exporter textfile collector).  This is a good idea anyway for 
> expensive metrics, as it avoids DoS problems if multiple clients are 
> scraping the same exporter.
>
> I can't really answer your other questions about staleness markers.  My 
> understanding is that staleness markers are not exposed to users 
> <https://www.robustperception.io/staleness-and-promql> (even though 
> internally they're a special kind of NaN); so if you query a timeseries 
> which is stale, I would expect that vector result would not include that 
> timeseries - it would be just as if the timeseries did not exist at that 
> point in time.  In other words, I'd expect that count(foo) would give the 
> number of timeseries for metric "foo" which are not stale.  But that's just 
> my expectation; you should test it if it matters to you.  It's completely 
> different from the question you originally raised about stopping and 
> restarting prometheus.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/57ec9e82-d408-4172-a017-e8a58de82058n%40googlegroups.com.

Reply via email to