Thank you so much, that was super helpful. I've read a bit about staleness after reading your posts. Our elasticsearch_exporter takes more than 1 minute to run. Because of that it doesn't seem a good idea to set scrape_interval at 2 minutes. We are now thinking of increasing the scrape_interval and changing --query.lookback-delta. The latter setting changes the 5 minutes look back default value. Do you have any thoughts on that?
On Thursday, September 24, 2020 at 4:27:02 AM UTC-4 [email protected] wrote: > On Wednesday, 23 September 2020 20:25:48 UTC+1, Francois Valiquette wrote: >> >> - What's the scrape interval for this metric? >> -> scrape_interval: 30m >> > > Aha! > > Prometheus only looks back 5 minutes for data points; anything older than > that is considered "stale". > > Therefore, if you sample the time series with steps of 10 or 20 minutes, > you may keep "missing" all the data points, and get no results. If you > sample at 9 or 11 minute intervals, you will hit some (but miss others). > > Example: let's say you have data points at t=0min, t=30min, t=60min, > t=90min. > > The current time is t=118min. > > You read data with step=10m. You are therefore sampling the data at > t=118min, t=108min, t=98min etc. > > Each sample looks up to 5 minutes in the past. It picks the latest data > point from t=113-118min, 103-108min, 93-98min, 83-88min, 73-78min, > 63-68min, 53-58min etc. > > It misses the data points at t=90min, t=60min etc. You get no results, > exactly as you see. > > Solution: scrape at least every 5 minutes. Scraping at 2 minutes is > strongly recommended, so that a single missed scrape does not cause these > sorts of staleness issues. > > Don't worry about additional storage utilisation. Prometheus puts all the > data points next to each other and uses delta encoding and compression. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e8c92f57-eb16-4fd3-a45a-ae39a8f2a894n%40googlegroups.com.
