On Wednesday, 23 September 2020 20:25:48 UTC+1, Francois Valiquette wrote: > > - What's the scrape interval for this metric? > -> scrape_interval: 30m >
Aha! Prometheus only looks back 5 minutes for data points; anything older than that is considered "stale". Therefore, if you sample the time series with steps of 10 or 20 minutes, you may keep "missing" all the data points, and get no results. If you sample at 9 or 11 minute intervals, you will hit some (but miss others). Example: let's say you have data points at t=0min, t=30min, t=60min, t=90min. The current time is t=118min. You read data with step=10m. You are therefore sampling the data at t=118min, t=108min, t=98min etc. Each sample looks up to 5 minutes in the past. It picks the latest data point from t=113-118min, 103-108min, 93-98min, 83-88min, 73-78min, 63-68min, 53-58min etc. It misses the data points at t=90min, t=60min etc. You get no results, exactly as you see. Solution: scrape at least every 5 minutes. Scraping at 2 minutes is strongly recommended, so that a single missed scrape does not cause these sorts of staleness issues. Don't worry about additional storage utilisation. Prometheus puts all the data points next to each other and uses delta encoding and compression. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d41ba6d-00ba-4ba8-83b5-8e9222a63ceeo%40googlegroups.com.
