Thanks for the explanation, I thought staleness is applicable only to Prometheus Targets, haven't imagined this concept to Prometheus restarts and unavailability. So, you say 'statelessness' is also applied to Prometheus availability.
Is there a way to know if a metric or series is marked stale through the general PromQL curl command and also through the expression browser? After reading a few Prometheus docs and Q&A, is it still a good practice to keep scraping intervals around 2 minutes and not change the default query look-back delta flag? On Wednesday, January 19, 2022 at 9:57:40 AM UTC+1 Brian Candler wrote: > I believe the following is true as well: > > - if prometheus does a scrape and a metric is missing (in the exporter > output) which was present in a previous scrape, it's immediately marked as > "stale", i.e. a staleness marker is inserted into the timeseries. > > - however in your case, you're turning off the prometheus server, so > there's no scraping taking place. You just get a point in the timeseries > at the time of the last scrape before prometheus shut down, and then at the > first scrape after prometheus starts up. There is no indication within the > timeseries data itself that anything is missing; therefore, PromQL queries > will look back up to 5 minutes. > > On Wednesday, 19 January 2022 at 07:59:57 UTC Brian Candler wrote: > >> Yes. Google "Prometheus Staleness". >> >> In short: the value of a metric at time T is the most recent value >> recorded *on or before time T*, and prometheus will look backwards in >> time up to --query.lookback-delta (default 5 minutes) to find this value. >> >> On Tuesday, 18 January 2022 at 23:54:07 UTC [email protected] wrote: >> >>> Hi, >>> >>> I just run Prometheus (2.32.0-beta.0) without changing any CLI flags. If >>> I shut it down for <5 Mins, I don't see a gap in my graphs. If I shut it >>> down for > 5Mins, I see a gap. Is this valid? >>> >>> Thanks, >>> Teja >>> [image: ppk.PNG] >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/dc085c50-6913-4984-b258-2a0a04c6aa33n%40googlegroups.com.
