> I noticed a somewhat unexpected behaviour, perhaps someone can explain why
> this happens.
>
> - on a Prometheus instance, with a scrape interval of 10s
> - doing the following queries via curl from the same node where Prometheus
> runs (so there cannot be any different system times or so
>
> Looking at the sample times via e.g.:
> $ while true; do curl -g 'http://localhost:9090/api/v1/query?query=up[1m]'
> 2> /dev/null | jq '.data.result[0].values' | grep '[".]' | paste - - ;
> echo; sleep 1 ; done
>
> the timings look suuuuper tight:
> 1711148768.175, "1"
> 1711148778.175, "1"
> 1711148788.175, "1"
> 1711148798.175, "1"
> 1711148808.175, "1"
> 1711148818.175, "1"
>
> 1711148768.175, "1"
> 1711148778.175, "1"
> 1711148788.175, "1"
> 1711148798.175, "1"
> 1711148808.175, "1"
> 1711148818.175, "1"
>
> I.e. it's *always* .175. I guess in reality it may not actually be that
> tight, and Prometheus just sets the timestamps artificially... but that
> doesn't matter for me.
I think that it does potentially matter for you in this situation. My
understanding is that Prometheus freezes the timestamp for all ingested
samples from a particular scrape at the time where the scrape begins
(including the 'up{}' sample). However, I wouldn't expect the samples to
appear in the TSDB and be available for queries until after the scrape
finishes, and that might take some amount of time (you can see this in
the scrape_duration_seconds metric).
(The timestamps are so exact because it is ten seconds from the start of
the previous scrape, not its end. As long as scrapes take less than ten
seconds, you'll see them start on this extremely even timeline.)
So if you query for live data as of 'right now' and do so often enough
and fast enough, I believe that you may miss a scrape that is currently
in progress and being ingested into the TSDB. If you offset your query
enough into the past that the scrape and ingestion will have finished,
you will reliably get the results you expect.
- cks
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/3658428.1711229055%40apps0.cs.toronto.edu.
