> I noticed a somewhat unexpected behaviour, perhaps someone can explain why > this happens. > > - on a Prometheus instance, with a scrape interval of 10s > - doing the following queries via curl from the same node where Prometheus > runs (so there cannot be any different system times or so > > Looking at the sample times via e.g.: > $ while true; do curl -g 'http://localhost:9090/api/v1/query?query=up[1m]' > 2> /dev/null | jq '.data.result[0].values' | grep '[".]' | paste - - ; > echo; sleep 1 ; done > > the timings look suuuuper tight: > 1711148768.175, "1" > 1711148778.175, "1" > 1711148788.175, "1" > 1711148798.175, "1" > 1711148808.175, "1" > 1711148818.175, "1" > > 1711148768.175, "1" > 1711148778.175, "1" > 1711148788.175, "1" > 1711148798.175, "1" > 1711148808.175, "1" > 1711148818.175, "1" > > I.e. it's *always* .175. I guess in reality it may not actually be that > tight, and Prometheus just sets the timestamps artificially... but that > doesn't matter for me.
I think that it does potentially matter for you in this situation. My understanding is that Prometheus freezes the timestamp for all ingested samples from a particular scrape at the time where the scrape begins (including the 'up{}' sample). However, I wouldn't expect the samples to appear in the TSDB and be available for queries until after the scrape finishes, and that might take some amount of time (you can see this in the scrape_duration_seconds metric). (The timestamps are so exact because it is ten seconds from the start of the previous scrape, not its end. As long as scrapes take less than ten seconds, you'll see them start on this extremely even timeline.) So if you query for live data as of 'right now' and do so often enough and fast enough, I believe that you may miss a scrape that is currently in progress and being ingested into the TSDB. If you offset your query enough into the past that the scrape and ingestion will have finished, you will reliably get the results you expect. - cks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3658428.1711229055%40apps0.cs.toronto.edu.