You need to scrape data faster than 5 minutes. 2 minutes is the recommended max to avoid stale data. Prometheus gets less efficient when you scrape slower than 1 minute.
On Sat, Jan 23, 2021 at 8:14 AM akshay sharma <[email protected]> wrote: > Hi, > > I've one some queries wrt to prometheus scraping and evaluating intervals. > > *Queries as follows.* > 1) > Say a prometheus job with *scrape interval 10 mins and evaluation > interval 5 mins*, has scraped metrics at at t1, when the same is *queried > on prometheus UI, as an instant vector, it is visible for around 3 minutes > 30 seconds* (Note:*Here I am querying repeatedly for 3 minutes 30 seconds*), > at say approximately 4th minute, when I query the same *I am seeing no > data. And even if I query the same for a range vector metric_name[5m]*, I > don't see the metrics (Note: Here, prometheus says, last s crape was > around 4 minutes ago), but the I see the metrics when I give > metric_name[10m]. Please explain the behaviour. > > No data, when executing *metric_name(instant vector)* > ElementValue > *no data* > > 2) > > This is* my Prometheus alert rule* > > throughput{instance="x.x.x.x:xxxx"} > 100 > > After *alert** fired once, when the metric data missing for more than 1 > minute, then alert will resolved.* Is there any way to avoid this? > Basically, *I do not want a resolved alert, when there is no metrics.* > > > 3) > Say, *there are 2 jobs, for one job, I need almost continuous monitoring, > so, scrape interval is 1 min, and global evaluation interval is 1 min, > there is another job which scrapes metrics at an interval of 10 mins, and > since evaluation interval is always global, I think evaluation for the > second job also is happening every 1 min, even though there are no metrics > scraped, and the evaluation results in no_data *?. This will cause a > problem in alerting as I mentioned above. *Isn't it possible to have > evaluation interval specific to job? Request you to provide some solution* > . > > > *Thanks,* > > *Akshay* > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrpHjX0qwqAEBPLjiLr7TKGeZHTouZhi9daP7A%2BnesLPg%40mail.gmail.com.
