Thanks for your reply. Actually the issue which are facing is , Let the exporter expose metric at every 15minutes, and let the Prometheus to scrape data every 2mintues.
But suppose at t1, Prometheus scrapes data and gets the metrics with some value say x(metric) and y(value). I've a alert rule if metric >= y, alerts should be raised. Now alert is raised as conditions is true, now Prometheus again scrapes data at t3 but now metrics is not available ..he gets nothing . Then when again alert rules evaluates it is getting no data . But why alert is getting resolved??? Because there is no value how the condition is true? ? Metric >=y ,will never be true in case of no data right.? Is need to understand this scenario. On Sat, Jan 23, 2021, 4:02 PM Stuart Clark <[email protected]> wrote: > Removing the metric data shouldn't be done. > > The expectation is that you can query an exporter at any point and receive > metrics. Ideally that would be instant data from the time of the scrape, > but for processes that might only happen occasionally or queries which take > a while to produce you should return the latest result. > > Prometheus needs to get data regularly (generally no more infrequently as > every 2 minutes as previously mentioned). Scrapes which return the same > value as the previous one are compressed heavily, so there is no real > worries about storage space. > > If a metric stops being scraped it becomes stale, and stops being returned > as you have seen. > > As a second point regarding exporters never removing state after a state > part of the design for Prometheus is the way HA works is achieved by having > both servers of a HA pair scrape each target. If exporters removed any > state after a scrape this would break. Also it is very useful to be able to > query an exporter manually if debugging potential metrics issues. > > So in summary the recommendation would be to increase the scrape period to > no slower than 2 minutes and adjust your exporter to always return the > latest set of metrics, with that state being updated every 15 minutes as > you do currently. > > On 23 January 2021 09:29:20 GMT, akshay sharma <[email protected]> > wrote: >> >> But my exporter is exporting the data every 15min after some processing >> ,and we are deleting the data as soon as Prometheus scrapes to avoid >> registering of same metric after 15minutes,as this a limitation with >> Prometheus go client. We need to unregister to avoid registering of same >> metric. >> >> If Prometheus scrape every 1minutes, then next minute there will be no >> data till next 15minute. >> >> How can we resolve this sync issue? >> >> On Sat, Jan 23, 2021, 2:49 PM Ben Kochie <[email protected]> wrote: >> >>> You need to scrape data faster than 5 minutes. 2 minutes is the >>> recommended max to avoid stale data. Prometheus gets less efficient when >>> you scrape slower than 1 minute. >>> >>> On Sat, Jan 23, 2021 at 8:14 AM akshay sharma <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I've one some queries wrt to prometheus scraping and evaluating >>>> intervals. >>>> >>>> *Queries as follows.* >>>> 1) >>>> Say a prometheus job with *scrape interval 10 mins and evaluation >>>> interval 5 mins*, has scraped metrics at at t1, when the same is *queried >>>> on prometheus UI, as an instant vector, it is visible for around 3 minutes >>>> 30 seconds* (Note:*Here I am querying repeatedly for 3 minutes 30 >>>> seconds*), at say approximately 4th minute, when I query the same *I >>>> am seeing no data. And even if I query the same for a range vector >>>> metric_name[5m]*, I don't see the metrics (Note: Here, prometheus >>>> says, last s crape was around 4 minutes ago), but the I see the >>>> metrics when I give metric_name[10m]. Please explain the behaviour. >>>> >>>> No data, when executing *metric_name(instant vector)* >>>> ElementValue >>>> *no data* >>>> >>>> 2) >>>> >>>> This is* my Prometheus alert rule* >>>> >>>> throughput{instance="x.x.x.x:xxxx"} > 100 >>>> >>>> After *alert** fired once, when the metric data missing for more than >>>> 1 minute, then alert will resolved.* Is there any way to avoid this? >>>> Basically, *I do not want a resolved alert, when there is no metrics.* >>>> >>>> >>>> 3) >>>> Say, *there are 2 jobs, for one job, I need almost continuous >>>> monitoring, so, scrape interval is 1 min, and global evaluation interval is >>>> 1 min, there is another job which scrapes metrics at an interval of 10 >>>> mins, and since evaluation interval is always global, I think evaluation >>>> for the second job also is happening every 1 min, even though there are no >>>> metrics scraped, and the evaluation results in no_data *?. This will >>>> cause a problem in alerting as I mentioned above. *Isn't it possible >>>> to have evaluation interval specific to job? Request you to provide some >>>> solution*. >>>> >>>> >>>> *Thanks,* >>>> >>>> *Akshay* >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKcJOUib%3D8XXS-TZe3%3DSiczMQK7-8vs4x%3DQ1AnQ9dcakQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOrgXNKG2a_-M%2BZ0kwztoYA%2B%2B0fcYcpUjiZ1CUJ8nnUL5nDBGg%40mail.gmail.com.
