To be more precise about maximum scrape times: an instant vector selector
in PromQL (e.g. foo{bar="baz"}) looks back a maximum of 5 minutes from the
current evaluation timestamp to find the latest sample for each matched
series before that timestamp. If the last sample is more than 5 minutes
old, nothing is returned at that evaluation step. This lookback duration is
configurable via the --query.lookback-delta command-line flag, though most
people will want to keep it at 5m (to not return super old data points as
"current"). So in practice, if you take into account that scrapes can also
fail once or twice due to various errors, it's a good idea to keep the
scrape interval significantly lower than the lookback delta (like 2m).And yeah, probably the best thing to do in your situation is to make the exporter cache the DB query results and scrape it more frequently. Or if really all your Prometheus does is scrape this exporter slowly, you could consider increasing the lookback delta. But in any case for such a slow computation in the backend it probably makes sense to decouple it from Prometheus scrapes, so that your scrapes don't take >15 minutes. On Fri, Apr 24, 2020 at 7:57 AM Srinivasa praveen <[email protected]> wrote: > Thanks for the response Stuark. The reason behind keeping the scraping > interval so long is, on receiving scrape request from Prometheus, my > exporter performs around 10 queries against database and exposes the result > as 10 metrics, which will take around 15 minutes to complete all the > queries. And Prometheus scrape was timing out. So, to increase the > scrape_timeout I had to increase the scrape_interval also. > > So, from your response, I understood that, if the scraping interval is > more than 2 minutes, mostly Prometheus considers this as stale data. I > think that is the reason, why I am seeing metrics for very few seconds, > when I query using the api. And when I use range vector in my api (like > <my_metric>[20m]) I see some data. Is my understanding correct here ? > > So, do you suggest to change the exporter to periodically perform db > queries in the back end and store the results in memory/file handy, so that > it can respond back immediately when ever Prometheus scrapes (Instead of > performing db queries on receiving the scrape request). Then scraping won't > timeout and I can configure less than 2 minutes of scraping interval. > > > > On Friday, April 24, 2020 at 2:02:45 AM UTC+5:30, Stuart Clark wrote: >> >> On 23/04/2020 19:59, Srinivasa praveen wrote: >> >> *My Setup* >> Hi, I am using one Prometheus instance(internal to my cluster) and there >> is a cloud instance of Prometheus. Internal Prometheus scrapes metrics from >> different exporters and there is a pusher process which federates metrics >> from internal Prometheus and pushes to push gateway sitting in the cloud. >> The cloud instance of Prometheus scrapes from this push gateway. >> Internal Prometheus scraping interval is 5seconds for almost all the >> exporters except one custom exporter. This custom exporter will execute >> some queries in the database and export the query results as metrics. >> Scraping interval for this particular exporter endpoint is set to 30 >> minutes in internal Prometheus. And my pusher scraping interval is 1 >> minute, which means every 60 seconds it federates from internal Prometheus >> and pushes to push gateway. >> >> *Problem that I am facing* >> Metrics from all exporters are being transmitted by pusher to push >> gateway regularly. But, some times(frequently), the metrics from custom >> exporter are sent as empty by pusher to push gateway. >> >> *My Observation* >> From my investigation, I can confirm that internal Prometheus is scraping >> the metrics from my exporter regularly every 30 minutes(I saw this in my >> exporter logs). And as part of my troubleshooting, when I query for >> metrics(related to my custom exporter) from internal Prometheus using >> */api/v1/query?query=<metric_name>* I see most of the time empty result >> set, except some times, which is around the time when Prometheus is >> scraping from my exporter, which is like every 30 minutes. So, I could see >> the metrics using the api, only during the time when Prometheus is >> scraping. And, I am able to get the metrics using this api for very less >> time hardly 20 to 30 seconds. After that again I am seeing empty result >> set. >> My understanding is like this, the api I am using(for troubleshooting) >> will give the instant metric values (not intended to provide the >> latest/recent data). The same thing is happening with the pusher process >> while it is federating (the interval is every 60 seconds). Since Prometheus >> scraping interval is too long (30 minutes) and the current metric values >> are retained for few seconds only, by the time pusher federating, >> Prometheus is giving the empty set for this particular exporter most of the >> times. >> >> Now, I want to know, is there a way to configure Prometheus to retain the >> recently/latest scrapped values as current values. Or is there a way to >> federate with a range like [30m] rather than the instant values. >> Please help me out here and do let me know if my understanding is not >> correct. >> >> While your mechanism to send data between servers wouldn't be recommended >> (either straight federation or something like Thanos would be preferable) >> it sounds like the issue you are seeing is due to your long scrape >> interval. Due to staleness the maximum interval is around 2 minutes, so in >> your case for the majority of the time Prometheus is recording the time >> series as stale and therefore not returning anything if queried. >> >> This is by design, so the main answer would be to reduce the scraping >> interval for your custom exporter. >> >> -- >> Stuart Clark >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/8eebdfd4-ab02-4632-bb9a-a60f824fe4fb%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/8eebdfd4-ab02-4632-bb9a-a60f824fe4fb%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6YoyuA1qa%3DE2X7OCLZoEm3EFLdOmNdmoy9pxb0RMK0Yv2wg%40mail.gmail.com.
