*My Setup* Hi, I am using one Prometheus instance(internal to my cluster) and there is a cloud instance of Prometheus. Internal Prometheus scrapes metrics from different exporters and there is a pusher process which federates metrics from internal Prometheus and pushes to push gateway sitting in the cloud. The cloud instance of Prometheus scrapes from this push gateway. Internal Prometheus scraping interval is 5seconds for almost all the exporters except one custom exporter. This custom exporter will execute some queries in the database and export the query results as metrics. Scraping interval for this particular exporter endpoint is set to 30 minutes in internal Prometheus. And my pusher scraping interval is 1 minute, which means every 60 seconds it federates from internal Prometheus and pushes to push gateway.
*Problem that I am facing* Metrics from all exporters are being transmitted by pusher to push gateway regularly. But, some times(frequently), the metrics from custom exporter are sent as empty by pusher to push gateway. *My Observation* >From my investigation, I can confirm that internal Prometheus is scraping the metrics from my exporter regularly every 30 minutes(I saw this in my exporter logs). And as part of my troubleshooting, when I query for metrics(related to my custom exporter) from internal Prometheus using */api/v1/query?query=<metric_name>* I see most of the time empty result set, except some times, which is around the time when Prometheus is scraping from my exporter, which is like every 30 minutes. So, I could see the metrics using the api, only during the time when Prometheus is scraping. And, I am able to get the metrics using this api for very less time hardly 20 to 30 seconds. After that again I am seeing empty result set. My understanding is like this, the api I am using(for troubleshooting) will give the instant metric values (not intended to provide the latest/recent data). The same thing is happening with the pusher process while it is federating (the interval is every 60 seconds). Since Prometheus scraping interval is too long (30 minutes) and the current metric values are retained for few seconds only, by the time pusher federating, Prometheus is giving the empty set for this particular exporter most of the times. Now, I want to know, is there a way to configure Prometheus to retain the recently/latest scrapped values as current values. Or is there a way to federate with a range like [30m] rather than the instant values. Please help me out here and do let me know if my understanding is not correct. Thanks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/05a6121c-022e-442f-81a2-5330bfc0c30c%40googlegroups.com.
