On 23/04/2020 19:59, Srinivasa praveen wrote:
*My Setup*
Hi, I am using one Prometheus instance(internal to my cluster) and
there is a cloud instance of Prometheus. Internal Prometheus scrapes
metrics from different exporters and there is a pusher process which
federates metrics from internal Prometheus and pushes to push gateway
sitting in the cloud. The cloud instance of Prometheus scrapes from
this push gateway.
Internal Prometheus scraping interval is 5seconds for almost all the
exporters except one custom exporter. This custom exporter will
execute some queries in the database and export the query results as
metrics. Scraping interval for this particular exporter endpoint is
set to 30 minutes in internal Prometheus. And my pusher scraping
interval is 1 minute, which means every 60 seconds it federates from
internal Prometheus and pushes to push gateway.
*Problem that I am facing*
Metrics from all exporters are being transmitted by pusher to push
gateway regularly. But, some times(frequently), the metrics from
custom exporter are sent as empty by pusher to push gateway.
*My Observation*
From my investigation, I can confirm that internal Prometheus is
scraping the metrics from my exporter regularly every 30 minutes(I saw
this in my exporter logs). And as part of my troubleshooting, when I
query for metrics(related to my custom exporter) from internal
Prometheus using */api/v1/query?query=<metric_name>* I see most of the
time empty result set, except some times, which is around the time
when Prometheus is scraping from my exporter, which is like every 30
minutes. So, I could see the metrics using the api, only during the
time when Prometheus is scraping. And, I am able to get the metrics
using this api for very less time hardly 20 to 30 seconds. After that
again I am seeing empty result set.
My understanding is like this, the api I am using(for troubleshooting)
will give the instant metric values (not intended to provide the
latest/recent data). The same thing is happening with the pusher
process while it is federating (the interval is every 60 seconds).
Since Prometheus scraping interval is too long (30 minutes) and the
current metric values are retained for few seconds only, by the time
pusher federating, Prometheus is giving the empty set for this
particular exporter most of the times.
Now, I want to know, is there a way to configure Prometheus to retain
the recently/latest scrapped values as current values. Or is there a
way to federate with a range like [30m] rather than the instant values.
Please help me out here and do let me know if my understanding is not
correct.
While your mechanism to send data between servers wouldn't be
recommended (either straight federation or something like Thanos would
be preferable) it sounds like the issue you are seeing is due to your
long scrape interval. Due to staleness the maximum interval is around 2
minutes, so in your case for the majority of the time Prometheus is
recording the time series as stale and therefore not returning anything
if queried.
This is by design, so the main answer would be to reduce the scraping
interval for your custom exporter.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ac43d127-2dd6-b9b0-3d04-a38ad33cdd9f%40Jahingo.com.
