On 22/03/2021 21:48, chuanjia xing wrote:
Hi there,
I recently hit an missing data point issue using prometheus.
Want to get some help here. Thanks.
*Issue:*
Increasing scrape_interval in prometheus resulted in missing data points.
*My scenario:*
I am using prometheus CloudWatch Exporter
<https://github.com/prometheus/cloudwatch_exporter> plus prometheus to
fetch aws cloudwatch metrics for ec2 instances cpuutilizaiton. The key
configs for the Exporter and Prometheus is initially as follows:
Config. Value
Scrape_interval (prometheus) 120s
Scrape_timeout (prometheus) 60s
Delay_seconds (Exporter) 600s
Range_seconds (Exporter) 600s
Period_seconds (Exporter) 60s
It is working fine with this set of configs, meaning the metrics I got
from cloudwatch has no missing data point.
Later on, I increased Prometheus scrape_interval to 320s and all other
configs are the same. I need to do this due to some other reason which
I am not explaining here. After this change, the same metrics started
to show some missing values, as shown below:
(attached graph)
You can see the missing data around time 11:30 and between 12:30 and
13:00.
There’re more of these data gaps in the metrics. And something I
noticed is that the length of the missing data gap seems to match the
scrape_interval config. For example, the first data gap above is from
11:24:26 to 11:30:08; the second data gap is from 12:44:14 to
12:50:53. Both length of gaps are around but not the same as the
scrape_interval which is 320s.
Is there something already known? This is making my graph looking bad.
The prometheus logs doesn’t provide much useful information as I can
find.
Any pointer how to investigate this issue? Thanks!
The maximum scrape interval is 5 minutes (otherwise time series will be
marked as stale), however it is recommended to have a maximum of 2-2.5
minutes to allow for a single scrape failure (which can happen due to a
timeout or slight network issue) without staleness. Is there a reason
you are trying to increase the scrape interval above 2 minutes?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0ebe8923-3acf-7003-2d69-402b7915691f%40Jahingo.com.
