The problem was that the scrape interval was 10 secs but all the
observations were happening during a 1 sec-burst,
and before that burst the histogram for the
{waited_for_ipam="false",waited_for_local_ca="true"} labels did not exist.
This means that the first time the metric got scraped it already had all of
its observations, hence rate() completely
missed the increment.
One hack is to define the metric with a call to .With(labels) ASAP where
the labels are {waited_for_ipam="false",waited_for_local_ca="true"}
and ensure that there's at least one scrape before the burst that adds all
the observations (this can be achieved, for instance,
with a sleep at least as long as the scrape interval).
Matteo.
On Wed, Apr 22, 2020 at 5:03 PM Bjoern Rabenstein <[email protected]>
wrote:
> On 19.04.20 11:01, Matteo Olivi wrote:
> > I have instrumented a Go application with Prometheus' client_golang v
> 0.9.2,
>
> Probably unrelated to your problem, but that version is more than 16
> months old. You might consider upgrading to the most recent version.
>
> > histogram_quantile(0.99, rate
> > (kos_agent_attachment_create_to_local_ifc_latency_seconds_bucket
> > {waited_for_ipam="false", waited_for_local_ca="true"}[1m]))
>
> My first intuition is that `[1m]` is way too short. You need a scrape
> interval of 15s or less to make this a meaningful duration for the
> `rate`. And even then, the observations you want to see have to happen
> in this 1m timeframe.
>
> What happens if to use 10m or 1h?
>
> _However_, if really no observations happen in the 1m range, then the
> result should be NaN rather than the lowest bucket. The latter would
> happen if the only bucket that gets an increase is the lowest one.
>
> To debug that, you could check the result without the
> `histogram_quantile`, i.e. run the following:
>
>
> rate(kos_agent_attachment_create_to_local_ifc_latency_seconds_bucket{waited_for_ipam="false",waited_for_local_ca="true"}[1m])
>
> Then check which buckets have a value at all, which have a value of
> zero, and which have a higher one.
>
> > Prometheus binary is v2.6.1 and runs within a Kubernetes Pod (like the
> > instrumented application).
>
> Again probably unrelated, but it's highly recommended to upgrade to a
> more recent version.
>
> --
> Björn Rabenstein
> [PGP-ID] 0x851C3DA17D748D03
> [email] [email protected]
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAEESUgmftgEpeVV65%3DM%3Do0o9SgLYGJtTTjF46yAD3YE_dUnhkQ%40mail.gmail.com.
