Hello,
I have instrumented a Go application with Prometheus' client_golang v
0.9.2, and among the metrics there are the following histograms:
attachmentCreateToLocalIfcHistograms := prometheus.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: metricsNamespace,
Subsystem: metricsSubsystem,
Name: "attachment_create_to_local_ifc_latency_seconds",
Help: "Seconds from attachment creation to finished creating local
interface",
Buckets: []float64{-0.125, 0, 0.125, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64, 128,
256},
ConstLabels: map[string]string{"node": node},
},
[]string{naWaitedForIPAMLabel, naWaitedForLocalCALabel})
I am displaying the 99th percentile (over 1 minute) of the histogram in
Grafana with the following query:
histogram_quantile(0.99,
> rate(kos_agent_attachment_create_to_local_ifc_latency_seconds_bucket{waited_for_ipam="false",
>
> waited_for_local_ca="true"}[1m]))
where "kos" is the metrics namespace and "agent" is the metrics subsystem.
Notice that the expression picks only one of the histograms in the vector
(the one with labels waited_for_ipam="false", waited_for_local_ca="true"),
that's because
I'm running some experiments where I generate load for the application and
currently those labels are the only ones that get used.
I repeated the same experiment multiple times and Grafana always displays
the value -0.125, which is the upper bound of the lowest bucket. Via log
statements, however,
I observed that there is always a total of 30 observations (because of how
the experiment is built) and that all the observations have a value that is
higher than 10. The same value
is returned if I execute the aforementioned query directly in Prometheus
expression browser.
Also, not sure whether it is relevant but all the 30 observations always
take place almost at the same time (within the same second).
Prometheus binary is v2.6.1 and runs within a Kubernetes Pod (like the
instrumented application).
How is it possible that histogram_quantile always returns -0.125?
Thanks,
Matteo.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/a33b9ee7-93bf-4543-b1d0-b72aecfed9af%40googlegroups.com.
