Thanks Brian. I am using loki for storing the logs. But as every
microservice is logging in a different format, its difficult for me to get
the trace id. So I am printing the istio-proxy logs in json format and
deriving the metrics using the Loki Recording rules. I am able to calculate
rate of REST requests. But not able to calculate the duration of the
requests.
- record: app:requests:rate5m
expr: 'sum(rate({container="istio-proxy"} | json |
path!~".*actuator.*|.*health.*",user_agent!~"kube-probe.*|.*HealthChecker.*|.*Prometheus.*",__error__=""[5m]))
by (app,path,response_code)'
labels:
cluster: dev
region: ap-south-1
On Tuesday, 5 October, 2021 at 2:20:16 pm UTC+5:30 Brian Candler wrote:
> With prometheus you can't have the exact duration of an individual request
> as a label, like "9.234" for example, because every single request would
> have a different duration, and hence a different timeseries, and you'd have
> an explosion of timeseries.
>
> The way to get an accurate average (mean) is to expose two metrics: the
> *number of requests* and the *total time servicing requests*. That is, for
> each request you add 1 to the request counter, and the duration of that
> request to the total time counter. Then it's easy to calculate the average
> request duration over the time window of interest.
>
> Another approach is to group requests into a fixed set of buckets, and the
> way to do that with prometheus is as histograms. Once you have histograms,
> then there are functions which act over that.
>
> https://prometheus.io/docs/concepts/metric_types/#histogram
> https://prometheus.io/docs/practices/histograms/
>
> From this you can estimate quantiles (e.g. "50% of requests were serviced
> in less than 3.4 seconds"). From one set of buckets you can query
> arbitrary quantiles, but the errors will depend on the sizes of the buckets.
>
> As an alternative to histograms there are also "summaries" which are for
> when you have fixed quantiles that you want to calculate (e.g. the 95th
> percentile) more accurately than can be derived from histogram buckets.
> These tend to be for more specialised applications.
>
> Finally, if you still do want to log the exact duration of each and every
> request, then you should be using a logging system (like loki or
> elasticsearch), not prometheus.
>
> On Tuesday, 5 October 2021 at 09:13:27 UTC+1 mvkri...@gmail.com wrote:
>
>> For one of the scenarios, I am getting the duration of http call as a
>> label. So, how can I aggregate it to get the average duration. For ex:
>> http_call{path="/example",method="GET",duration=10,app="my-app"} 1
>>
>> The value is always 1, but the labels change. How can I get the avg
>> duration of the calls that were processed in a minute?
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/b31c8e74-e530-4d0d-b345-756859478132n%40googlegroups.com.
