Do you mean requests with result status code 500?
This is a bit tricky. First thing you have to be careful of is that
"probe_http_duration_seconds" is not the total, it's broken down into
phases, as you can see if you try the exporter with curl:
$
*curl
'localhost:9115/probe?module=http_2xx_example&target=https:%2f%2fwww.google.com'*
...
probe_duration_seconds 0.471663605
...
probe_http_duration_seconds{phase="connect"} 0.010641254
probe_http_duration_seconds{phase="processing"} 0.046997224
probe_http_duration_seconds{phase="resolve"} 0.001434721
probe_http_duration_seconds{phase="tls"} 0.421022725
probe_http_duration_seconds{phase="transfer"} 0.001299392
...
probe_http_status_code 200
So really you should be using probe_duration_seconds which is the total
time.
Now, you can generate a filtered query like this:
probe_duration_seconds and (probe_http_status_code == 500)
The logical operators are described here
<https://prometheus.io/docs/prometheus/latest/querying/operators/#logical-set-binary-operators>
and
depend on the LHS and RHS having the same set of labels, unless you start
doing grouping. This should return only LHS data points where the RHS has
a data point.
The trouble is, to do avg_over_time on that you'll need a subquery:
avg_over_time( (probe_duration_seconds and (probe_http_status_code ==
500))[5m:1m] ) > 3
Subqueries will resample your data - in the above example it will take 1
minute steps over 5 minutes. So you need to align this with whatever
scraping rate you are using. It might be good enough.
In general, a 500 error means your server is failing. If you're getting a
noticeable number of 500 errors during a 5 minute period, you probably have
bigger problems to worry about than the response time! That is you should
fix the error, not worry about how long the error takes to be returned.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/6838ca02-d986-4f34-8b0b-df13da87a709%40googlegroups.com.
