Hi,
I am currently creating some metrics for a legacy application. I cannot
(sadly) change the code, I can only query the database to get some
information about the state of the application. So I created a small "proxy
application" which queries the database and exposes the results as
prometheus metrics.
I have one query which returns me two values A and B which are two related
object types. Over time, objects of type A will become B, so the metric
here is a gauge.
Currently I'm exporting
object{type="A"} 123
object{type="B"} 456
My problem is that the object counts are only valid per day. So on the next
day, I'll get values from A and B, but they are not related to the values
of yesterday.
So when the value of A at 23:59 is 100, and at 00:02 it is 120, then it did
not increase by 20, in fact it got is more like it turned to 0 at 00:00 and
then increased to 120.
What I need:
I need an alert which fires if the value of A / B is under a threshold
after 3:00 a.m. I also need to know the "end" values of each day to compare
todays values with yesterdays and so on.
I tried to fullfill this by adding a "date" tag like this:
object{type="A", date="2020-10-01"} 123
object{type="B", date="2020-10-01"} 456
object{type="A", date="2020-10-02"} 10
object{type="B", date="2020-10-02"} 4
On each day change, I set the "old" metrics to 0.
So max_over_time(object{date!=""}[5d]) will give me a nice overview here.
However, the approach makes alerting hard as prometheus might fetch the
wrong (zeroed) metrics. I solved this application wise by unregistering the
old metrics.
All of this feels terribly wrong to be honest. Is there a better way to
solve this?
Thanks in advance!
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/a33af96b-e4f8-4ff8-93b5-0b904e3aedd4n%40googlegroups.com.
