> Prometheus' answer is to construct time series of the number of runs, > and cumulative run time, starting at some arbitrary point in time > (together these are a summary). By looking at the change in these > numbers over time, we can calculate the duty cycle (what fraction of > time is spent running vs. idle) or average run time (cumulative run > time divided by the number of runs in the same timeframe). Note that > this is all phrased in terms of numbers that exist continuously (time > spent since …) rather than individual events (time spent in the > fifteenth run). > > Unfortunately there is no trivial way to keep these accumulated > counters over multiple process invocations since the client libraries > only hold them in memory. Ideally, you could get them from the > long-running process that starts these individual runs. If that is not > possible, the third party aggregating pushgateway may be useful to > you.
Probably the easiest way to generate cumulative counters from separate one-time jobs is to use the statsd gateway, the statsd_exporter. Statsd supports incrementing persistent counters and it has a very easy wire protocol to talk to: echo 'our.counter:+3|c|#label1:aname,area:prod' | nc statsd-host 9125 (There are several ways to add labels; see the statsd_exporter readme at https://github.com/prometheus/statsd_exporter ) This would let you keep track of the total time jobs have taken to run and the count of jobs run, among other things. I think you can even do histograms through statsd_exporter if you want to (with the statsd exporter doing the hard work for your script). Pushgateway is easier to deal with in a number of ways, but it only supports setting metrics; you can't update them the way you can with the statsd exporter. - cks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20210318210840.479593200CA%40apps1.cs.toronto.edu.
