You haven't shown any examples of the metrics, nor the queries relating to each of those graphs.
Speaking generally though, given that counters can reset, I think the best you can do is to use increase(foo[time]) which will give you an *estimate* of the amount the counter has increased over the given time window (it calculates the rate, skipping over counter resets, and scales it up to cover the whole window period. This may give a non-integer result). You should then be able to sum() over that: sum(increase(foo[time])). Note that it will give you the amount of processing done *in that specified time window*, not since you started monitoring. You said you tried sum(increase), and maybe it's one of the graphs you showed. Suppose you made a graph of sum(increase(foo[24h])); then each point on that graph represents the amount of work done *in the 24 hours up to that point*. This value will of course go up and down, since the amount of work done in any given 1 day period may go up and down. You can't possibly know the total amount of work done since the beginning of time, if the counters arbitrarily reset. You would have to have to create a persistent, non-resetting counter. increase(sum) is wrong because it can't handle the counter resets properly: see https://www.robustperception.io/rate-then-sum-never-sum-then-rate (rate and increase are essentially the same function, except increase scales its output by the width of the window) sum_over_time is very wrong: it would add together all the values within the time window. On Friday, 22 September 2023 at 14:32:19 UTC+1 Michał Idzikowski wrote: > Hello! > I'm fightning hard to get corect result. The problem is - I need to sum > data processed by service. Metrics are counters, instances are sometimes > replaced by newer ones, so they don't outlive time range most of the time. > > I've tried multiple combinations of increase, sum_over_time, > sum(increase), increase(sum) and even tried on VictoriaMetrics. Each time I > got a result were the final sum is dropping in multiple places - and as you > may image - there's is no un-processing of the data. > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/db682876-4bfa-4e6c-b2fd-acab62f2174cn%40googlegroups.com.
