I have read here https://www.robustperception.io/what-range-should-i-use-with-rate that I should stick to a single range for my rate() queries - especially in recording rules - and use avg_over_time() when I want to aggregate data for example in Grafana. Common ranges I also come across in the wild are 1m and 5m. But what I don't fully understand it *what factors are decisive when choosing either 1m or 5m range as the base?*
Looking at GitLab, they have recording rules covering multiple ranges. And it's the same in their panels. Sometimes data with 5m ranges, sometimes 1m. I was not able to find an applicable pattern. My environment: At any time a few hundred small targets online with a few hundred k up to single-digit million number of time series. Fairly low usage. So far from hundreds of requests per second or even minute. Meaning basically all my application specific counters are slow-moving. Considering all this: *Would you go for 1m or 5m range (to use in my recording rules)?* I definitely cannot use ad-hoc queries everywhere because especially dealing with histogram data gets very demanding after exceeding a certain number of time series in a query. Advantages of going with 1m: - more "spikey" graphs - closest to the raw data Disadvantages of going with 1m: - too spikey, especially with slow-moving time series. Often I end up with a lot of tall spikes close to each other. is this good or bad? - users might interpret the graph as "exact" which is not the case. Advantages of going with 5m: - smooths out noise at lower time ranges - looks better at least with slow moving metrics like in my environment Disadvantages of going with 5m: - losing too many details? I would be happy to know your (probably way more pragmatic) approach to this. Cheers, T.S. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cb354938-f770-48fd-9adb-0272e2c2d369o%40googlegroups.com.
