Thanks Brain, I don't want to go towards Summaries, but with histograms, mainly with Native Histograms, is there a possibility to get Max and Min values for a period of time?
With OTEL-based metrics instrumentation, it is possible to record max and min values. See https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram *Histograms consist of the following:* - An *Aggregation Temporality* of delta or cumulative. - A set of data points, each containing: - An independent set of Attribute name-value pairs. - A time window (of (start, end]) time for which the Histogram was bundled. - The time interval is inclusive of the end time. - Time values are specified as nanoseconds since the UNIX Epoch (00:00:00 UTC on 1 January 1970). - A count (count) of the total population of points in the histogram. - A sum (sum) of all the values in the histogram. - *(optional) The min (min) of all values in the histogram.* - *(optional) The max (max) of all values in the histogram.* Br, Teja On Monday, June 23, 2025 at 2:21:03 PM UTC+2 Brian Candler wrote: > Also relevant: > > https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33645 > https://groups.google.com/g/prometheus-developers/c/dGEaTR7Hyi0 > > On Monday, 23 June 2025 at 13:17:57 UTC+1 Brian Candler wrote: > >> Nice explanation of summaries here: >> >> https://grafana.com/blog/2022/03/01/how-summary-metrics-work-in-prometheus/ >> >> On Monday, 23 June 2025 at 12:42:35 UTC+1 Brian Candler wrote: >> >>> Remember that histograms don't store values. All they do is increment a >>> counter by 1; the value is only used to select which bucket to increment. >>> This means that the amount of storage used by a histogram is very small - a >>> fixed number of buckets with one counter each. It doesn't matter if you are >>> processing 1 sample per second or 10,000 samples per second. >>> >>> If you wanted to retrieve the *exact* lowest or highest value, over >>> *any* arbitrary time period that you query, you would have to store every >>> single value into a database. Prometheus is not a event logging system, and >>> it will never work this way. A columnar datastore like Clickhouse can do >>> that quite well, but if the number of samples is large, you will still have >>> a very large storage issue. >>> >>> More realistically, you could find the minimum or maximum value seen >>> over a fixed time period (say one minute), and at the end of that minute, >>> export the min/max value seen. That's cheap and quick. Indeed, you could do >>> it over a relatively short time period (e.g. 1 second), and use prometheus' >>> min/max_over_time functions if you want to query a longer period, i.e. to >>> find the min of the mins, or the max of the maxes. You need to make sure >>> that every distinct min/max value ends up in the database though; either >>> use remote_write to push them, or scrape your exporter at least twice as >>> fast as the min/max values are changing. >>> >>> In my experience, people are often not so interested in the single >>> minimum or maximum value, but in the quantiles, such as the 1st percentile >>> ("the fastest 1% of queries were answered in less than X seconds") or the >>> 99th percentile ("the slowest 1% of queries were answered in more than Y >>> seconds"). Prometheus can help you using a data type called a "summary": >>> https://prometheus.io/docs/concepts/metric_types/#summary >>> https://prometheus.io/docs/practices/histograms/#quantiles >>> >>> A summary can give you very good estimates of the percentiles over a >>> sliding time window (of a size you have to choose in advance), and uses a >>> relatively small amount of storage like a histogram. It is better than a >>> histogram in the case where you don't know in advance what the highest and >>> lowest values are likely to be (i.e. you don't need to pre-allocate your >>> bucket boundaries correctly). >>> >>> On Monday, 23 June 2025 at 08:15:42 UTC+1 tejaswini vadlamudi wrote: >>> >>>> Thanks Brain, for the clear heads-up and explanation! >>>> >>>> It looks to me that there is no possibility to secure exact maximum and >>>> exact minimum values for durations (based on Prometheus histograms) :-( >>>> >>>> However, for performing exploratory data analysis on the application >>>> software, need this summary statistics information, such as minimum and >>>> maximum values. Legacy monitoring systems have always had this support, >>>> which in turn expects the new technology to fit the use case to ensure >>>> backward compatibility. >>>> >>>> Please share what can be done in this regard to secure this info. >>>> >>>> I'm thinking out loud, please correct/add wherever possible: >>>> >>>> 1. Does changing from Prometheus to OTEL instrumentation provide this >>>> feature (exact max and min duration time)? >>>> 2. Can metrics derived from distributed traces (instrumented with >>>> OTEL/Jaeger) be used to obtain minimum and maximum request durations? >>>> 3. Is it possible to secure the max and min duration time with >>>> Prometheus with any hack? >>>> a. For Classic Histograms? >>>> b. For Native Histograms? >>>> 4. A new PR/contribution on Prometheus to offer this support? >>>> >>>> Thanks, >>>> Teja >>>> >>>> On Thursday, June 19, 2025 at 6:38:59 PM UTC+2 Brian Candler wrote: >>>> >>>>> In general, I don't think you can get an accurate answer to that >>>>> question from a histogram. >>>>> >>>>> You can work out which *bucket* the lowest and highest request >>>>> durations sat in, which means you could give the lower and upper bounds >>>>> of >>>>> the minimum, and the lower and upper bounds of the maximum. Just compare >>>>> the bucket counters at the start and end of the time range, and find the >>>>> lowest boundary (le) which has changed, and the highest boundary which >>>>> has >>>>> changed. But this still doesn't tell you what the *actual* value was. >>>>> >>>>> I don't think there's any point in trying to make an estimate of the >>>>> actual value; these values are, by definition, outliers, so even if your >>>>> data points fitted a nice distribution, these ones would be at the ends >>>>> of >>>>> the curve and subject to high error. >>>>> >>>>> Your LLM answer is essentially what it says in the documentation >>>>> <https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile> >>>>> >>>>> for histogram_quantile: >>>>> >>>>> *You can use histogram_quantile(0, v instant-vector) to get the >>>>> estimated minimum value stored in a histogram.* >>>>> >>>>> *You can use histogram_quantile(1, v instant-vector) to get the >>>>> estimated maximum value stored in a histogram.* >>>>> I thought it was worth testing. Here is a metric from my home >>>>> prometheus server, running 2.53.4: >>>>> >>>>> *go_gc_pauses_seconds_bucket* >>>>> => >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="6.399999999999999e-08"} 0 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="6.399999999999999e-07"} 0 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="7.167999999999999e-06"} 12193 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="8.191999999999999e-05"} 15369 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="0.0009175039999999999"} 27038 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="0.010485759999999998"} 27085 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="0.11744051199999998"} 27086 >>>>> go_gc_pauses_seconds_bucket{instance="localhost:9090", >>>>> job="prometheus", le="+Inf"} 27086 >>>>> >>>>> *go_gc_pauses_seconds_bucket - go_gc_pauses_seconds_bucket offset 10m* >>>>> => >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="6.399999999999999e-08"} 0 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="6.399999999999999e-07"} 0 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="7.167999999999999e-06"} 5 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="8.191999999999999e-05"} 5 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.0009175039999999999"} 10 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.010485759999999998"} 10 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.11744051199999998"} 10 >>>>> {instance="localhost:9090", job="prometheus", le="+Inf"} 10 >>>>> >>>>> *rate(go_gc_pauses_seconds_bucket[10m])* >>>>> => >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="6.399999999999999e-08"} 0 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="6.399999999999999e-07"} 0 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="7.167999999999999e-06"} 0.007407407407407408 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="8.191999999999999e-05"} 0.007407407407407408 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.0009175039999999999"} 0.014814814814814815 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.010485759999999998"} 0.014814814814814815 >>>>> {instance="localhost:9090", job="prometheus", >>>>> le="0.11744051199999998"} 0.014814814814814815 >>>>> {instance="localhost:9090", job="prometheus", le="+Inf"} >>>>> 0.014814814814814815 >>>>> >>>>> Those exponential bucket boundaries in scientific notation aren't very >>>>> readable, but you can see that: >>>>> * the lowest response time must have been somewhere >>>>> between 6.399999999999999e-07 and 7.167999999999999e-06 >>>>> * the highest response time must have been somewhere between >>>>> 8.191999999999999e-05 and 0.0009175039999999999 >>>>> >>>>> Here are the answers from the formula the LLM suggested: >>>>> >>>>> >>>>> *histogram_quantile(0, rate(go_gc_pauses_seconds_bucket[10m]))*=> >>>>> {instance="localhost:9090", job="prometheus"} *NaN* >>>>> >>>>> *histogram_quantile(1, rate(go_gc_pauses_seconds_bucket[10m]))* >>>>> => >>>>> {instance="localhost:9090", job="prometheus"} *0.0009175039999999999* >>>>> >>>>> The lower boundary of "NaN" is not useful at all (possibly this is a >>>>> bug?), but I found I could get a value by specifying a very low, but >>>>> non-zero, quantile: >>>>> >>>>> >>>>> *histogram_quantile(0.000000001, >>>>> rate(go_gc_pauses_seconds_bucket[10m]))* >>>>> => >>>>> {instance="localhost:9090", job="prometheus"} *6.40000013056e-07* >>>>> >>>>> Those values *do* sit between the boundaries given: >>>>> >>>>> >>> 6.399999999999999e-07 < 6.40000013056e-07 <= 7.167999999999999e-06 >>>>> True >>>>> >>> 8.191999999999999e-05 < 0.0009175039999999999 <= >>>>> 0.0009175039999999999 >>>>> True >>>>> >>>>> In fact, the "minimum" answer is very close to the lower edge of the >>>>> relevant bucket, and the "maximum" is the upper edge of the relevant >>>>> bucket. >>>>> >>>>> Therefore, these are not the *actual* minimum and maximum request >>>>> times. In effect, they are saying "the minimum request time was *more >>>>> than* 6.399999999999999e-07, and the maximum request time was *no >>>>> more than* 0.0009175039999999999". But that's as good as you can get >>>>> with a histogram. >>>>> >>>>> On Wednesday, 18 June 2025 at 18:17:15 UTC+1 tejaswini vadlamudi wrote: >>>>> >>>>>> Including answer from Gen-AI: >>>>>> >>>>>> | Description | PromQL Query >>>>>> >>>>>> >>>>>> | Notes >>>>>> >>>>>> | >>>>>> >>>>>> |-------------------------------------|------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------| >>>>>> | Minimum request duration (1m) | histogram_quantile(0, sum by >>>>>> (le) (rate(http_request_duration_seconds_bucket[1m]))) >>>>>> >>>>>> | Fast but may be noisy or return NaN if low traffic. Good for >>>>>> near-real-time. | >>>>>> | Maximum request duration (1m) | histogram_quantile(1, sum by >>>>>> (le) (rate(http_request_duration_seconds_bucket[1m]))) >>>>>> >>>>>> | Same as above, for longest duration estimate. >>>>>> >>>>>> | >>>>>> | Minimum request duration (5m) | histogram_quantile(0, sum by >>>>>> (le) (rate(http_request_duration_seconds_bucket[5m]))) >>>>>> >>>>>> | More stable, smoother estimate over a slightly longer window. >>>>>> >>>>>> | >>>>>> | Maximum request duration (5m) | histogram_quantile(1, sum by >>>>>> (le) (rate(http_request_duration_seconds_bucket[5m]))) >>>>>> >>>>>> | Recommended when traffic is bursty or histogram series are >>>>>> sparse. | >>>>>> >>>>>> Please confirm if the above answer is reliable or not. >>>>>> On Wednesday, June 18, 2025 at 3:23:54 PM UTC+2 tejaswini vadlamudi >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I’m using Prometheus to monitor request durations via a histogram >>>>>>> metric, e.g., http_request_duration_seconds_bucket. I would like to >>>>>>> query: >>>>>>> >>>>>>> - The minimum time taken by a request >>>>>>> - The maximum time taken by a request >>>>>>> >>>>>>> …over a given time range (say, the last 1h or 24h). >>>>>>> >>>>>>> I understand that histogram buckets give cumulative counts of >>>>>>> requests below certain durations, but I’m not sure how to extract the >>>>>>> actual min or max values of request durations during a time window. >>>>>>> >>>>>>> Is this possible directly via PromQL? Or is there a recommended >>>>>>> workaround (e.g., recording rules, external processing, or using >>>>>>> histogram_quantile() in a specific way)? >>>>>>> >>>>>>> Thanks in advance for any guidance! >>>>>>> >>>>>>> Br, >>>>>>> Teja >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/b0ea258b-2465-4c63-aa66-88fa03fa62abn%40googlegroups.com.
