To put it another way. If you can read every event raw from a log line, like every request has a "took X milliseconds", there are better ways to reconstruct metrics for your use case.
On Sun, Aug 7, 2022 at 9:46 PM Ben Kochie <[email protected]> wrote: > Right, but more basic, how do you get this information from the > application right now? Are you reading logs? Does it emit statsd data? > > You're saying what, but not how. > > On Sun, Aug 7, 2022 at 7:15 PM Johny <[email protected]> wrote: > >> Gauge contains most recent values of a metric, sampled every 1 min or so, >> and exported by a user application, e.g. some latency sampled at 1 minute >> intervals by a client application. Lets presume this time series (scraped >> by Prometheus or sent via remote write) is absolute containing all the >> information we need for calculating derived statistics. In the most raw >> form, you can fetch the data points, sort them and calculate percentile. >> Incidentally, legacy backend has efficient mechanisms to calculate >> percentiles by scanning and reducing data using map-reduce. >> >> >> >> On Sunday, August 7, 2022 at 7:49:05 AM UTC-4 [email protected] wrote: >> >>> So, let's take a step back and find out some more information, because >>> this question is sounding a lot like an XY Problem. >>> >>> How are the current applications generating their metrics right now? >>> How are you getting the data to create these histograms? >>> >>> On Sun, Aug 7, 2022 at 9:23 AM Johny <[email protected]> wrote: >>> >>>> We are migrating telemetry backend from legacy database to Prometheus >>>> and require estimating percentiles on gauge metrics published by user >>>> applications. Estimating percentiles on a gauge metric in Prometheus is not >>>> feasible and for a number of reasons, client applications will be difficult >>>> to modify to start publishing histograms. >>>> >>>> I am exploring feasibility of creating a histogram in a recording rule >>>> in Prometheus based on the metrics published by users. The partial work put >>>> in so far seems inefficient, also illegible. Is there a recommended >>>> approach to solve this problem? As stated earlier, it will be extremely >>>> hard to solve the problem on the client side and I am looking for a >>>> solution within Prometheus. >>>> >>>> *Current metric is a gauge with with values representing request >>>> latency.* >>>> http_duration_milliseconds_gauge{instance="instance1:port1"}[1h] >>>> 1659752188 100 >>>> 1659752068 120 >>>> .. >>>> 1659751708 150 >>>> 1659751588 160 >>>> >>>> *Desired histogram after conversion -* >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=100} 133 >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=120} 222 >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=140} 311 >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=160} 330 >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=180} 339 >>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1", >>>> le=200} 340 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/f95b5512-1c81-4e12-9670-7c7eb0d29f5en%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/f95b5512-1c81-4e12-9670-7c7eb0d29f5en%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/c8e3c184-e88d-4217-badc-f5f779b52af3n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/c8e3c184-e88d-4217-badc-f5f779b52af3n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqRSt0M1cuEo%2Bbutq%2BedqocGgyFi%3Dw6nxELUfAWq1L4JQ%40mail.gmail.com.
