During query evaluation, Prometheus tracks the current samples held in memory at evaluator.currentSamples <https://github.com/prometheus/prometheus/blob/f0003bc0ba77fca5ed4c1fe30337beea85dd95d1/promql/engine.go#L871>. This might be a good proxy for the "work" that Prometheus had to do to get the query result?
On Wednesday, 24 November 2021 at 18:25:46 UTC+5:30 [email protected] wrote: > Hello all, > > *TL;DR: *measuring `http_request_duration_seconds` on the query path is a > bad proxy for query latency as it does not account for data distribution > and number of samples/series touched by a query (both of which have > significant implications on the performance of a query) > > --- > > I'm exploring more granular performance metrics for prom queries > <https://github.com/thanos-io/thanos/issues/4895> downstream in Thanos > (inspired by this discussion from Ian Billet > <https://github.com/thanos-io/thanos/discussions/4674>) and wanted to > reach out to the Prometheus developer community for ideas on how people are > measuring and tracking query performance systematically. > > The aim is to create a new metric that captures these additional > dimensions with respect to the query to better understand/quantify query > performance SLI's with respect to number of samples/series touched > *before* a query is executed. > > The current solution I have arrived at is crude n-dimensional histogram, > where query_duration is observed/bucketed with labels representing some > scale (simplified to t-shirt sizes) of samples touched and series queried. > This would allow me to query for query_duration quantiles for some ranges > of sample/series sizes (e.g. 90% of queries for up to 1,000,000 samples and > up to 10 series complete in less than 2s) > > I would love to hear about other approaches members of the community have > taken for capturing this level of performance granularity in a metric (as > well as stir the pot wrt the thanos proposal). > > Thanks, > > Moad. -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/a392a3c9-21b0-4174-9219-53cda79de0f1n%40googlegroups.com.
