Thanks for the quick response, Julien. I Will update to 2.17.2 and report back.
Cheers! On Friday, 22 May 2020 16:04:12 UTC+5, Julien Pivotto wrote: > > On 22 May 03:38, Sohaib Omar wrote: > > Hi all, I am running Prometheus *2.17.1 *as a stateful set inside the > > Kubernetes cluster. I want to debug what's causing Prometheu's memory > usage > > to increase over time even when cardinality or unique time series are > > constant over time. How can I make sure that Prometheus is not leaking > > memory? > > Below is some of my Prometheus config: > > > Prometheus 2.17.1 has a known memory leak. Please upgrade to at least > 2.17.2 :) > > > > > > storage.tsdb.max-block-duration 12m > > storage.tsdb.min-block-duration 2h > > storage.tsdb.retention.time 2h > > query.lookback-delta 5m > > query.max-concurrency 20 > > > > > > *Unique series over time:* > > > > [image: unique series over time.png] > > > > > > *Memory usage over time:* > > > > [image: image (3).png] > > In a larger cluster, which has around 80k cardinality(constant over > time), > > the memory usage increased up to *5GB *in a span of two days before > being > > OOMKilled. I am assuming Prometheus is not freeing up its cache. How can > I > > debug it better or know what's causing this? > > Thanks > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c982234b-27b8-4b2c-a6d3-4be5bb11b72e%40googlegroups.com. > > > > > > > -- > Julien Pivotto > @roidelapluie > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c75e3535-3639-4a90-b824-452133ab4ad4%40googlegroups.com.
