On 22 May 03:38, Sohaib Omar wrote: > Hi all, I am running Prometheus *2.17.1 *as a stateful set inside the > Kubernetes cluster. I want to debug what's causing Prometheu's memory usage > to increase over time even when cardinality or unique time series are > constant over time. How can I make sure that Prometheus is not leaking > memory? > Below is some of my Prometheus config:
Prometheus 2.17.1 has a known memory leak. Please upgrade to at least 2.17.2 :) > > storage.tsdb.max-block-duration 12m > storage.tsdb.min-block-duration 2h > storage.tsdb.retention.time 2h > query.lookback-delta 5m > query.max-concurrency 20 > > > *Unique series over time:* > > [image: unique series over time.png] > > > *Memory usage over time:* > > [image: image (3).png] > In a larger cluster, which has around 80k cardinality(constant over time), > the memory usage increased up to *5GB *in a span of two days before being > OOMKilled. I am assuming Prometheus is not freeing up its cache. How can I > debug it better or know what's causing this? > Thanks > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/c982234b-27b8-4b2c-a6d3-4be5bb11b72e%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200522110405.GA448351%40oxygen.
