Hi all, I am running Prometheus *2.17.1 *as a stateful set inside the Kubernetes cluster. I want to debug what's causing Prometheu's memory usage to increase over time even when cardinality or unique time series are constant over time. How can I make sure that Prometheus is not leaking memory? Below is some of my Prometheus config:
storage.tsdb.max-block-duration 12m storage.tsdb.min-block-duration 2h storage.tsdb.retention.time 2h query.lookback-delta 5m query.max-concurrency 20 *Unique series over time:* [image: unique series over time.png] *Memory usage over time:* [image: image (3).png] In a larger cluster, which has around 80k cardinality(constant over time), the memory usage increased up to *5GB *in a span of two days before being OOMKilled. I am assuming Prometheus is not freeing up its cache. How can I debug it better or know what's causing this? Thanks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c982234b-27b8-4b2c-a6d3-4be5bb11b72e%40googlegroups.com.
