[prometheus-users] Re: Prometheus using a large amount of memory when managing storage.

I was just tweaking the TSDB block durations to match prod. I was reading that we might want to reduce the TSDB block durations to help free up memory. We are seeing OOM in the system logs. I am watching memory using the following command. watch "ps ax -o pcpu,rss,ppid,pid,stime,args |

Re: [prometheus-users] Re: Prometheus using a large amount of memory when managing storage.

when prometheus is writing metrics? I am thinking there must be an easy way to rule out race conditions, On Saturday, October 16, 2021 at 5:47:32 AM UTC-4 sup...@gmail.com wrote: > On Fri, Oct 15, 2021 at 9:51 PM Chad Sesvold wrote: > >> At this point I am the only one running quer

[prometheus-users] Re: Prometheus using a large amount of memory when managing storage.

At this point I am the only one running queries. When I have no target defined the memory seems to be flat. When I changed the follow in non-pro it seemed to stabilize the memory usage. --storage.tsdb.max-block-duration 15d --storage.tsdb.min-block-duration 1h I will try copying the binaries

[prometheus-users] Prometheus using a large amount of memory when managing storage.

We have been running prometheus for 3 or 4 years now. In production we have 6 month retention and in non-production we have a retention of 45 days. in production we are capturing 1.8 million metrics with 2,300 targets. In non-prod we are capturing 800 K metrics with 2,200 targets. The