It looks like the head chunks has been growing without compacting. Is the disk full? What's in the logs? What's in the data directory?
On Mon, Oct 26, 2020 at 11:45 AM Tim Schwenke <[email protected]> wrote: > Is it expected that Prometheus will take all the memory it gets over time? > I'm running Prometheus in a very "stable" environment with about 200 > containers / targets in addition with Cadvisor and Node Exporter over 3 to > 4 nodes. And now after a few weeks of uptime Prometheus reached the memory > limits and I get memory limit hit events all the time. > > Here are screenshots of relevant dashboards: > > https://github.com/trallnag/random-data/issues/1#issuecomment-716463651 > > Could be something wrong with my setup or is it all ok? The performance of > queries in Grafana etc is not impacted > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqrfUJcqJH%3D%3Dn5QM4nYwrh0EpAybisAuWn2041m8ggYhA%40mail.gmail.com.
