Thanks for your answers, @mierzwa and Ben @mierzwa, I have checked the robust perception blog post you have linked and it really seems like my Prometheus is using too much memory. I have only around 20k series, but with quite high cardinality over a long period of time. If we just look at a few hours or days the churn should be very low / zero and only increase when ever an API endpoint gets called the first time and so on.
@Ben, the Prometheus is using AWS EFS for storage and so the disk size should be unlimited. It sits currently at just 13 GB while resident size is at around 2 GB and virtual memory around 7 GB. The node Prometheus is running on has 8 GB of RAM. Now that you have mentioned storage, I have read quite some time ago that Prometheus does not officially support NFS and I think the AWS EFS volume is mounted via NFS into the node. --------------------------------------------------------- Here is the info I get from the Prometheus web interface: https://trallnag.htmlsave.net/ This links to a html document that I saved [email protected] schrieb am Montag, 26. Oktober 2020 um 12:57:58 UTC+1: > It looks like the head chunks has been growing without compacting. Is the > disk full? What's in the logs? What's in the data directory? > > On Mon, Oct 26, 2020 at 11:45 AM Tim Schwenke <[email protected]> > wrote: > >> Is it expected that Prometheus will take all the memory it gets over >> time? I'm running Prometheus in a very "stable" environment with about 200 >> containers / targets in addition with Cadvisor and Node Exporter over 3 to >> 4 nodes. And now after a few weeks of uptime Prometheus reached the memory >> limits and I get memory limit hit events all the time. >> >> Here are screenshots of relevant dashboards: >> >> https://github.com/trallnag/random-data/issues/1#issuecomment-716463651 >> >> Could be something wrong with my setup or is it all ok? The performance >> of queries in Grafana etc is not impacted >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d44d701c-7cc0-44c2-bf7a-79e1383d9e00n%40googlegroups.com.
