The currently latest version v2.22.0 [email protected] schrieb am Donnerstag, 29. Oktober 2020 um 21:12:00 UTC+1:
> We see something similar for recent versions of Prometheus — but something > else might cause it. What version are you at? > > torsdag 29. oktober 2020 skrev Tim Schwenke <[email protected]>: > >> I think I will just kick the bucket and move my Promstack to a dedicated >> single node AWS ECS cluster, this way I can use EBS for persistent storage >> >> Tim Schwenke schrieb am Montag, 26. Oktober 2020 um 14:10:49 UTC+1: >> >>> Thanks for your answers, @mierzwa and Ben >>> >>> @mierzwa, I have checked the robust perception blog post you have linked >>> and it really seems like my Prometheus is using too much memory. I have >>> only around 20k series, but with quite high cardinality over a long period >>> of time. If we just look at a few hours or days the churn should be very >>> low / zero and only increase when ever an API endpoint gets called the >>> first time and so on. >>> >>> @Ben, the Prometheus is using AWS EFS for storage and so the disk size >>> should be unlimited. It sits currently at just 13 GB while resident size is >>> at around 2 GB and virtual memory around 7 GB. The node Prometheus is >>> running on has 8 GB of RAM. Now that you have mentioned storage, I have >>> read quite some time ago that Prometheus does not officially support NFS >>> and I think the AWS EFS volume is mounted via NFS into the node. >>> >>> --------------------------------------------------------- >>> >>> Here is the info I get from the Prometheus web interface: >>> >>> https://trallnag.htmlsave.net/ >>> >>> This links to a html document that I saved >>> >>> >>> >>> [email protected] schrieb am Montag, 26. Oktober 2020 um 12:57:58 UTC+1: >>> >>>> It looks like the head chunks has been growing without compacting. Is >>>> the disk full? What's in the logs? What's in the data directory? >>>> >>>> On Mon, Oct 26, 2020 at 11:45 AM Tim Schwenke <[email protected]> >>>> wrote: >>>> >>>>> Is it expected that Prometheus will take all the memory it gets over >>>>> time? I'm running Prometheus in a very "stable" environment with about >>>>> 200 >>>>> containers / targets in addition with Cadvisor and Node Exporter over 3 >>>>> to >>>>> 4 nodes. And now after a few weeks of uptime Prometheus reached the >>>>> memory >>>>> limits and I get memory limit hit events all the time. >>>>> >>>>> Here are screenshots of relevant dashboards: >>>>> >>>>> https://github.com/trallnag/random-data/issues/1#issuecomment-716463651 >>>>> >>>>> Could be something wrong with my setup or is it all ok? The >>>>> performance of queries in Grafana etc is not impacted >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/prometheus-users/f89538c2-c07b-424d-a377-614f0098a337n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/06d6ba09-4558-4cbd-9efd-f6f8bf31d90fn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > -- > David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9390befa-8ab4-4456-89fa-6a33cd812c22n%40googlegroups.com.
