Prometheus needs memory to buffer incoming data before writing it to disk. The more you scrape, the more it needs.
You can see a summary of this information on prometheus:9090/tsdb-status On Mon, Aug 23, 2021 at 1:55 PM Yaron B <[email protected]> wrote: > can anyone understand from this image why is the server is using so much ? > production-prometheus-server-869bffc459-r92nh 1186m > 54937Mi > thats crazy! > ב-יום שני, 23 באוגוסט 2021 בשעה 13:35:18 UTC+3, Yaron B כתב/ה: > >> at the moment we did add some scrape jobs that bumped the memory usage >> from around 30gb to 40gb but we are not sure why the self scraping takes so >> much ram. >> its not a new implementation, we did notice it is using a lot of memory >> but it didn't crash on us so we let it run. today >> as you can see in the attached image, it crashed, skyrocket the memory >> usage to 60gb ,then we started to disable jobs until the server didn't >> crash anymore but it is using more than it used in the last 15 days >> >> >> ב-יום שני, 23 באוגוסט 2021 בשעה 13:29:59 UTC+3, Stuart Clark כתב/ה: >> >>> On 23/08/2021 11:23, Yaron B wrote: >>> >>> I am attaching the heap.svg if someone can help me figure out what is >>> using the memory >>> ב-יום שני, 23 באוגוסט 2021 בשעה 12:23:33 UTC+3, Yaron B כתב/ה: >>> >>>> Hi, >>>> >>>> we are facing an issue with the prometheus server memory usage. >>>> when starting the server it starts with around 30GB of ram , even >>>> without any jobs configured other than the self one. >>>> in the image attached you can see the heap size usage for the >>>> prometheus job. >>>> is there a way to reduce this size? when we add our kubernetes scrape >>>> job we reach our node limit and get OOMKilled. >>>> >>> So at the moment it isn't scraping anything other than itself via the >>> /metrics endpoint? >>> >>> Is this a brand new service (i.e. no existing data stored on disk)? >>> >>> Is there anything querying the server (e.g. Grafana dashboards, etc.)? >>> >>> -- >>> Stuart Clark >>> >>> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/0659c262-daeb-452e-8dc4-4df8df22021dn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/0659c262-daeb-452e-8dc4-4df8df22021dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqqFmME%3DCcg_QB98EGUsxb1UX%2BW6OAf88nZ64%2BTTZYE1w%40mail.gmail.com.
