Thanks for the information, my pod is already requesting 24gb and the host it's running on has 32gb available, so I'm running out of space to maneuver on these machines. I can create a dedicated node group for prometheus alone but I'm trying to avoid that if possible since it will make it more of a special creature inside our cluster, and I'm usually try to avoid that whenever possible.
On Saturday, July 31, 2021 at 12:24:44 PM UTC+3 [email protected] wrote: > There is no single correct answer. It depends on your ingestion rate and > labels. > > Just give the pod more memory so that it does not crash. You do not want > to be too tight on memory anyway, as Prometheus needs some non-process > memory for page cache in the pod in order to perform queries. > > On Fri, Jul 30, 2021 at 9:21 PM 'Yaron Idan' via Prometheus Users < > [email protected]> wrote: > >> I have a prometheus pod running on my cluster requesting 24gb of memory. >> It runs great most of the time but when it needs to read a large WAL it >> OOMs and the WAL has to be deleted manually. >> I've saw multiple messages suggesting that the instance just needs more >> memory in order to replay the WAL, but I wanted to get a better grasp of >> how the size of WAL relates to the memory the pod needs - if I can find out >> said ratio, I can probably alert well in advance if the WAL grows too big >> and needs to be handled with before we get in a crashloop. >> >> Any idea how can I find out this information? >> >> Thanks. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/d1b4df82-6be0-4b50-8731-9fe9890e41f0n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/d1b4df82-6be0-4b50-8731-9fe9890e41f0n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cf7cc795-217d-4855-b369-6e802283d6e8n%40googlegroups.com.
