Thanks for the explanation, it actually matches with my observation that Prometheus is often running at almost the memory-limit in a cluster with high churn rate, but I was looking if this is documented or explained anywhere to avoid any surprises.
On Friday, May 22, 2020 at 5:17:07 PM UTC+2, Bjoern Rabenstein wrote: > > On 15.05.20 03:17, Shadi Abdelfatah wrote: > > Thanks but I'm more looking into information on how this goes with > memory > > limits on kubernetes pods or cgroups limits, is the mmap() function or > > prometheus aware of this limit or will it unintentionally cause the > process to > > be killed. > > In my understanding (which might be wrong, memory management is > complicated), it works like the following: > > Prometheus itself isn't aware of any K8s memory limits, but by > mmap'ing, Prometheus has delegated the management of that part of the > memory to the OS anyway. And the OS is certainly aware of the K8s > memory limits (it is, in fact, managing those). So the OS should first > evict mmap'd data from memory before it OOM-kills Prometheus. In > practice, Prometheus will therefore often look like using its memory > limit to the brink, but it should not get OOM-killed, unless all the > mmap'd memory is already evicted, and now just the normal userspace > memory requirement of Prometheus is exceeding the reservation. > > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] [email protected] <javascript:> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/daa034ac-aebc-43cb-aa48-ccc402539d0d%40googlegroups.com.
