Hey, I wrote about this here https://www.bwplotka.dev/2019/golang-memory-monitoring/ at some point as well.
Unless something changes by the exporter that actually gives your this metric (it's not Prometheus question, Prometheus is merely collecting those from other exporters) container_memory_working_set_bytes used to be the best bet indeed. AND what you see is somehow expected. Even "container_memory_working_set_bytes" is not exactly 1:1 to `Total - Available` for node `free -h` as there are so many caches that kernel uses memory for, that there will be some differences. One is that free is just some utility on the container, vs working set are (if we trust cadvisor doing it well) is what cgroup is showing from outside, on the host. (see: https://github.com/google/cadvisor/issues/638) What you see it's quite a big difference though. What is an exact value for just container_memory_working_set_bytes{id="/",node="test-instance-e1"} ? Don't use `sum`, you might accidentally sum something irrelevant, etc. Since you expect one series, why you sum? As per Anoop, your findings are quite interesting indeed. I seen similar inconsistencies, but well, we don't have anything else more accurate. The memory strategy is changing every Go and kernel version (and for the good!). The memory bookkeeping of the kernel is so dynamic that those things can happen. Ideally, you always have some room before your memory limit to not worry about those inconsistencies. Kind Regards, Bartek Płotka (@bwplotka) On Sat, 12 Sep 2020 at 20:51, [email protected] <[email protected]> wrote: > Welcome to the club. > > https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/golang-nuts/LsOYrYc_Occ/LbjLAsL6BwAJ > > On Friday, September 11, 2020 at 5:26:19 AM UTC-4 Anoop wrote: > >> HI, >> >> Anyone have any suggestions on this? >> >> >> >> Thanks & Regards, >> >> Anoop Mohan >> >> Mob# +91-7293009486 <+91%2072930%2009486> >> E-mail : [email protected] >> >> >> On Thu, Sep 10, 2020 at 1:05 PM Anoop Mohan <[email protected]> wrote: >> >>> Thanks Harkishen Singh for your suggestion. >>> But, when we use *container_memory_usage_bytes *instead of >>> *container_memory_working_**set_bytes *it is showing above 55%. >>> Also referred some of the blogs which were also recommending to use >>> *container_memory_working_**set_bytes *instead of * >>> container_memory_usage_bytes * to get the accurate result. >>> >>> Ref: >>> https://medium.com/faun/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d >>> >>> Thanks & Regards, >>> >>> Anoop Mohan >>> >>> Mob# +91-7293009486 <+91%2072930%2009486> >>> E-mail : [email protected] >>> >>> >>> On Wed, Sep 9, 2020 at 10:56 PM Harkishen Singh <[email protected]> >>> wrote: >>> >>>> >>>> You don't require to sum by a node when you are already querying with >>>> exact labels since I expect the output to be one. >>>> BTW, try with *container_memory_usage_bytes *instead of >>>> *container_memory_working_set_bytes* >>>> On Wednesday, September 9, 2020 at 7:28:49 PM UTC+5:30 Anoop wrote: >>>> >>>>> Hi, >>>>> >>>>> I am using Prometheus metric "container_memory_working_set_bytes" to >>>>> display the memory usage graph in %. However, it is showing a higher value >>>>> than what I can see when I use the linux command "free -h". >>>>> >>>>> The actual usage using linux command: >>>>> [image: memory_linux.JPG] >>>>> >>>>> Based on this, it is actually taking less than 19%. >>>>> >>>>> *Now I am using below query to get the % value for memory usage:* >>>>> sum by (node) >>>>> (container_memory_working_set_bytes{id="/",node="test-instance-e1"}) / sum >>>>> by (node) (machine_memory_bytes{node=" test-instance-e1"}) * 100 >>>>> >>>>> But, this is showing 41%, which is not correct. >>>>> >>>>> Can someone please guide me if there is any mistake in the Prometheus >>>>> query or if I can usage some other metrics/query to get the current memory >>>>> usage? >>>>> >>>>> Thank You, >>>>> Anoop >>>>> >>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/bb710ee9-6cec-432d-9ec0-5703720ea5b1n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/prometheus-users/bb710ee9-6cec-432d-9ec0-5703720ea5b1n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/3546a8f3-f03c-4a82-8e2e-bd3586398279n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/3546a8f3-f03c-4a82-8e2e-bd3586398279n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMssQwZDfXAReNfYiFNd1eCii_jxpuSKb_vtTyxqyFNDSwhpow%40mail.gmail.com.
