That's good. Some people have setup federation before mistakenly thinking it's HA.
Looking over your heap dumps, you're right, it seems like it's all cache. This is fine, there's no reason to worry about cache memory use. The kernel will reclaim cache if it's needed for other processes. As for RSS use, Prometheus 2.16.0 binaries were built with a version of Go that has some differences with how RSS is reported. I highly recommend upgrading to the current release. There have been a lot of memory use improvements since that release. On Tue, Jul 27, 2021 at 11:56 AM Venkata Bhagavatula < [email protected]> wrote: > Hi Ben > No there is no federation between node0 and node1. This is HA setup. > > thanks n Regards, > chalapathi. > > On Sat, Jul 24, 2021 at 8:38 PM Ben Kochie <[email protected]> wrote: > >> Are you using federation to replicate the data from node0 to node1? >> >> That could be a major cause of excess memory use. >> >> On Sat, Jul 24, 2021, 08:22 Venkata Bhagavatula <[email protected]> >> wrote: >> >>> Queries are happening only on Node1. Node0 is only scrapping targets. >>> >>> On Fri, Jul 23, 2021 at 4:54 PM Stuart Clark <[email protected]> >>> wrote: >>> >>>> On 23/07/2021 12:20, Venkata Bhagavatula wrote: >>>> >>>> Forgot to mention, we are using prometheus version 2.16.0 and cannot >>>> update to the latest version. >>>> attaching the heapdump for both nodes. >>>> >>>> On Fri, Jul 23, 2021 at 4:41 PM Venkata Bhagavatula < >>>> [email protected]> wrote: >>>> >>>>> Hi All, >>>>> >>>>> In one of our production setups, we have configured prometheus HA on >>>>> Virtual machines(node0, node1). I see that node0 prometheus takes around >>>>> 5gb of ram and node1 takes just 1gb of ram. >>>>> >>>>> >>>> Are you performing queries on both nodes or just node0? >>>> >>>> -- >>>> Stuart Clark >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/CABXnQPtnoGU4b8r5VhCjetuO745Ukx%2BJw9fBurd8qxvPx_yi-Q%40mail.gmail.com >>> <https://groups.google.com/d/msgid/prometheus-users/CABXnQPtnoGU4b8r5VhCjetuO745Ukx%2BJw9fBurd8qxvPx_yi-Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr-CB1Uz8t7ues8KuUYbrC0DdCiKhJqknyu4V5zpXguXQ%40mail.gmail.com.
