Hi All, In one of our production setups, we have configured prometheus HA on Virtual machines(node0, node1). I see that node0 prometheus takes around 5gb of ram and node1 takes just 1gb of ram.
user has changed min.block-duration to 30m and max block duration to 2h. I told them to not modify them as these are only for developers. Will this cause a RAM usage difference? But then both nodes should use more RAM. I checked that both prometheus are using the same configuration file. I see that the number of allocs in node0 are more than that of node1 under the "/debug/pprof" page. I collected the heap dump from "curl http://localhost:9090/debug/pprof/heap ". tried to check the top heap allocations using "go tool pprof heap_node0" Following is the output: File: prometheus Type: inuse_space Time: Jul 22, 2021 at 4:36am (EDT) Entering interactive mode (type "help" for commands, "o" for options) (pprof) top Showing nodes accounting for 81.25MB, 68.48% of *118.65MB total* Dropped 68 nodes (cum <= 0.59MB) Showing top 10 nodes out of 101 flat flat% sum% cum cum% 16MB 13.49% 13.49% 16MB 13.49% github.com/prometheus/prometheus/tsdb/chunkenc.NewXORChunk 12.34MB 10.40% 23.89% 13.94MB 11.75% compress/flate.NewWriter 8MB 6.75% 30.63% 8MB 6.75% github.com/prometheus/prometheus/tsdb/chunkenc.(*bstream).writeByte 7.50MB 6.32% 36.96% 7.50MB 6.32% github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels 7.50MB 6.32% 43.28% 7.50MB 6.32% github.com/prometheus/prometheus/tsdb.newMemSeries 7.40MB 6.24% 49.51% 7.40MB 6.24% github.com/prometheus/prometheus/scrape.(*scrapeCache).trackStaleness 6.50MB 5.48% 55.00% 6.50MB 5.48% github.com/prometheus/prometheus/tsdb/chunkenc.(*bstream).writeBit 6MB 5.06% 60.05% 6MB 5.06% github.com/prometheus/prometheus/pkg/textparse.(*PromParser).Metric 5.50MB 4.64% 64.69% 9MB 7.59% github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet 4.50MB 3.79% 68.48% 22.50MB 18.97% github.com/prometheus/prometheus/tsdb.(*memSeries).cut top output says that only ~120MB is in inuse bytes. What about the rest of the memory? I read in one of blog( https://source.coveo.com/2021/03/03/prometheus-memory/) that it is cached memory allocated by mmap. Is it OK to have that much memory in cache? Currently I am trying to plot a graph for go_memstats_heap_.*_bytes. Can you let me know how to debug further with respect to RAM usage difference between two nodes? In production setup there is a management application which monitors the total ram usage of the node and if it reaches a threshold then some action would be performed on that node. Thanks n Regards, chalapathi -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABXnQPuOxtYzgUCjpGg%2BpYQiFFgC0JbESAqL_MmCqM9QhxE2Zg%40mail.gmail.com.
