FYI, container_memory_working_set_bytes is a misleading metric. It includes page cache memory, which can be unallocated any time, but improves performance of queries.
If you want to know the real memory use, I would recommend using container_memory_rss On Wed, Aug 16, 2023 at 9:31 AM Peter Nguyễn <[email protected]> wrote: > Hi Prometheus experts, > > I have a Prometheus Pod (v2.40.7) running on our Kubernetes (k8s) cluster > for metric scraping from multiple k8s targets. > > Recently, I have observed that whenever I restart a target (a k8s Pod) or > perform a Helm upgrade, the memory consumption of Prometheus keeps > increasing. After investigating, I discovered that each time the pod gets > restarted, new set of time series from that target is generated due to > dynamic values of `instance` and `pod_name`. > > The instance label value we use is in the format <pod_IP>:port, and > `pod_name` label value is the pod name. Consequently, whenever a Pod is > restarted, it receives a new allocated IP address, and a new pod name (if > not statefulset's Pod) resulting in new values for the instance & pod_name > label. > > When comes to HEAD truncation, and the number of time series in the HEAD > block goes back to the previous low value, Prometheus memory still does not > go back to the point before the target restarted. Here is the graph: > > [image: prometheus_instance_ip_port_concern.jpg] > > I am writing to seek advice on the best practices for handling these label > values, particularly for the instance. Do you have any advice on what value > format should be for those labels so we ge rid of the memory increased > every time pod gets restarted? Any time e.g. after retention triggered, the > memory would go back to the previous point? > > Regards, Vu > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/27961908-8362-42a7-b1ce-ab27dcece7b1n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/27961908-8362-42a7-b1ce-ab27dcece7b1n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpZYBidQBLD--0o%3D4OQPkT-w5hcHZ8OSizxO9xozyLEmg%40mail.gmail.com.
