Thanks for your replies. > There is nothing to handle, the instance/pod IP is required for uniqueness tracking. Different instances of the same pod need to be tracked individually. In addition, most Deployment pods are going to get new generated pod names every time anyway.
Then if we have a deployment with a large number of active time series like 01 million, every upgrade or fallback of the deployment would cause a significant memory increase because of the number time series is doubled, 02 millions in this case and Prometheus would get OOM if we don't reserve a huge memory for that scenario. > Prometheus compacts memory every 2 hours, so old data is flushed out of memory. I have re-run the test with Prometheus's latest version, v.2.46.0, capturing Prometheus memory using container_memory_rss metric. To me, it looks like the memory is not dropped after cutting HEAD to persistent block. [image: prometheus_instance_ip_port_concern_latest.jpg] Do you think it is expected? If yes, could you please share with us why the Memory is not freed up for inactive time series that are no longer in the HEAD block? On Wednesday, August 16, 2023 at 6:15:35 PM UTC+7 Ben Kochie wrote: > FYI, container_memory_working_set_bytes is a misleading metric. It > includes page cache memory, which can be unallocated any time, but improves > performance of queries. > > If you want to know the real memory use, I would recommend using > container_memory_rss > > On Wed, Aug 16, 2023 at 9:31 AM Peter Nguyễn <[email protected]> wrote: > >> Hi Prometheus experts, >> >> I have a Prometheus Pod (v2.40.7) running on our Kubernetes (k8s) cluster >> for metric scraping from multiple k8s targets. >> >> Recently, I have observed that whenever I restart a target (a k8s Pod) or >> perform a Helm upgrade, the memory consumption of Prometheus keeps >> increasing. After investigating, I discovered that each time the pod gets >> restarted, new set of time series from that target is generated due to >> dynamic values of `instance` and `pod_name`. >> >> The instance label value we use is in the format <pod_IP>:port, and >> `pod_name` label value is the pod name. Consequently, whenever a Pod is >> restarted, it receives a new allocated IP address, and a new pod name (if >> not statefulset's Pod) resulting in new values for the instance & pod_name >> label. >> >> When comes to HEAD truncation, and the number of time series in the HEAD >> block goes back to the previous low value, Prometheus memory still does not >> go back to the point before the target restarted. Here is the graph: >> >> [image: prometheus_instance_ip_port_concern.jpg] >> >> I am writing to seek advice on the best practices for handling these >> label values, particularly for the instance. Do you have any advice on what >> value format should be for those labels so we ge rid of the memory >> increased every time pod gets restarted? Any time e.g. after retention >> triggered, the memory would go back to the previous point? >> >> Regards, Vu >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/27961908-8362-42a7-b1ce-ab27dcece7b1n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/27961908-8362-42a7-b1ce-ab27dcece7b1n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/771be8ec-e37a-490b-bcf2-01de2cea591en%40googlegroups.com.
