Hi Prometheus experts, I have a Prometheus Pod (v2.40.7) running on our Kubernetes (k8s) cluster for metric scraping from multiple k8s targets.
Recently, I have observed that whenever I restart a target (a k8s Pod) or perform a Helm upgrade, the memory consumption of Prometheus keeps increasing. After investigating, I discovered that each time the pod gets restarted, new set of time series from that target is generated due to dynamic values of `instance` and `pod_name`. The instance label value we use is in the format <pod_IP>:port, and `pod_name` label value is the pod name. Consequently, whenever a Pod is restarted, it receives a new allocated IP address, and a new pod name (if not statefulset's Pod) resulting in new values for the instance & pod_name label. When comes to HEAD truncation, and the number of time series in the HEAD block goes back to the previous low value, Prometheus memory still does not go back to the point before the target restarted. Here is the graph: [image: prometheus_instance_ip_port_concern.jpg] I am writing to seek advice on the best practices for handling these label values, particularly for the instance. Do you have any advice on what value format should be for those labels so we ge rid of the memory increased every time pod gets restarted? Any time e.g. after retention triggered, the memory would go back to the previous point? Regards, Vu -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/27961908-8362-42a7-b1ce-ab27dcece7b1n%40googlegroups.com.
