Hi all, I have a Prometheus Pod (v2.40.7) running on our Kubernetes (k8s) cluster for metric scraping from multiple k8s targets.
Recently, I have observed that whenever I restart a target (a k8s Pod) or perform a Helm upgrade, the memory consumption of Prometheus keeps increasing. After diving, I discovered that the cause lies in the dynamic nature of the values assigned to the instance and pod_name labels for each time series. The instance label currently uses the __address__ format, which is <pod_IP>:port. Consequently, whenever a Pod gets restarted, it receives a new allocated IP address, resulting in a new value for the instance label. Besides, we have pod_name label set to Kubernetes pod name so when a Pod does not belong to a StatefulSet controller, it will receive a new pod name upon restart. Given these circumstances, I am writing to seek advice on the best practices for handling these label values, particularly for the instance label. Thank you. Regards, Vu -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d9a8b926-70f7-4865-bf2b-b8e61336aaa1n%40googlegroups.com.
