Hi prometheus expert, we have a production cluster, 5 masters, 20 workers. And we run our service in this cluster. And we install prometheus 2.8.0 with a helm chart. After one year running, we recently keep getting OOM of prometheus pod. >From the prometheus stats dashboard, we got the peak RSS 20 GB when clients access to our service. We have been keeping increasing mem again and again. Now, the limit mem of this container is 32 GB and CPU is 1.
I am not sure how huge we will increase the resource. But 32GB is really big for a pod/container. So I wonder if this is limit of prometheus and we hit it? Or is there any best practice we should comply to make our service available to our clients. Thanks in advance. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a7b64985-0352-46a0-a426-d0bd8239bb02o%40googlegroups.com.
