On 24 Jun 14:41, Stuart Clark wrote: > On 24/06/2020 14:23, wang dong wrote: > > Hi prometheus expert, > > > > we have a production cluster, 5 masters, 20 workers. And we run our > > service in this cluster. > > And we install prometheus 2.8.0 with a helm chart. > > After one year running, we recently keep getting OOM of prometheus pod. > > From the prometheus stats dashboard, > > we got the peak RSS 20 GB when clients access to our service. > > We have been keeping increasing mem again and again. Now, the limit mem > > of this container is 32 GB and CPU is 1. > > > > I am not sure how huge we will increase the resource. But 32GB is really > > big for a pod/container. > > > > > > So I wonder if this is limit of prometheus and we hit it? Or is there > > any best practice we should comply > > to make our service available to our clients. Thanks in advance.
1 CPU is also too low, as we generally expect 1 CPU available for the TSDB itself. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200626080604.GA157813%40oxygen.
