Hello everyone, I am not sure that it is a bug or something with my configuration, I will provide a little bit more details about my setup and problem. We use several EKS cluster in different AWS accounts. We use helm latest helm prometheus helm-chart( https://github.com/helm/charts/tree/master/stable/prometheus) as well as latest prometheus version. We collect data with scrapping configuration from 5 EKS clusters. The problem what we faced: The disk space is growing pretty fast, 11 GB disk space used in 12 hrs. We use Prometheus as a data source for Grafana, and the big disk space cause the problem with prometheus - The node was low on resource: memory. Container prometheus-server-configmap-reload was using 2420Ki, which exceeds its request of 0. Container prometheus-server was using 30903632Ki, which exceeds its request of 20Gi. as the result - a lot of evicted states:
opengine-backend opengine-prometheus-server-6df9449899-22f9l 0/3 Evicted 0 32m opengine-backend opengine-prometheus-server-6df9449899-2fxdc 0/3 Evicted 0 64m opengine-backend opengine-prometheus-server-6df9449899-2gzlb 0/3 Evicted 0 7h55m opengine-backend opengine-prometheus-server-6df9449899-699gr 0/3 Evicted 0 131m opengine-backend opengine-prometheus-server-6df9449899-6s9q9 0/3 Evicted 0 78m opengine-backend opengine-prometheus-server-6df9449899-blx9q 0/3 Evicted 0 110m opengine-backend opengine-prometheus-server-6df9449899-d5cfw 0/3 Evicted 0 120m opengine-backend opengine-prometheus-server-6df9449899-fkjsp 0/3 Evicted 0 8h opengine-backend opengine-prometheus-server-6df9449899-h8pmp 0/3 Evicted 0 150m opengine-backend opengine-prometheus-server-6df9449899-nf8pb 0/3 Evicted 0 21m opengine-backend opengine-prometheus-server-6df9449899-qgl6b 0/3 Init:0/1 0 4s opengine-backend opengine-prometheus-server-6df9449899-sjzhd 0/3 Evicted 0 7h46m opengine-backend opengine-prometheus-server-6df9449899-txwqx 0/3 Evicted 0 21h opengine-backend opengine-prometheus-server-6df9449899-wh5ck 0/3 Evicted 0 100m opengine-backend opengine-prometheus-server-6df9449899-whpfb 0/3 Evicted 0 42m opengine-backend opengine-prometheus-server-6df9449899-ws4sq 0/3 Evicted 0 7h29m Please, help me with this. Also, please tell if you need any additional information. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/82587d4a-f231-468c-9bae-66c365a76cabo%40googlegroups.com.
