Hello, I am using a Prometheus 2.21 server and I am seeing that the memory is growing at an untenable rate. I will start the pod at 4GB and eventually it will move to 10GB after a few days and go into a crashloop back off state.
The pod is configured to only keep around 8 hours of data, and this data is stored to empyDir, not a persistent file system. We are doing remote write to postgres for only about 4 metrics. We do have some no-nos (high cardinality labels & pod names) but unfortunately these are needed. I don't understand why after a retention of 8 hours we would still cause memory like this to grow and i'm looking for some guidance on how I can troubleshoot this. Please let me know, Thank you! -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e0cb10c1-a93e-4a6b-9365-cc41202ebf2fn%40googlegroups.com.
