Ah, I see this issue was fixed in 2.17.2. I am using 2.17.1. https://github.com/prometheus/prometheus/issues/7120 On Thursday, July 9, 2020 at 4:24:27 PM UTC-7 YY Wan wrote:
> Looks like there was an issue with attaching photos in my post. I've > attached them here: > > On Thursday, July 9, 2020 at 4:17:33 PM UTC-7 YY Wan wrote: > >> Hello, >> >> I have a Prometheus instance A deployed that scrapes a few k8s targets >> and has a retention set at 3h. It federates those metrics to another >> Prometheus instance B which has persistent storage for longer retention. >> >> I am seeing a long term increase in memory usage for Prometheus instance >> A. It gradually increases in memory usage over days until it reaches the >> memory limit we configured for it and OOMs. Additionally, when it is near >> the limit of its memory request, the federated Prometheus B will start to >> have "context deadline exceeded" when scraping Prom Instance A's federated >> endpoint. >> >> >> >> I have one Prometheus instance A deployed in several regions, and this >> only happens to the Prometheus instances in some regions. Other regions >> look healthy, with the memory usage dropping as expected every couple of >> hours as our retention is 3h. >> >> >> >> Looking at the heap profile of the healthy Prom instance A and unhealthy >> (memory increasing long term trend) Prom instance A, I see an obvious >> difference. >> >> Healthy: >> >> >> Unhealthy: >> >> >> I see that in the unhealthy instance, most of the memory is being used >> for tsdb.add. I'm a little confused since we have the retention period set >> to be 3h for the Prom instance A in both of these regions, yet the >> unhealthy one is using up a lot of memory for some tsdb related functions. >> >> My question is: What is causing this Prometheus instance to use so much >> memory on this tsdb function, and how can I prevent this? >> >> Thanks! >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/30b75bc5-831c-41ab-a301-4653ce2f358cn%40googlegroups.com.
