On 04 Mar 14:09, Dhruv Patel wrote: > Hi Folks, > We are seeing an issue in our current Prometheus Setup where we are not > able to ingest beyond 22 million metrics/min. We have run several Load Test > at 25 Million, 29 Million and 35 Million but the ingestion rate remains > constant around the same 22 million metrics/min. Moreover, we are also > seeing that our CPU Usage is around 70% and have more than 50% memory > available memory. Looking at this it feels like we are not hitting resource > limitations but something to do with lock contention. > > *Prometheus Version:* 2.9.1
Your Prometheus version is pretty old (2019). Could you run your benchmarks again with a recent release? Thanks > *Host Shape:* x7-enclave-104 (It is a bare metal host with 104 processor > units). More info can be obtained in below screenshots > *Memory Info: * > total used free shared > buff/cache available > Mem: 754G 88G 528G 67M 136G > 719G > Swap: 1.0G 0B 1.0G > Total: 755G 88G 529G > > We also ran some profiling during our load test setup at 20Million, 22 > Million and 25 Million and have seen an increase in time taken taken for > running runtime.mallocgc which leads to an increased usage in > runtime.futex. Some how we are not able to figure out what could be the > issue of the lock contention. I have attached our profiling results at > different load test levels if thats any useful. Any ideas on what could be > causing the high time taken in runtime malloc gc? > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/abccd4c0-c69d-4869-8598-899b3de693f7n%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20210304221339.GA2515163%40oxygen.
