Please do not post new issues to other people's threads. On Fri, Mar 5, 2021 at 8:47 AM Nguyen Phu Quy <[email protected]> wrote:
> Sorry i have a problem after installed Promethes following: > > *Warning: *Error fetching server time: Detected 18681.458999872208 > seconds time difference between your browser and the server. Prometheus > relies on accurate time and time drift might cause unexpected query results. > > Who can explain help me? > > Thank you very much > Vào lúc 05:13:45 UTC+7 ngày Thứ Sáu, 5 tháng 3, 2021, Julien Pivotto đã > viết: > >> On 04 Mar 14:09, Dhruv Patel wrote: >> > Hi Folks, >> > We are seeing an issue in our current Prometheus Setup where we are not >> > able to ingest beyond 22 million metrics/min. We have run several Load >> Test >> > at 25 Million, 29 Million and 35 Million but the ingestion rate remains >> > constant around the same 22 million metrics/min. Moreover, we are also >> > seeing that our CPU Usage is around 70% and have more than 50% memory >> > available memory. Looking at this it feels like we are not hitting >> resource >> > limitations but something to do with lock contention. >> > >> > *Prometheus Version:* 2.9.1 >> >> Your Prometheus version is pretty old (2019). Could you run your >> benchmarks >> again with a recent release? >> >> Thanks >> >> > *Host Shape:* x7-enclave-104 (It is a bare metal host with 104 >> processor >> > units). More info can be obtained in below screenshots >> > *Memory Info: * >> > total used free shared >> > buff/cache available >> > Mem: 754G 88G 528G 67M 136G >> > 719G >> > Swap: 1.0G 0B 1.0G >> > Total: 755G 88G 529G >> > >> > We also ran some profiling during our load test setup at 20Million, 22 >> > Million and 25 Million and have seen an increase in time taken taken >> for >> > running runtime.mallocgc which leads to an increased usage in >> > runtime.futex. Some how we are not able to figure out what could be the >> > issue of the lock contention. I have attached our profiling results at >> > different load test levels if thats any useful. Any ideas on what could >> be >> > causing the high time taken in runtime malloc gc? >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/abccd4c0-c69d-4869-8598-899b3de693f7n%40googlegroups.com. >> >> >> >> >> >> >> >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/a764c0b1-d023-424a-ba66-4ffe8cbb45e3n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/a764c0b1-d023-424a-ba66-4ffe8cbb45e3n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmr0%2Bx-GEGJk2N5%3D89%2BGkaqeHx-FqHn%3D0Ad-5-%2B_b2%3D7eA%40mail.gmail.com.
