If you are doing large queries which touch a lot of timeseries you will need lots of memory and CPU.
Ideally you would minimise such queries, or use pre-aggregated metrics (created with recording rules) to simplify what is being requested. I'd suggest looking at what you are try to achieve. Are you looking over a long period? Could you aggregate? Scaling horizontally will quite likely not help if you are just asking Prometheus to process huge amounts of data. On 19 May 2020 15:29:26 BST, Dinesh Nithyanandam <[email protected]> wrote: >Can someone please help here > >On Tuesday, May 19, 2020 at 2:25:01 AM UTC+5:30, Dinesh N wrote: >> >> Hi Team, >> >> I have been using Thanos-Prometheus stack and running into high >> cardinality issues where CPU goes till ~80% and then goes down and >this >> happens when firing high cardinality queries which results in "Http >> superfluous" exception and then promethus instance goes down. >> >> We are trying following things as listed below - >> >> 1) We are running only with 2 instances of Prometheus on top of >Thanos >> querier and need guidance where can we increase more to handle huge >queries >> >> 2) Any front end cache like cortex cache can help here for high >> cardinality queries ? >> >> 3) Looking for any optimal linux parameters like hugepages which >would >> suffice high cardinality issues >> >> >> RCA So far, I have observed was CPU was clocking till ~80% and >prometheus >> server was doing down and I also see lot of memory residing at cache >memory >> . >> Even with above options we are not sure whether we are looking things >at >> right direction hence need Your pair of eyes and pointers would be >greatly >> appreciated here Brain. >> >> >> Thanks and Regards >> Dinesh >> > >-- >You received this message because you are subscribed to the Google >Groups "Prometheus Users" group. >To unsubscribe from this group and stop receiving emails from it, send >an email to [email protected]. >To view this discussion on the web visit >https://groups.google.com/d/msgid/prometheus-users/7aeb7ca5-3f81-4670-b80f-6b9030d913df%40googlegroups.com. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CDEBB67B-2F8F-4B00-B8AA-69350108003A%40Jahingo.com.
