Hi Aliaksandr, Thanks for the valuable insights..
I shall take a look at bomb-squad in the meanwhile do you foresee any generic options optimizations using metrics_relabel_configs or by using sample_limit can help to reduce the cardinalities. Time series - Currently we have close to 8 million time series for a single block which compacts in an event of every 2 hours Promethus config - RAM - 120 GB CPU - 32 core CPU Storage - 1 TB Problem statement - Once the RSS memory spikes more than 110 GB it crashes and which kind of makes our system very unstable ... Even we can't be increasing the resources more as we already operating with higest config. Any directions/approaches/mechanism are highly appreciated . Thanks in anticipation Dinesh On Fri, 5 Jun, 2020, 5:06 pm Aliaksandr Valialkin, <[email protected]> wrote: > High cardinality labels may be monitored at /api/v1/status/tsdb page. See > https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats for > details. Note that this page became available starting from Prometheus > v2.14 . > > Take a look also at https://github.com/open-fresh/bomb-squad project, > which detects high cardinality labels and automatically adds relabeling > rules in order to reduce the cardinality. > > On Thu, Jun 4, 2020 at 11:16 PM Dinesh N <[email protected]> > wrote: > >> Thanks Murali for the quick response >> >> But how do we analyse it and can we below options >> >> 1) metrics_relabel_configs >> 2) recording rules >> >> >> On Fri, 5 Jun, 2020, 12:38 am Murali Krishna Kanagala, < >> [email protected]> wrote: >> >>> This should be taken care by the exporter you are collecting metrics >>> from. If you are writing your own exporter then make validate what labels >>> stay unique all the time. For ex. If you are collecting ngnix request >>> metrics then passing request uuid as a label makes the metrics highly >>> cardinal. >>> >>> On Thu, Jun 4, 2020, 2:01 PM Dinesh N <[email protected]> >>> wrote: >>> >>>> Hi Team, >>>> >>>> Figuring out ways to optimise high cardinality labels, any suggestions >>>> are welcomed here. >>>> >>>> Regards >>>> Dinesh >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/CAA6KEskjY1TpjVddTSAAx8AL3aH7Zc7yyNcg0McpOBOfrrow0g%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/prometheus-users/CAA6KEskjY1TpjVddTSAAx8AL3aH7Zc7yyNcg0McpOBOfrrow0g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAA6KEs%3D4Zf29aOdMTXG_0JExSkXKaow3JG6aZSh-X2xb1PSHgA%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAA6KEs%3D4Zf29aOdMTXG_0JExSkXKaow3JG6aZSh-X2xb1PSHgA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Best Regards, > > Aliaksandr Valialkin, CTO VictoriaMetrics > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAA6KEsmHJHp9Uv8Uiw4Vz7QJkh5fcOs_22eSBUMunusCWZwZnw%40mail.gmail.com.
