Hi Aliaksandr, We will potentially consider Victoria Metrics in the future, Between how does increasing scrape interval can reduce RAM usage and any other resource level optimization can are also would be off great help.
Regards Dinesh On Fri, 5 Jun, 2020, 6:09 pm Aliaksandr Valialkin, <[email protected]> wrote: > > > On Fri, Jun 5, 2020 at 2:55 PM Dinesh N <[email protected]> > wrote: > >> Hi Aliaksandr, >> >> Thanks for the valuable insights.. >> >> I shall take a look at bomb-squad in the meanwhile do you foresee any >> generic options optimizations using metrics_relabel_configs or by using >> sample_limit can help to reduce the cardinalities. >> > > `sample_limit` won't help here, since it limits the number of samples that > can be scraped from a single target. It doesn't limit the number of unique > label=value pairs. > The generic solution is to identify label with the biggest number of > unique values via `/api/v1/status/tsdb` page and then remove these labels > via `metrics_relabel_configs` using `action: labeldrop`. > > > >> >> Time series - >> >> Currently we have close to 8 million time series for a single block which >> compacts in an event of every 2 hours >> >> Promethus config - >> >> RAM - 120 GB >> CPU - 32 core CPU >> Storage - 1 TB >> >> Problem statement - >> >> Once the RSS memory spikes more than 110 GB it crashes and which kind of >> makes our system very unstable ... Even we can't be increasing the >> resources more as we already operating with higest config. >> > >> >> Any directions/approaches/mechanism are highly appreciated . >> > > Try increasing scrape_interval for all the metrics. This should reduce RAM > usage for Prometheus. > > Another option is to try VictoriaMetrics - it should use lower amounts of > RAM comparing to Prometheus for this workload. > > -- > Best Regards, > > Aliaksandr Valialkin, CTO VictoriaMetrics > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAA6KEsn7CRFDSTSmr3rJkAD8fQzBgg3gWgibAK%3DqKQhzmG359w%40mail.gmail.com.
