Hi, On 8/4/20 12:24 PM, Vinod M V wrote: > > I am facing Memory usage with Prometheus service and > Maintaining 30 days of data from Node exporter, Process exporter and JMX > exporter for 95 servers in Prometheus Database. > > Grafana and Prometheus are running on the same node. When > loading node_exporter or JVM Grafana dashboards, memory usage will shoot > up to more than 10 GB and sometime usage will hit max and Prometheus > will be restarted. Moreover memory usage will not reduce even after > closing JVM/Node exporter Grafana Dashboards. > > Always need to restart Prometheus to release the memory. I am > expecting 300 more servers to this configuration in future.
95 servers don't sound that much. However, it really depends you many metrics you are scraping from them. node_exporter should usually be now problem. process_exporter's metric count may highly depend on your configuration. jvm_exporter is probably what accounts for the largest number of metrics. You can try to check if you can drop some of them (e.g. by reworking the jmx_exporter config, or, if nothing helps, via relabelling). You may also want to look into your dashboards. Queries which load a lot of metrics over longer timeframes might cause memory spikes. It can help to set up recording rules which you can then use in your dashboard. > Can someone please suggest a solution for this memory issue ( or > suggest a best database for save the metrics) and suggest a open source > solution for High availability Prometheus already contains a highly optimized TSDB. It can also be set up as high availability setup. Just run two identically configured Prometheus instances in order to ensure that at least one stays available. You can then either point Grafana to just one of them (especially if you want to avoid dashboard queries to overload both instances) or you could load balance to both of your Prometheus instances. If you are looking for alternatives, you can try Thanos, Cortex or VictoriaMetrics. Kind regards, Christian -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/abf47c43-12f7-975e-f711-4466ee2fb12b%40hoffmann-christian.info.
