I am setting up monitoring set up for Elastic Search using Prometheus. I am running into one hurdle and need some help in making a design decision.
I am planning to run metric exporter on all master-eligible nodes in Elastic Search (3) for resiliency and also because masters have low footprint. The exporter fetches stats from local process and exports metrics for Prometheus to scrape. ES requires a quorum of at least n/2 master eligible nodes to elect a master. This implies one or more master-eligible nodes may not be part of quorum hence export out of sync metrics. If I set up all exporters as targets in Prometheus, it will not be feasible to de-duplicate metrics in Prometheus (due to split-brain issue). So the question is on setting up targets for scraping in Prometheus. Ideally, I would like to set up the current active master as the target. However, I am unable to solve service discovery issue without setting up another service. ES exposes current active master as http://cluster:9200/_cat/master but it is not in format Prometheus expects. Also, there is no metric that shows current active master. Does anyone have any experience dealing with such an issue? thanks -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a8b42e43-2650-4aa4-a503-df52b6ae7b70n%40googlegroups.com.
