4.6TB for 50 days seems like a lot. How many metrics and how many samples per second are you collecting? Just estimating based on the data, it sounds like you might have more than 10 million series and 600-700 samples per second. This might be the time to start thinking about sharding.
You can check for sure with these queries: prometheus_tsdb_head_series rate(prometheus_tsdb_head_samples_appended_total[1h]) For handling HA clustering and sharding, I recommend looking into Thanos. It can be added to your existing Prometheus and rolled out incrementally. > d) d) If we do use 2 separate disks for the 2 instances, how will we manage the config files? If you don't have any configuration management, I recommend using https://github.com/cloudalchemy/ansible-prometheus. It's very easy to get going. On Sat, Oct 10, 2020 at 9:06 AM [email protected] < [email protected]> wrote: > Hi. I have a vanilla Prometheus setup with 50 days retention and data size > of around 4.6TB for this much retention. I want to move to HA set up to > avoid a single point of failure. > I'm a little confused on how to approach the below points: > > a) With a HA pair, does the Prometheus data necessarily be local to both > the Prometheus instances? Because it would require me to provision 2 5TB > disks, one for each instance. > Is it a good idea to have these 2 Prometheus instances write to an NFS > disk? > > b) With both the HA pairs scraping the same targets, how do I build a > global view of these local Prometheus instances? Is federation preferable > or is there any other better way to approach this? > > c) Since, 2 instances will be scraping the same targets, does it add any > overhead to the target side? > > Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ad374ee9-c5fa-41a5-bb02-24d062366ffan%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ad374ee9-c5fa-41a5-bb02-24d062366ffan%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqhWDGMYOfAt-oisLW6XPquY%2BVLB4J0Hu9T27y2NdiKwg%40mail.gmail.com.
