Thanks a lot, Stuart. What is your opinion on using VictoriaMetrics over HA Prometheus setup for long-term storage? This would allow me to reduce the retention period on the individual Prometheus instance and put the majority of the data on VictoriaMetrics' head. This way I'll have to manage only a single huge persistent storage disk.
On Saturday, October 10, 2020 at 1:24:50 PM UTC+5:30 Stuart Clark wrote: > On 10/10/2020 08:43, [email protected] wrote: > > d) If we do use 2 separate disks for the 2 instances, how will we manage > the config files? I mean is there any way to make the changes on any one > instance and those get replicated to other instance automatically or will > we have to do that manually? > > On Saturday, October 10, 2020 at 12:36:25 PM UTC+5:30 [email protected] > wrote: > >> Hi. I have a vanilla Prometheus setup with 50 days retention and data >> size of around 4.6TB for this much retention. I want to move to HA set up >> to avoid a single point of failure. >> I'm a little confused on how to approach the below points: >> >> a) With a HA pair, does the Prometheus data necessarily be local to both >> the Prometheus instances? Because it would require me to provision 2 5TB >> disks, one for each instance. >> Is it a good idea to have these 2 Prometheus instances write to an NFS >> disk? >> >> b) With both the HA pairs scraping the same targets, how do I build a >> global view of these local Prometheus instances? Is federation preferable >> or is there any other better way to approach this? >> >> c) Since, 2 instances will be scraping the same targets, does it add any >> overhead to the target side? >> >> Thanks in advance. >> > a) Yes you would have two disks. NFS is not recommended for a number of > reasons including performance. Additionally it would create a single point > of failure which could break both machines at once. Additionally with NFS > it is easy to accidentally try to have both instances pointing to the same > storage location which would cause data corruption. > > b) There are a number of solutions, but one option with be to run promxy > in front of Prometheus (so things like Grafana would query promxy) which > will query both servers to create a single view, removing any gaps. > > c) Exporters and client libraries are designed to be low impact when used > correctly, so additional scraping should have minimal impact. > > d) Control of you config files is down to whatever container/configuration > management solution you use. Generally the only difference between the > servers might be external label settings. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2c04fa81-30f7-4c0b-8ec2-828048cb095bn%40googlegroups.com.
