On 10/10/2020 08:43, [email protected] wrote:
d) If we do use 2 separate disks for the 2 instances, how will we
manage the config files? I mean is there any way to make the changes
on any one instance and those get replicated to other instance
automatically or will we have to do that manually?
On Saturday, October 10, 2020 at 12:36:25 PM UTC+5:30
[email protected] wrote:
Hi. I have a vanilla Prometheus setup with 50 days retention and
data size of around 4.6TB for this much retention. I want to move
to HA set up to avoid a single point of failure.
I'm a little confused on how to approach the below points:
a) With a HA pair, does the Prometheus data necessarily be local
to both the Prometheus instances? Because it would require me to
provision 2 5TB disks, one for each instance.
Is it a good idea to have these 2 Prometheus instances write to an
NFS disk?
b) With both the HA pairs scraping the same targets, how do I
build a global view of these local Prometheus instances? Is
federation preferable or is there any other better way to approach
this?
c) Since, 2 instances will be scraping the same targets, does it
add any overhead to the target side?
Thanks in advance.
a) Yes you would have two disks. NFS is not recommended for a number of
reasons including performance. Additionally it would create a single
point of failure which could break both machines at once. Additionally
with NFS it is easy to accidentally try to have both instances pointing
to the same storage location which would cause data corruption.
b) There are a number of solutions, but one option with be to run promxy
in front of Prometheus (so things like Grafana would query promxy) which
will query both servers to create a single view, removing any gaps.
c) Exporters and client libraries are designed to be low impact when
used correctly, so additional scraping should have minimal impact.
d) Control of you config files is down to whatever
container/configuration management solution you use. Generally the only
difference between the servers might be external label settings.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4c951415-9ec4-0fb2-38fd-ead84a05d42c%40Jahingo.com.
