On 02/06/2021 11:01, nina guo wrote:
So the better solution would be mount another storage rather than NFS
separately to each Pod.
Yes.
For example, 2 Prometheus Pods are running with 2 separate volumes, if
one of the Pod goes down(but the data is still in memory), according
to k8s mechanism, another Pod will be started automatically. Currently
the data which was in memeroy will be lost. It will cause data
inconsistency. Because the other running Pod probably already have
written the data to persistent volume.
The two instances will record different data to each other, as they will
be scraping the common set of targets at slightly different times. The
usual way to handle this (as well as deal with gaps due to
restarts/errors) is to use a "sticky" load balancer or something like
Thanos or promxy.
Anything which is purely in memory and not written to disk when
Prometheus crashes (or is forceable destroyed) will be lost, but
Prometheus tries to write to the WAL regularly to reduce that risk. When
restarted (assuming the same storage is reattached) the WAL is read back
into memory.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/e202144c-c1e5-f11e-6221-c4bd88791227%40Jahingo.com.
