On 2020-11-20 12:25, Andy Pan wrote:
Got it! Thank you very much! About the inconsistent metrics, is it possible to dispatch metrics by hash and put those metrics with the same labels into the same push gateway, avoiding the inconsistent metrics?
You were originally talking about having multiple push gateway servers to remove single points of failure. Having different instances of push gateway for different subsets of usage is a fairly normal pattern, either due to performance, error domain or location reasons, but it doesn't create any form of high availability.
A "normal" push gateway setup, with auto-restart in case of failure (e.g. Kubernetes, systemd, auto scaling group), optionally with state storage on disk is usually adequate to handle failures (which should be pretty rare). If there was a failure there would be a temporary outage for however long it takes to restart the application (shouldn't be more than a few minutes). You would miss data from any attemted pushes during that outage period.
In general Prometheus doesn't have any guarantee that metric deliver will always succeed. Scrape failures do occasionally happen, due to networking blips, timeouts or other transient issues.
One alternative to the push gateway that some use is the textfile collector of the node exporter. This is particularly useful for cron jobs that run on a server. One advantage is that the failure domains are more tightly coupled - if the server fails the cron job will also fail alongside the node exporter. Equally if the node exporter fails or has a blip it doesn't stop the cron job from recording metrics, as that is just a file creation operation on the server rather than requiring an API call.
-- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d9b131ac9e015bcf3cb828c3d75f3b27%40Jahingo.com.
