HI team I am using the Prometheus instance having the more than 1 replica, When replica as 1 there is no wal corruption in data directory and now for the sake of zero down time updates for instance I make the replicas count as 2 the instance is up now but the wal corruptions are happening the logs are below
level=error ts=2020-12-22T04:36:00.860Z caller=scrape.go:1076 component="scrape manager" scrape_pool=depl/node-exporter/0 target=http://x.x.x.x:9100/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000003: stale NFS file handle" level=error ts=2020-12-22T04:36:00.862Z caller=scrape.go:1076 component="scrape manager" scrape_pool=depl/prometheus-kubelet/0 target=https://x.x.x.x:10250/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000003: stale NFS file handle" level=error ts=2020-12-22T04:36:00.881Z caller=scrape.go:1076 component="scrape manager" scrape_pool=depl/prometheus-kubelet/0 target=https://x.x.x.x:10250/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000003: stale NFS file handle" level=error ts=2020-12-22T04:36:00.898Z caller=scrape.go:1076 component="scrape manager" scrape_pool=depl/prometheus-kubelet/0 target=https://x.x.x.x:10250/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000003: stale NFS file handle" level=error ts=2020-12-22T04:36:00.970Z caller=scrape.go:1076 component="scrape manager" scrape_pool=depl/node-exporter/0 target=http://x.x.x.x:9100/metrics msg="Scrape commit failed" err="write to WAL: log samples: write /prometheus/wal/00000003: stale NFS file handle" Getting more logs like this at one replica no errors but when i am using the more than 1 replica getting above errors. Or is there any other way for prometheus zero down time and why does i am getting this errors but if i used the replicas as 1 there is no errors in data directory this is happening more than 1 replica Thank you mohan nagandlla -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALSpdGnJi39HKoNrdZQamGYEF5%2BwYwXfNfrzGJk4E8bkQQ43FA%40mail.gmail.com.
