Try substituting Prometheus with vmagent inside the K8S and pushing metrics to a remote storage outside the Kubernetes. See https://victoriametrics.github.io/vmagent.html#drop-in-replacement-for-prometheus
On Thu, Feb 11, 2021 at 7:29 PM David Frank <[email protected]> wrote: > Hey community, I am currently running Prometheus inside a K8s cluster in > our production. However, we have frequent updates to K8s version and this > keeps restarting our pods, causing outage for couple of hours (WAL > loading). > > Is it possible to run Prometheus on a VM instance outside the K8s cluster > but have service discovery to scrape the metrics from the pods running > inside K8s cluster? > I have had no success so far to get this to work. > > Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/0702263b-7219-480c-8346-ed0289345b8bn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/0702263b-7219-480c-8346-ed0289345b8bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Best Regards, Aliaksandr Valialkin, CTO VictoriaMetrics -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmD2zpEuiYy_nRrVmqdUZ4Aqnx9ZgGGFB1RboVcVMU8Sgw%40mail.gmail.com.
