Another option is to configure Prometheus instances to replicate data to remote storage via remote_write <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write>. Prometheus replicates data to the configured remote storage systems as soon as the data is scraped, so it shouldn't lose big amounts of data on unclean shutdown. See the list of supported remote storage systems here <https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage>. The most promising systems are: Cortex, M3DB and VictoriaMetrics. You can evaluate multiple systems at once - just add multiple `remote_write->url` entries in Prometheus config.
It is worth reading these docs <https://prometheus.io/docs/practices/remote_write/> on remote_write config tuning in Prometheus. On Mon, Mar 30, 2020 at 10:21 PM Shaam Dinesh <[email protected]> wrote: > Hi Brain > > Thanks for the response, yes I am still leveraging persistent disk to > mitigate the restarts but it was not helping to save 2 hours data as > configured > > I there any better way to address it > > On Tuesday, March 31, 2020 at 12:35:40 AM UTC+5:30, Brian Candler wrote: >> >> When you stop prometheus it writes out its WAL to disk, and when you >> start it it reads in WAL back from disk. This is why a prometheus restart >> can take several minutes (and you should ensure that your supervisor >> process isn't configured to do a hard kill after a short timeout). >> >> Of course it won't be ingesting data during that time, but if you have a >> second instance, that one still will be. >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/d1adfd24-a199-4f12-b85d-28394a7d52c0%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/d1adfd24-a199-4f12-b85d-28394a7d52c0%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Best Regards, Aliaksandr -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPbKnmDP_PFJ80fbhq48U7FMMB%3DaGNA5TF-Ao5OBjTWO2Ec%2BpA%40mail.gmail.com.
