I found several posts which ask if it is possible that prometheus agent can store the metrics locally if remote_write destination is not available.
This post describes that it should work at least for 2hrs.: https://prometheus.io/blog/2021/11/16/agent/ However here is a guthub issue which seems to describe the same issue I saw in my environment: https://github.com/prometheus/prometheus/issues/11919 Can someone confirm this is working with 2.49.1 to store metrics for some time on prometheus agent until remotw_write is available again? if yes how did you configure this? Alexander Wilke schrieb am Dienstag, 12. März 2024 um 23:16:10 UTC+1: > Hello, > > I am running a prometheus server as a central instance. > I configured prometheus in agent mode on another system and send the logs > via remote_write from agent to central prometheus. This works. > > I changed the password in web.config.yml on the central server so it does > not match for the prometheus agent anymore. I could see that metrics did > not arrive anymore. > > I waited form 2-3 minutes and then reconfigured the correct password on > the central prometheus, di a reload and I coud see metrics arrive > immediately after that change. Unfortunately I did not receive the metrics > from the outage time. > > my main intention was exactly this - that I can get the metrics if there > is an issue in the network or whatever between agent and central prometheus. > > Is this expected behaviour or misconfiguration/misunderstanding on my site? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f257c25-700a-4ed7-bdff-87e4fbcaabe9n%40googlegroups.com.
