Yes, the only good option to retain data in remote nodes like this is to have Prometheus running in the remote location. If it's one-off nodes, you'll need a localhost Prometheus. The good news is, this is relatively efficient, as the minimum footprint of Prometheus is pretty small.
As for how to get the data home, this sounds like a use case for remote write or Thanos sidecars. With remote write, the systems will stream the data up to a central service, this is tracked via the write ahead log, so you won't have any data loss. With the sidecar, it will upload blocks of data every 2 hours, and require a reverse connection for reading recent data. The nice thing about remote write here is that no VPN is necessary, it can stream securely via https. I recommend Cortex or Thanos Remote Write server to receive the data from the remote host Prometheus servers. On Sat, Aug 22, 2020 at 8:25 AM James Peverill <[email protected]> wrote: > I have an application where I am trying to aggregate metrics from hosts > that have spotty connectivity. I want to be logging locally when the hosts > are periodically offline, then sync all the metrics to a central server > once they connect again. Another complication is that these remote hosts > sometimes get rebooted. In the ideal case the remote hosts would delete > data locally after some amount of time, allowing plenty of time for it all > to be synced to the centralized server. > > It sounds like I could run Prometheus on all my remote hosts, with > appropriate retention rules. Then have my central Prometheus server pull > from them all periodically via remote_read? Will I be able to get ALL data > to the central server that way? All the hosts are connected over a VPN so > there are no firewall issues. > > Does this sound like an appropriate use of Prometheus? > > Thank you! > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/29c35e2a-58f7-48b7-ad99-28e890eacc26n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/29c35e2a-58f7-48b7-ad99-28e890eacc26n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqpWW3aNozzbhPLS8%3DiduRLc7MimbOD3Z%2B%3DhVONwP4gmA%40mail.gmail.com.
