Hi > Can we configure multiple remote-write receivers in agent mode?
Yes, you can. All what's written in Remote Write tuning post (https://prometheus.io/docs/practices/remote_write/) applies to both agent and normal mode. Remote Write code in Prometheus does not care what type of TSDB DB Prometheus is running with (Agent or normal). It just reads from the WAL. Agent mode checks all active remote write queues to check what could be removed and removes with some delay. The only consequence of adding new remote write endpoints after a while is that new endpoint will not receive data that were already streamed to other endpoints (minus some buffer). > Is federation supported in agent mode? Not at the moment, and there are no plans for that. Feel free to add feature request, but generally users prefer remote write to "federate"their data - better accuracy and consistency of data and better integration with other systems. Hope that helps! (: Kind Regards, Bartek Płotka (@bwplotka) On Friday, June 9, 2023 at 5:31:33 PM UTC+2 Brian Candler wrote: > Perhaps you could test it? It certainly *accepts* multiple URLs in agent > mode: > > $ *cat test-agent.yml* > remote_write: > - url: http://127.0.0.1:1234/ > - url: http://127.0.0.1:1235/ > $ */opt/prometheus/prometheus --config.file=test-agent.yml > --storage.agent.path=/var/tmp/agent --enable-feature=agent > --web.listen-address=127.0.0.1:9091 <http://127.0.0.1:9091>* > ... > ts=2023-06-09T15:22:27.935Z caller=main.go:957 level=info msg="Server is > ready to receive web requests." > $ *curl -fsS localhost:9091/metrics | grep remote_storage_samples_pending* > # HELP prometheus_remote_storage_samples_pending The number of samples > pending in the queues shards to be sent to the remote storage. > # TYPE prometheus_remote_storage_samples_pending gauge > prometheus_remote_storage_samples_pending{remote_name="5e94e3",url=" > http://127.0.0.1:1234/"} 0 > prometheus_remote_storage_samples_pending{remote_name="d296a2",url=" > http://127.0.0.1:1235/"} 0 > > Whether it actually *delivers* to both endpoint or not, I have not > tested. However, I looked in the source code under tsdb/agent/db.go (the > WAL-only database used in agent mode) and I see nothing special that > suggests it wouldn't work. > > If you find that it doesn't work with multiple remote_write endpoints, > then it's a bug one way or the other. At very least, it ought to be changed > to reject a config like the one above. > > On Thursday, 8 June 2023 at 22:11:56 UTC+1 tejaswini vadlamudi wrote: > >> Thanks, Brain for the answer! >> >> I think Prometheus Agent supports only one remote-write end-point by >> implementation (not due to config), please correct me if I'm wrong. >> +1 to your answer on federation. >> >> On Thursday, June 8, 2023 at 9:35:24 AM UTC+2 Brian Candler wrote: >> >>> On Wednesday, 7 June 2023 at 23:41:31 UTC+1 tejaswini vadlamudi wrote: >>> >>> Can we configure multiple remote-write receivers in agent mode? >>> >>> >>> Good question. The configuration file >>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file> >>> >>> allows a list of remote_write receivers: >>> >>> # Settings related to the remote write feature. remote_write: [ - >>> <remote_write> >>> <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write> >>> >>> ... ] >>> ... but since agent mode >>> <https://prometheus.io/blog/2021/11/16/agent/#prometheus-agent-mode> uses >>> a "customized TSDB WAL" that deletes data as soon as it has been >>> successfully delivered, I don't know if it can handle the case where one >>> receiver is backlogging but another is not. >>> >>> Is federation supported in agent mode? >>> >>> >>> I would expect not, since federation involves making queries to the >>> prometheus database - and agent mode "disables querying, alerting, and >>> local storage" completely. >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b6345fed-bc93-4477-bf04-02dee2aece4bn%40googlegroups.com.
