[prometheus-users] Re: prometheus agent 2.49.1 - remote_write metrics while disconnected
I found several posts which ask if it is possible that prometheus agent can store the metrics locally if remote_write destination is not available. This post describes that it should work at least for 2hrs.: https://prometheus.io/blog/2021/11/16/agent/ However here is a guthub issue which seems to describe the same issue I saw in my environment: https://github.com/prometheus/prometheus/issues/11919 Can someone confirm this is working with 2.49.1 to store metrics for some time on prometheus agent until remotw_write is available again? if yes how did you configure this? Alexander Wilke schrieb am Dienstag, 12. März 2024 um 23:16:10 UTC+1: > Hello, > > I am running a prometheus server as a central instance. > I configured prometheus in agent mode on another system and send the logs > via remote_write from agent to central prometheus. This works. > > I changed the password in web.config.yml on the central server so it does > not match for the prometheus agent anymore. I could see that metrics did > not arrive anymore. > > I waited form 2-3 minutes and then reconfigured the correct password on > the central prometheus, di a reload and I coud see metrics arrive > immediately after that change. Unfortunately I did not receive the metrics > from the outage time. > > my main intention was exactly this - that I can get the metrics if there > is an issue in the network or whatever between agent and central prometheus. > > Is this expected behaviour or misconfiguration/misunderstanding on my site? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f257c25-700a-4ed7-bdff-87e4fbcaabe9n%40googlegroups.com.
[prometheus-users] Re: Best practive: "job_name in prometheus agent? Same job_name allowed ?
Thanks for your response. What would you recommend in a situation with several hundredes or thousands of servers or systems within a kubernetes cluster which should have the node_exporter installed. my idea was to install the node_exporter + prometheus agent. agent scrapes local node_exporter and then remotw_writes the results to a central prometheus server or a loadbalancer which distributes to different prometheus servers. my idea was to user the same configu for alle node_exporter + prometheus agents. For that reason they all have the same job name which would be ok. However I think I will have a problem because if I use "127.0.0.1:9100" as target to scrape then all instances are equal. Is there any possibility to use a variable in the scrape_config which reflects any environment variable from linux system or any other mechanism to make this instance unique? Brian Candler schrieb am Donnerstag, 14. März 2024 um 13:04:07 UTC+1: > As long as all the time series have distinct label sets (in particular, > different "instance" labels), and you're not mixing scraping with > remote-writing for the same targets, then I don't see any problem with all > the agents using the same "job" label when remote-writing. > > On Tuesday 12 March 2024 at 22:30:22 UTC Alexander Wilke wrote: > >> At the moment I am running the job with name >> "node_exporter" which has 20 different targets. (instances) >> With this configuration there should not be any conflict. >> >> my idea is to install the prometheus agent on the nodes itself. >> technically it looks like it work if I use the same job_name on the agent >> and central prometheus as long as the targets/instances are different. >> >> In general I avoid conflicting job_names but in this situation it may be >> ok from my point of view. >> >> what do you think, recommend in this specific scenario ? >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/706133c4-aa70-4d60-b1a0-dc0d85bcd5een%40googlegroups.com.
[prometheus-users] Re: Best practive: "job_name in prometheus agent? Same job_name allowed ?
As long as all the time series have distinct label sets (in particular, different "instance" labels), and you're not mixing scraping with remote-writing for the same targets, then I don't see any problem with all the agents using the same "job" label when remote-writing. On Tuesday 12 March 2024 at 22:30:22 UTC Alexander Wilke wrote: > At the moment I am running the job with name > "node_exporter" which has 20 different targets. (instances) > With this configuration there should not be any conflict. > > my idea is to install the prometheus agent on the nodes itself. > technically it looks like it work if I use the same job_name on the agent > and central prometheus as long as the targets/instances are different. > > In general I avoid conflicting job_names but in this situation it may be > ok from my point of view. > > what do you think, recommend in this specific scenario ? > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/24268b8d-0313-44d5-8227-3be95eaacde7n%40googlegroups.com.
