Hi I tried the things you mentioned but DNS is showing the correct address using dig and getent command. I think the Prometheus is caching the address because on another target, we had a DNS record "kafka-a.x.y.z" which was pointing to an IP address, now we deleted the DNS query and the dig command can't resolve the name "kafka-a.x.y.z" but the Prometheus is still reading metrics from it. On Tuesday, June 30, 2020 at 11:00:28 AM UTC+4:30 Christian Hoffmann wrote:
> Hi, > > > On 6/23/20 8:32 AM, Yashar Nesabian wrote: > > Hi, > > A few days ago I realized IPMI exporter is not running on one of our > > bare metals but we didn't get any alert from our Prometheus. Although I > > cannot get the metrics via curl on the Prometheus server, our Prometheus > > is scraping metrics successfully from this server! > > here is the Prometheus page indicating the Prometheus can scrap metrics > > successfully : > > > > prom.png > > > > > > But when I SSH to the server, no one is listening on port 9290: > > > > > > > > And I've checked the DNS records, they are correct (when I ping the > > address, it returns the correct address. Here is the curl result from > > the Prometheus server for one-08: > > | > > curl http://one-08.compute.x.y.z:9290 > > curl: (7) Failed to connect to one-08.compute.x.y.z port 9290: > > Connection refused > > | > > > > The weird thing is I can see one-08 metrics on the Prometheus server > > (for the moment): > > > > prom1.png > > > > > > > > I tried to put this job on another Prometheus server but I get an error > > on the second one claiming context deadline exceeded which is correct. > > Could a DNS cache be involved? > > Try comparing > getent hosts one-08 > vs. > dig one-08 > on the Prometheus machine. > > You can also try tcpdump to analyze where Prometheus is actually > connecting to. > > Kind regards, > Christian > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cf7e445d-fb1a-4b7c-af2c-86fd498e0333n%40googlegroups.com.
