So now the case is simply that the target exporter is not reachable from the prometheus server (as demonstrated by curl).
This is not a prometheus problem, it's a network problem that you have to debug. Maybe it involves network ACLs, firewall rules, iptables on the remote host etc. On Wednesday, 13 October 2021 at 15:08:44 UTC+1 [email protected] wrote: > Yes I understand that if up returns 0, that means scraping failed. That > was inconsistent with the behavior I was seeing on the Prometheus web UI, > where it was still returning metrics from the server even though it was > returning 0 for up(). It didn't make any sense. > > After Prometheus was restarted yesterday afternoon the behavior is now as > expected and the metrics accurately reflect that the host can't be reached > and can't be scraped. > > > > On Wednesday, October 13, 2021 at 9:34:01 AM UTC-4 Brian Candler wrote: > >> That's a contradiction. If up is 0, it means that scraping failed, so >> you can't have any metrics. >> >> You can query the local Prometheus API to prove this one way or the other: >> >> promtool query instant http://localhost:9090 'up{job="node"}' # adjust >> job name as required to match your scrape job >> promtool query instant http://localhost:9090 '{job="node"}' >> >> Or if you don't have promtool, you can get something similar using curl: >> curl -g 'http://localhost:9090/api/v1/query?query=up{job="node"}' | >> python3 -m json.tool >> >> I find it highly unlikely that you'll find the same instance with up=0 >> but with metrics collected. >> >> Also: if you see up=0, you should also see the scraping problem in the >> prometheus web UI. Go to http://x.x.x.x:9090 and select Status > >> Targets. >> >> I think that up=0 is consistent with the problem you originally reported, >> i.e. being unable to scrape the targets directly using curl from the >> prometheus server. If you're seeing metrics somewhere else, then my only >> guess is that you're looking at a completely different prometheus server. >> >> On Wednesday, 13 October 2021 at 14:08:40 UTC+1 [email protected] wrote: >> >>> Would anyone happen to know what is causing this behavior? up() PromQL >>> statements return 0 for the hosts in question, but they are all scraping >>> metrics successfully. >>> >>> I don't believe I see any service discovery or pushes from the host, >>> these were all set up traditionally to be scraped from Prometheus. >>> >>> On Monday, October 11, 2021 at 11:02:51 AM UTC-4 Kristopher Kahn wrote: >>> >>>> I'm able to use PromQL and return metrics in the dashboard for >>>> *endpoint.example.com:9100 >>>> <http://endpoint.example.com:9100> *but when actually on the >>>> Prometheus host and running *curl endpoint.example.com:9100/metrics >>>> <http://endpoint.example.com:9100/metrics>* nothing is returned and >>>> the request just times out. >>>> >>>> What might be causing this behavior? I looked through the Prometheus >>>> config but could not see anything special about what DNS resolver it's set >>>> to use. I would imagine it's using the system-wide resolvers. This is a >>>> systemd host - Ubuntu 18.04.1. >>>> >>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/954224f8-0224-4340-b541-e0119dde5e91n%40googlegroups.com.
