That's a contradiction. If up is 0, it means that scraping failed, so you can't have any metrics.
You can query the local Prometheus API to prove this one way or the other: promtool query instant http://localhost:9090 'up{job="node"}' # adjust job name as required to match your scrape job promtool query instant http://localhost:9090 '{job="node"}' Or if you don't have promtool, you can get something similar using curl: curl -g 'http://localhost:9090/api/v1/query?query=up{job="node"}' | python3 -m json.tool I find it highly unlikely that you'll find the same instance with up=0 but with metrics collected. Also: if you see up=0, you should also see the scraping problem in the prometheus web UI. Go to http://x.x.x.x:9090 and select Status > Targets. I think that up=0 is consistent with the problem you originally reported, i.e. being unable to scrape the targets directly using curl from the prometheus server. If you're seeing metrics somewhere else, then my only guess is that you're looking at a completely different prometheus server. On Wednesday, 13 October 2021 at 14:08:40 UTC+1 [email protected] wrote: > Would anyone happen to know what is causing this behavior? up() PromQL > statements return 0 for the hosts in question, but they are all scraping > metrics successfully. > > I don't believe I see any service discovery or pushes from the host, these > were all set up traditionally to be scraped from Prometheus. > > On Monday, October 11, 2021 at 11:02:51 AM UTC-4 Kristopher Kahn wrote: > >> I'm able to use PromQL and return metrics in the dashboard for >> *endpoint.example.com:9100 >> <http://endpoint.example.com:9100> *but when actually on the Prometheus >> host and running *curl endpoint.example.com:9100/metrics >> <http://endpoint.example.com:9100/metrics>* nothing is returned and the >> request just times out. >> >> What might be causing this behavior? I looked through the Prometheus >> config but could not see anything special about what DNS resolver it's set >> to use. I would imagine it's using the system-wide resolvers. This is a >> systemd host - Ubuntu 18.04.1. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6b4907ef-b706-45cb-ab07-d769eed958een%40googlegroups.com.
