Try looking at scrape_duration_seconds{job="Ping-All-Servers"}. Maybe it's
borderline to the scrape interval.
>> That's interesting. Here are the top 20 scrape_duration_seconds maxed
for last 1 hour by instance. Close to 5 seconds. Can this lead to some
issue? But again the thing comes why not Prometheus 2.12.0 having a
problem with it.
I'll try and correlate this with probe_success being zero.
[image: image.png]
What does min_over_time(up{job="Ping-All-Servers"}[5m]) show? In other
> words, is it the scrape to BBE which is failing, or the BBE probe? (I think
> the latter).
>
Is there a different network path between the two prometheus servers and
BBE?
>> No. All are under are same DC and under the same cluster of ESX hosts.
It still bothers me that BBE is logging panics. Something weird is going
on in your BBE. Could even be a hardware problem.
>> I can try and run the BBE on a different machine.
I think you should paste your entire scrape config and BBE config, in case
something else jumps out.
>> My Prometheus scrape config is quite huge and messy at the moment(Around
3k lines). I am attaching the blackbox jobs from the entire scrape. Let me
if you still need the entire scrape config, I'll share that too.
Attached bbe.yml and scrape.yml with Blackbox Jobs.
But after everything one questions still remains - *Why isn't Prometheus
2.12.0 showing this discrepancy in evaluation?*
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Prometheus Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/prometheus-users/oKAPrUljkU0/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/b466378c-050d-45ad-9910-7af69fc92d69o%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/b466378c-050d-45ad-9910-7af69fc92d69o%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAFGi5vC%2BB0tkHu9TctuFK9ftw5Rr3CGjsaq0W2u3%2BYXHrK_%2B-Q%40mail.gmail.com.
scrape.yml
Description: application/yaml
bbe.yml
Description: application/yaml
