Sure, You can find both config on the following gist : https://gist.github.com/bmoyet/a3438a881d12caa4b4b9e2b109cdeda8
I added config_ui.yaml which is what the UI is reporting to me. And prometheus.env.yaml which is what you can find in the prometheus config path (/etc/prometheus/config_out/prometheus.env.yaml). As you can see some job defined in the server config (push_gateway, istio) are absent from the UI config. Targets UI is reporting only the jobs defined in the Config UI, which is htmltoimage-metrics, htmltoimage-metrics, htmltoimage-metrics and a whole lot of monitoring/xxx In the logs I have from the docker container (kubectl logs...) I cannot find any relevant information... Mainly some rule manager warning and some info reports like : level=info ts=2021-07-06T09:00:48.938Z caller=compact.go:495 component=tsdb msg="write block" mint=1625551200000 maxt=1625558400000 ulid=01F9XGQAQGF5GNVQYNXQEC3ASW duration=48.826038743s level=info ts=2021-07-06T09:01:03.783Z caller=head.go:662 component=tsdb msg="Head GC completed" duration=4.365682297s level=info ts=2021-07-06T09:01:26.172Z caller=head.go:734 component=tsdb msg="WAL checkpoint complete" first=75464 last=75475 duration=22.389155689s level=info ts=2021-07-06T09:02:14.296Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1625529600000 maxt=1625551200000 ulid=01F9XGT72PP4D2N8FHJZ2606VC sources="[01F9WW4519JMZAVQTXABY99TD5 01F9X2ZWA131TPVJBT6BNYQGR5 01F9X9VKJ1VFRNRWM87AY209PB]" duration=39.617840386s If I can supply any other information that might help I will be glad to do so. On Tuesday, 6 July 2021 at 11:57:17 UTC+2 [email protected] wrote: > Bastien, > > Hmm interesting. At this point it would be helpful if you could share some > more information so that we can help you figure out what is wrong here - I > would be interested to know what configuration is reported in the > Configuration > UI <https://prometheus.demo.do.prometheus.io/config>, and what is being > reported in the Targets UI > <https://prometheus.demo.do.prometheus.io/targets>. The logs from > Prometheus itself almost always contains useful information too. > > Best, > > Ian > > On Tue, Jul 6, 2021 at 8:59 AM Bastien Moyet <[email protected]> wrote: > >> Hi, >> >> Thanks for your answer. >> >> That's what I thought too, but I'm not seeing any push_gateway in the UI, >> neither healthy nor unhealthy. As if prometheus wasn't using my >> configuration. >> >> That's why I'm asking my question here, and if there is a way to see why >> my configuration is not used. >> >> On Monday, 5 July 2021 at 17:54:13 UTC+2 [email protected] wrote: >> >>> Hi Bastien, >>> >>> If your prometheus instance is correctly configured to scrape metrics >>> from push_gateway, you should be able to see this in the Prometheus > >>> Status > Targets UI. It should look 'healthy' like the following from the >>> demo >>> prometheus <https://prometheus.demo.do.prometheus.io/targets>: >>> >>> Best, >>> >>> Ian >>> >>> On Mon, Jul 5, 2021 at 8:16 AM Bastien Moyet <[email protected]> >>> wrote: >>> >>>> Otherwise, are there any logs I can find regarding which endpoints are >>>> effectively scrapped, or if there is an error scrapping them ? Though I >>>> suspect there is no error since my prometheus gateway is not showing up in >>>> targets list. >>>> >>>> On Friday, 2 July 2021 at 10:39:19 UTC+2 Bastien Moyet wrote: >>>> >>>>> Hi, >>>>> >>>>> I deployed a prometheus instance using prometheus operator (8.16.1) >>>>> alongside a pushgateway (1.4.3) in a kubernetes cluster. But I cannot >>>>> access my pushgateway metrics from prometheus. >>>>> >>>>> My pushgateway is correctly deployed, accessible and its data can be >>>>> viewed on its UI. >>>>> >>>>> Prometheus is up and running too. However I cannot find the >>>>> pushgateway in the listed target on http://prometheus:9090/targets . >>>>> That seemed logic at first since the http://prometheus:9090/config >>>>> page does not show any pushgateway either. >>>>> >>>>> However if I go check the config file I can find in >>>>> http://prometheus:9090/flags >>>>> (/etc/prometheus/config_out/prometheus.env.yaml) it gives me differents >>>>> values than the one I can find on the UI. >>>>> >>>>> Notably in my pod the config file contains : >>>>> >>>>> - honor_labels: true >>>>> job_name: pushgateway >>>>> static_configs: >>>>> - targets: >>>>> - prometheus-pushgateway:9091 >>>>> >>>>> So i'm wondering why the config files differs from the config >>>>> displayed on the UI. Is it a normal behavior or does it explain my issue ? >>>>> >>>>> Thanks >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com.
