Thanks for the comprehensive information! > As you can see some job defined in the server config (push_gateway, istio) are absent from the UI config.
In general, the config UI should show exactly the same configuration as contained in your prometheus.env.yaml. Perhaps you just need to ask Prometheus to reload its configuration <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#:~:text=Prometheus%20can%20reload%20its%20configuration,%2Dlifecycle%20flag%20is%20enabled).> ? I am not an expert in the Prometheus Operator, but if you are making modifications to a file that the operator expects to control, then they may be reasonably overwritten. The operator expects to be configured by PodMonitor and ServiceMonitor-like kubernetes objects. If reloading the config does not give you results you are expecting then it may be an idea to raise with #prometheus-operator <https://kubernetes.slack.com/archives/CFFDS2Z7F>, who will be able to give more specific advice. Best, Ian On Tue, Jul 6, 2021 at 1:58 PM Bastien Moyet <[email protected]> wrote: > Sure, > > You can find both config on the following gist : > https://gist.github.com/bmoyet/a3438a881d12caa4b4b9e2b109cdeda8 > > I added config_ui.yaml which is what the UI is reporting to me. And > prometheus.env.yaml which is what you can find in the prometheus config > path (/etc/prometheus/config_out/prometheus.env.yaml). As you can see some > job defined in the server config (push_gateway, istio) are absent from the > UI config. > > Targets UI is reporting only the jobs defined in the Config UI, which is > htmltoimage-metrics, htmltoimage-metrics, htmltoimage-metrics and a whole > lot of monitoring/xxx > > In the logs I have from the docker container (kubectl logs...) I cannot > find any relevant information... Mainly some rule manager warning and some > info reports like : > > level=info ts=2021-07-06T09:00:48.938Z caller=compact.go:495 > component=tsdb msg="write block" mint=1625551200000 maxt=1625558400000 > ulid=01F9XGQAQGF5GNVQYNXQEC3ASW duration=48.826038743s > level=info ts=2021-07-06T09:01:03.783Z caller=head.go:662 component=tsdb > msg="Head GC completed" duration=4.365682297s > level=info ts=2021-07-06T09:01:26.172Z caller=head.go:734 component=tsdb > msg="WAL checkpoint complete" first=75464 last=75475 duration=22.389155689s > level=info ts=2021-07-06T09:02:14.296Z caller=compact.go:441 > component=tsdb msg="compact blocks" count=3 mint=1625529600000 > maxt=1625551200000 ulid=01F9XGT72PP4D2N8FHJZ2606VC > sources="[01F9WW4519JMZAVQTXABY99TD5 01F9X2ZWA131TPVJBT6BNYQGR5 > 01F9X9VKJ1VFRNRWM87AY209PB]" duration=39.617840386s > > > If I can supply any other information that might help I will be glad to do > so. > On Tuesday, 6 July 2021 at 11:57:17 UTC+2 [email protected] wrote: > >> Bastien, >> >> Hmm interesting. At this point it would be helpful if you could share >> some more information so that we can help you figure out what is wrong here >> - I would be interested to know what configuration is reported in the >> Configuration >> UI <https://prometheus.demo.do.prometheus.io/config>, and what is being >> reported in the Targets UI >> <https://prometheus.demo.do.prometheus.io/targets>. The logs from >> Prometheus itself almost always contains useful information too. >> >> Best, >> >> Ian >> >> On Tue, Jul 6, 2021 at 8:59 AM Bastien Moyet <[email protected]> wrote: >> >>> Hi, >>> >>> Thanks for your answer. >>> >>> That's what I thought too, but I'm not seeing any push_gateway in the >>> UI, neither healthy nor unhealthy. As if prometheus wasn't using my >>> configuration. >>> >>> That's why I'm asking my question here, and if there is a way to see why >>> my configuration is not used. >>> >>> On Monday, 5 July 2021 at 17:54:13 UTC+2 [email protected] wrote: >>> >>>> Hi Bastien, >>>> >>>> If your prometheus instance is correctly configured to scrape metrics >>>> from push_gateway, you should be able to see this in the Prometheus > >>>> Status > Targets UI. It should look 'healthy' like the following from the >>>> demo >>>> prometheus <https://prometheus.demo.do.prometheus.io/targets>: >>>> >>>> Best, >>>> >>>> Ian >>>> >>>> On Mon, Jul 5, 2021 at 8:16 AM Bastien Moyet <[email protected]> >>>> wrote: >>>> >>>>> Otherwise, are there any logs I can find regarding which endpoints are >>>>> effectively scrapped, or if there is an error scrapping them ? Though I >>>>> suspect there is no error since my prometheus gateway is not showing up in >>>>> targets list. >>>>> >>>>> On Friday, 2 July 2021 at 10:39:19 UTC+2 Bastien Moyet wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I deployed a prometheus instance using prometheus operator (8.16.1) >>>>>> alongside a pushgateway (1.4.3) in a kubernetes cluster. But I cannot >>>>>> access my pushgateway metrics from prometheus. >>>>>> >>>>>> My pushgateway is correctly deployed, accessible and its data can be >>>>>> viewed on its UI. >>>>>> >>>>>> Prometheus is up and running too. However I cannot find the >>>>>> pushgateway in the listed target on http://prometheus:9090/targets . >>>>>> That seemed logic at first since the http://prometheus:9090/config >>>>>> page does not show any pushgateway either. >>>>>> >>>>>> However if I go check the config file I can find in >>>>>> http://prometheus:9090/flags >>>>>> (/etc/prometheus/config_out/prometheus.env.yaml) it gives me differents >>>>>> values than the one I can find on the UI. >>>>>> >>>>>> Notably in my pod the config file contains : >>>>>> >>>>>> - honor_labels: true >>>>>> job_name: pushgateway >>>>>> static_configs: >>>>>> - targets: >>>>>> - prometheus-pushgateway:9091 >>>>>> >>>>>> So i'm wondering why the config files differs from the config >>>>>> displayed on the UI. Is it a normal behavior or does it explain my issue >>>>>> ? >>>>>> >>>>>> Thanks >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CADoMBNVTw1qG0SRYJEDtr3Cxs2jATuUkq8o3trawyX03W0StqQ%40mail.gmail.com.
