Thanks for your input. I tried to reload the configuration to no avail. You may be right that's an issue with prometheus operator I'll ask them.
On Tuesday, 6 July 2021 at 18:16:47 UTC+2 [email protected] wrote: > Thanks for the comprehensive information! > > > As you can see some job defined in the server config (push_gateway, > istio) are absent from the UI config. > > In general, the config UI should show exactly the same configuration as > contained in your prometheus.env.yaml. Perhaps you just need to ask > Prometheus to reload its configuration > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#:~:text=Prometheus%20can%20reload%20its%20configuration,%2Dlifecycle%20flag%20is%20enabled).> > ? > > I am not an expert in the Prometheus Operator, but if you are making > modifications to a file that the operator expects to control, then they may > be reasonably overwritten. The operator expects to be configured by > PodMonitor and ServiceMonitor-like kubernetes objects. > > If reloading the config does not give you results you are expecting then > it may be an idea to raise with #prometheus-operator > <https://kubernetes.slack.com/archives/CFFDS2Z7F>, who will be able to > give more specific advice. > > Best, > > Ian > > > > On Tue, Jul 6, 2021 at 1:58 PM Bastien Moyet <[email protected]> wrote: > >> Sure, >> >> You can find both config on the following gist : >> https://gist.github.com/bmoyet/a3438a881d12caa4b4b9e2b109cdeda8 >> >> I added config_ui.yaml which is what the UI is reporting to me. And >> prometheus.env.yaml which is what you can find in the prometheus config >> path (/etc/prometheus/config_out/prometheus.env.yaml). As you can see some >> job defined in the server config (push_gateway, istio) are absent from the >> UI config. >> >> Targets UI is reporting only the jobs defined in the Config UI, which is >> htmltoimage-metrics, htmltoimage-metrics, htmltoimage-metrics and a >> whole lot of monitoring/xxx >> >> In the logs I have from the docker container (kubectl logs...) I cannot >> find any relevant information... Mainly some rule manager warning and some >> info reports like : >> >> level=info ts=2021-07-06T09:00:48.938Z caller=compact.go:495 >> component=tsdb msg="write block" mint=1625551200000 maxt=1625558400000 >> ulid=01F9XGQAQGF5GNVQYNXQEC3ASW duration=48.826038743s >> level=info ts=2021-07-06T09:01:03.783Z caller=head.go:662 component=tsdb >> msg="Head GC completed" duration=4.365682297s >> level=info ts=2021-07-06T09:01:26.172Z caller=head.go:734 component=tsdb >> msg="WAL checkpoint complete" first=75464 last=75475 duration=22.389155689s >> level=info ts=2021-07-06T09:02:14.296Z caller=compact.go:441 >> component=tsdb msg="compact blocks" count=3 mint=1625529600000 >> maxt=1625551200000 ulid=01F9XGT72PP4D2N8FHJZ2606VC >> sources="[01F9WW4519JMZAVQTXABY99TD5 01F9X2ZWA131TPVJBT6BNYQGR5 >> 01F9X9VKJ1VFRNRWM87AY209PB]" duration=39.617840386s >> >> >> If I can supply any other information that might help I will be glad to >> do so. >> On Tuesday, 6 July 2021 at 11:57:17 UTC+2 [email protected] wrote: >> >>> Bastien, >>> >>> Hmm interesting. At this point it would be helpful if you could share >>> some more information so that we can help you figure out what is wrong here >>> - I would be interested to know what configuration is reported in the >>> Configuration >>> UI <https://prometheus.demo.do.prometheus.io/config>, and what is being >>> reported in the Targets UI >>> <https://prometheus.demo.do.prometheus.io/targets>. The logs from >>> Prometheus itself almost always contains useful information too. >>> >>> Best, >>> >>> Ian >>> >>> On Tue, Jul 6, 2021 at 8:59 AM Bastien Moyet <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> Thanks for your answer. >>>> >>>> That's what I thought too, but I'm not seeing any push_gateway in the >>>> UI, neither healthy nor unhealthy. As if prometheus wasn't using my >>>> configuration. >>>> >>>> That's why I'm asking my question here, and if there is a way to see >>>> why my configuration is not used. >>>> >>>> On Monday, 5 July 2021 at 17:54:13 UTC+2 [email protected] wrote: >>>> >>>>> Hi Bastien, >>>>> >>>>> If your prometheus instance is correctly configured to scrape metrics >>>>> from push_gateway, you should be able to see this in the Prometheus > >>>>> Status > Targets UI. It should look 'healthy' like the following from the >>>>> demo >>>>> prometheus <https://prometheus.demo.do.prometheus.io/targets>: >>>>> >>>>> Best, >>>>> >>>>> Ian >>>>> >>>>> On Mon, Jul 5, 2021 at 8:16 AM Bastien Moyet <[email protected]> >>>>> wrote: >>>>> >>>>>> Otherwise, are there any logs I can find regarding which endpoints >>>>>> are effectively scrapped, or if there is an error scrapping them ? >>>>>> Though I >>>>>> suspect there is no error since my prometheus gateway is not showing up >>>>>> in >>>>>> targets list. >>>>>> >>>>>> On Friday, 2 July 2021 at 10:39:19 UTC+2 Bastien Moyet wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I deployed a prometheus instance using prometheus operator (8.16.1) >>>>>>> alongside a pushgateway (1.4.3) in a kubernetes cluster. But I cannot >>>>>>> access my pushgateway metrics from prometheus. >>>>>>> >>>>>>> My pushgateway is correctly deployed, accessible and its data can be >>>>>>> viewed on its UI. >>>>>>> >>>>>>> Prometheus is up and running too. However I cannot find the >>>>>>> pushgateway in the listed target on http://prometheus:9090/targets >>>>>>> . That seemed logic at first since the http://prometheus:9090/config >>>>>>> page does not show any pushgateway either. >>>>>>> >>>>>>> However if I go check the config file I can find in >>>>>>> http://prometheus:9090/flags >>>>>>> (/etc/prometheus/config_out/prometheus.env.yaml) it gives me differents >>>>>>> values than the one I can find on the UI. >>>>>>> >>>>>>> Notably in my pod the config file contains : >>>>>>> >>>>>>> - honor_labels: true >>>>>>> job_name: pushgateway >>>>>>> static_configs: >>>>>>> - targets: >>>>>>> - prometheus-pushgateway:9091 >>>>>>> >>>>>>> So i'm wondering why the config files differs from the config >>>>>>> displayed on the UI. Is it a normal behavior or does it explain my >>>>>>> issue ? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Prometheus Users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/prometheus-users/8b19a767-c1d8-4190-8e37-cb9e16822dc2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Prometheus Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/prometheus-users/af839193-de1d-4148-b76f-e3de9c7308a2n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/34d73e9e-b756-49ce-9bb1-36e843c30922n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9c039f93-576e-4692-b4f5-14f3da9114b0n%40googlegroups.com.
