Hmm, I see your use case, and that restarting the exporter when it's the sidecar of a stateful pod is painful. However, wherever we have introduced live reload (such as the statsd exporter) it has brought a lot of complexity and potential bugs that need to be taken into account for all subsequent development.
Can you say more about the use case for turning individual collectors on or off while a pod is running? Under what circumstances do you need to do this? Would it be an on/off for everything at once, or do you need to enable specific collectors for specific pods at specific times? There's a mechanism to filter collectors at scrape time <https://github.com/prometheus/mysqld_exporter#filtering-enabled-collectors>, i.e. in *Prometheus* configuration. I believe you may be able to use that instead of incurring the cost of live reload: configure the exporter with the superset of collectors you may want for a given pod, then use the filter mechanism to only *request* the metrics that you need. With this, you can make use of Prometheus' service discovery (and the prometheus-exporter) to change what you actually collect at run time: either change the *Monitor to toggle a collector on for all monitored pods, or write relabeling rules to make use of a pod annotation to decide whether to include or exclude certain collectors. This will probably get very annoying if you are looking for a generic "let me configure every collector differently for every pod" mechanism, but it should be manageable for either "configure the list of collectors for *all* pods" (change the monitor) or "toggle extra debug collectors for a specific pod" (have a relabeling rule that matches a "debug yes/no" annotation). An alternative fallback solution would be to wrap the exporter in a binary (or shell script) that reads the list of exporters from a (configmap) file and restarts it as necessary. Would either of these solve the use case without incurring the maintenance overhead of live reload capabilities in the exporter? /MR On Mon, Jul 31, 2023 at 4:46 AM 'Max Englander' via Prometheus Developers < prometheus-developers@googlegroups.com> wrote: > Hello, > > I would like the propose adding to mysqld_exporter the ability to > /-/reload the scraper configuration. Currently scrapers are only > configurable via --collect.<name>[.arg] and cannot be reconfigured at > runtime, so this proposal also entails allowing collectors to be configured > via a file, and then to have the ability to reconfigure scrapers when the > contents of that file change and /-/reload is called. > > I opened a GH issue here > <https://github.com/prometheus/mysqld_exporter/issues/759>. It was closed > by Ben saying that the maintainers had no plans to implement, but didn't > say whether they would be open to a contribution, which I'd be happy to > offer if so. > > The reason this would be useful to me is that my organization runs > mysqld_exporter in a container co-located with a mysqld container in the > same Kubernetes pod. mysqld has a socket but no port so there's no way to > reconfigure the mysqld_exporter container without recreating the pod. I > realize this situation may not be common, but hopefully this idea would be > generally useful and merit consideration here. > > I have prototyped a solution locally that works pretty well and is fully > backwards compatible. > > Thanks, > Max > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/9c110603-3652-44c6-a196-1334dd1f1d65n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-developers/9c110603-3652-44c6-a196-1334dd1f1d65n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAMV%3D_gbYwVnGegCjXr2tEDS0_gjOK4i1T6mWO_duchD4BxL7ng%40mail.gmail.com.
