Thanks Brian for sharing the details 👍 Thanks & Regards, Anoop
On Mon, Mar 7, 2022 at 5:29 PM Brian Candler <[email protected]> wrote: > > how to configure the annotation > > I'd say that's a kubernetes question, not a prometheus question. > > From the prometheus point of view: first you work out what exactly it is > you're trying to scrape (pods, services etc). Then you read in a list of > those entities using kubernetes_sd. Then you have some rewriting rules to > convert the metadata/labels/annotations into __address__ and > __metrics_path__. That's it. > > You say you have multiple containers in the same pod which need scraping > separately. I don't think Kubernetes exposes metadata at the container > level, only the pod level. Some options you could consider: > > 1. Create separate scrape jobs (one for /perf-metrics, one for > /app-metrics, one for /auth-metrics). Each job uses kubernetes_sd to get a > list of suitable pods to scrape. > Or: > 2. Create a separate 'service' for each container, and then use > kubernetes_sd to get a list of services to scrape. > Or: > 3. Looking at the kubernetes_sd docs > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpointslice> > : > > *The endpointslice role discovers targets from existing endpointslices. > For each endpoint address referenced in the endpointslice object one target > is discovered. If the endpoint is backed by a pod, all additional container > ports of the pod, not bound to an endpoint port, are discovered as targets > as well.* > > So maybe that will work too. That will just give you a list of port > numbers, so you may need to use fixed rewriting rules to map these to the > correct __metrics_path__. > > On Monday, 7 March 2022 at 11:47:14 UTC Anoop wrote: > >> Thanks Brian for answering my question. >> >> I am using Kubernetes SD and I believe my prometheus configuration is >> discovering targets correctly. My current use case is to see whether we can >> add multiple ports and paths configurations under the annotation section in >> a service? >> I am running multiple containers in a single pod and using a single >> service for this. Each container exposes metrics to their own port and >> metric path (not in /metrics path). So, how to configure the annotation in >> that single service to monitor the ports and paths of multiple containers? >> >> Thanks & Regards, >> Anoop >> >> >> On Mon, Mar 7, 2022 at 2:12 PM Brian Candler <[email protected]> wrote: >> >>> You didn't say anything about where this list of targets is coming from. >>> >>> If it's static, then very simply: >>> >>> - job_name: blah >>> static_configs: >>> - targets: >>> - 1.2.3.4:9288 >>> labels: >>> __metrics_path__: /perf-metrics >>> - targets: >>> - 1.2.3.4:9277 >>> labels: >>> __metrics_path__: /auth-metrics >>> - targets: >>> - 1.2.3.4:9266 >>> labels: >>> __metrics_path__: /app-metrics >>> >>> Or being a bit fancier with target relabelling, something like this >>> (untested): >>> >>> - job_name: blah >>> static_configs: >>> - targets: >>> - 1.2.3.4:9288/perf-metrics >>> - 1.2.3.4:9277/auth-metrics >>> - 1.2.3.4:9266/app-metrics >>> relabel_configs: >>> - source_labels: [__address__] >>> target_label: instance >>> - source_labels: [__address__] >>> regex: '([^/]+)(.*)' >>> target_label: __metrics_path__ >>> replacement: '$2' >>> - source_labels: [__address__] >>> regex: '([^/]+)(.*)' >>> target_label: __address__ >>> replacement: '$1' >>> >>> If it's coming from some service discovery mechanism, then you can use >>> target rewriting rules to replace the __address__ label with <addr>:<port> >>> and __metrics_path__ with <path>, where <addr>, <port> and <path> come from >>> some data returned by the service discovery mechanism. For example, with >>> kubernetes SD you can access the label values and annotations on >>> nodes/pods/services in __meta_XXX labels. >>> >>> On Monday, 7 March 2022 at 06:47:18 UTC Anoop wrote: >>> >>>> Hi, >>>> >>>> Any suggestions? >>>> >>>> Thanks, >>>> Anoop >>>> >>>> >>>> >>>> On Fri, Mar 4, 2022 at 12:28 PM Anoop <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have a kubernetes pod running with 3 containers in it. I am >>>>> exporting metrics to different port and path for each container. Is there >>>>> any way to enable monitoring for three ports and paths using a single >>>>> service? >>>>> >>>>> prometheus.io/path: /perf-metrics >>>>> prometheus.io/scrape_port: '9288' >>>>> >>>>> prometheus.io/path: /auth-metrics >>>>> prometheus.io/scrape_port: '9277' >>>>> >>>>> prometheus.io/path: /app-metrics >>>>> prometheus.io/scrape_port: '9266' >>>>> >>>>> Thanks, >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Prometheus Users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/prometheus-users/ca7ecdde-3efa-40ab-a5f1-62613a5ec742n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/prometheus-users/ca7ecdde-3efa-40ab-a5f1-62613a5ec742n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/bf343662-e122-40a3-8840-3d44b10e8e57n%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/bf343662-e122-40a3-8840-3d44b10e8e57n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/70a3faf2-859b-4706-92af-c2f75c5a4fbbn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/70a3faf2-859b-4706-92af-c2f75c5a4fbbn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEQqSpfpY6w_HcZSJ7fjrFkxBqYo6Ub%2BicoFNjdxpFRAoMnbqg%40mail.gmail.com.
