[prometheus-users] kubernetes_sd_configs not targeting required pods

[Ansh Lehri]

I am trying to target and scrape node-exporter and Kube-state-metric pods 
running in the EKS cluster using Prometheus. For service discovery, I am 
using kubernetes_sd_configs but Prometheus is not able to target the 
required pods.
Node-exporter and Prometheus are running in the same namespace while 
Kube-state-metrics is running in another namespace.
I am using pod annotations to discover the pods.

Following is the snippet of node-exporter.yml, prometheus.yml

Node-exporter.yaml:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: node-exporter
  name: node-exporter
  namespace: amazon-cloudwatch
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      annotations:
        prometheus.io/scrape: true
        prometheus.io/path: /metrics
        prometheus.io/port: "9100"
        prometheus.io/scheme: "http"
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: node-exporter

The same goes for the Kube-state-metrics.yml file only with minor changes 
but not in the annotations part. As for Prometheus, my code snippet of 
Prometheus scrape configs is follow:

- job_name: "kubernetes-pods"
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
        - source_labels: 
[__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: 
[__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, 
__meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: ${1}:${2}
          target_label: __address__
        - source_labels: 
[__meta_kubernetes_pod_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (.+)

Can anyone please point out any error I am doing because the above snippets 
are not working.  Also suggestions for using some other configurations are 
also appreciated.

Thanks 
Ansh Lehri




-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1ecaec40-ad65-46d0-8995-e80288aba5f1n%40googlegroups.com.