I am trying to target and scrape node-exporter and Kube-state-metric pods running in the EKS cluster using Prometheus. For service discovery, I am using kubernetes_sd_configs but Prometheus is not able to target the required pods. Node-exporter and Prometheus are running in the same namespace while Kube-state-metrics is running in another namespace. I am using pod annotations to discover the pods.
Following is the snippet of node-exporter.yml, prometheus.yml Node-exporter.yaml: apiVersion: apps/v1 kind: DaemonSet metadata: labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter name: node-exporter namespace: amazon-cloudwatch spec: selector: matchLabels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter template: metadata: annotations: prometheus.io/scrape: true prometheus.io/path: /metrics prometheus.io/port: "9100" prometheus.io/scheme: "http" labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter The same goes for the Kube-state-metrics.yml file only with minor changes but not in the annotations part. As for Prometheus, my code snippet of Prometheus scrape configs is follow: - job_name: "kubernetes-pods" kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: (.+):(?:\d+);(\d+) replacement: ${1}:${2} target_label: __address__ - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (.+) Can anyone please point out any error I am doing because the above snippets are not working. Also suggestions for using some other configurations are also appreciated. Thanks Ansh Lehri -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ecaec40-ad65-46d0-8995-e80288aba5f1n%40googlegroups.com.