yujinchoi-94 opened a new issue, #488: URL: https://github.com/apache/apisix-helm-chart/issues/488
Hi, I'm trying to collect some metrics using apisix prometheus plugin. I've found this [PR](https://github.com/apache/apisix-ingress-controller/pull/1683) and followed the document. (I manually configured prometheus port based on [this](https://github.com/apache/apisix-helm-chart/pull/478/files)) However, prometheus is not collecting any metrics from apisix service monitor. I found that `kubernetes_sd_config` has configured for apisix through Prometheus UI. #### Prometheus configuration (through Prometheus UI) ``` scrape_configs: - job_name: serviceMonitor/ingress-apisix/dev-apisix/0 honor_timestamps: true scrape_interval: 15s scrape_timeout: 5s metrics_path: /apisix/prometheus/metrics scheme: http follow_redirects: true enable_http2: true relabel_configs: - source_labels: [job] separator: ; regex: (.*) target_label: __tmp_prometheus_job_name replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_instance, __meta_kubernetes_service_labelpresent_app_kubernetes_io_instance] separator: ; regex: (dev-apisix);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_managed_by, __meta_kubernetes_service_labelpresent_app_kubernetes_io_managed_by] separator: ; regex: (Helm);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name, __meta_kubernetes_service_labelpresent_app_kubernetes_io_name] separator: ; regex: (apisix);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_service, __meta_kubernetes_service_labelpresent_app_kubernetes_io_service] separator: ; regex: (apisix-gateway);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_version, __meta_kubernetes_service_labelpresent_app_kubernetes_io_version] separator: ; regex: (3.1.0);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_service_label_helm_sh_chart, __meta_kubernetes_service_labelpresent_helm_sh_chart] separator: ; regex: (apisix-1.1.0);true replacement: $1 action: keep - source_labels: [__meta_kubernetes_pod_container_port_name] separator: ; regex: prometheus replacement: $1 action: keep - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Node;(.*) target_label: node replacement: ${1} action: replace - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Pod;(.*) target_label: pod replacement: ${1} action: replace - source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: service replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: pod replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_container_name] separator: ; regex: (.*) target_label: container replacement: $1 action: replace - source_labels: [__meta_kubernetes_pod_phase] separator: ; regex: (Failed|Succeeded) replacement: $1 action: drop - source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: job replacement: ${1} action: replace - separator: ; regex: (.*) target_label: endpoint replacement: prometheus action: replace - source_labels: [__address__] separator: ; regex: (.*) modulus: 3 target_label: __tmp_hash replacement: $1 action: hashmod - source_labels: [__tmp_hash] separator: ; regex: "2" replacement: $1 action: keep kubernetes_sd_configs: - role: endpoints kubeconfig_file: "" follow_redirects: true enable_http2: true namespaces: own_namespace: false names: - ingress-apisix ``` By the way, there was no target for apisix. Moreover, in service discovery page, there were some discovered metrics but it was dropped. <img width="1017" alt="스크린샷 2023-02-28 오후 6 17 55" src="https://user-images.githubusercontent.com/30385786/221808212-b3b0d4b8-e037-4c29-82d7-7afaae6cc06f.png";>  FYI, Prometheus has deployed in another namespace so I set this option `serviceMonitorSelectorNilUsesHelmValues ` to false. Here's what I've configured. # Gateway Service ```json kubectl get service dev-apisix-gateway -o json { "apiVersion": "v1", "kind": "Service", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"dev-apisix\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"apisix\",\"app.kubernetes.io/service\":\"apisix-gateway\",\"app.kubernetes.io/version\":\"3.1.0\",\"argocd.argoproj.io/instance\":\"dev-apisix\",\"helm.sh/chart\":\"apisix-1.1.0\"},\"name\":\"dev-apisix-gateway\",\"namespace\":\"ingress-apisix\"},\"spec\":{\"externalTrafficPolicy\":\"Cluster\",\"ports\":[{\"name\":\"apisix-gateway\",\"port\":80,\"protocol\":\"TCP\",\"targetPort\":9080}],\"selector\":{\"app.kubernetes.io/instance\":\"dev-apisix\",\"app.kubernetes.io/name\":\"apisix\"},\"type\":\"NodePort\"}}\n" }, "creationTimestamp": "2023-01-30T08:01:53Z", "labels": { "app.kubernetes.io/instance": "dev-apisix", "app.kubernetes.io/managed-by": "Helm", "app.kubernetes.io/name": "apisix", "app.kubernetes.io/service": "apisix-gateway", "app.kubernetes.io/version": "3.1.0", "argocd.argoproj.io/instance": "dev-apisix", "helm.sh/chart": "apisix-1.1.0" }, "name": "dev-apisix-gateway", "namespace": "ingress-apisix", "resourceVersion": "520319969", "uid": "c30b9ffb-feab-46a9-a174-96e7e19c91ae" }, "spec": { "clusterIP": "172.20.54.80", "clusterIPs": [ "172.20.54.80" ], "externalTrafficPolicy": "Cluster", "ipFamilies": [ "IPv4" ], "ipFamilyPolicy": "SingleStack", "ports": [ { "name": "apisix-gateway", "nodePort": 31768, "port": 80, "protocol": "TCP", "targetPort": 9080 }, { "name": "prometheus", "nodePort": 31040, "port": 9091, "protocol": "TCP", "targetPort": 9091 } ], "selector": { "app.kubernetes.io/instance": "dev-apisix", "app.kubernetes.io/name": "apisix" }, "sessionAffinity": "None", "type": "NodePort" }, "status": { "loadBalancer": {} } } ``` # ServiceMonitor ``` kubectl get servicemonitor dev-apisix -o json { "apiVersion": "monitoring.coreos.com/v1", "kind": "ServiceMonitor", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"ServiceMonitor\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"dev-apisix\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"apisix\",\"app.kubernetes.io/version\":\"3.1.0\",\"argocd.argoproj.io/instance\":\"dev-apisix\",\"helm.sh/chart\":\"apisix-1.1.0\",\"release\":\"kube-prometheus-stack\"},\"name\":\"dev-apisix\",\"namespace\":\"ingress-apisix\"},\"spec\":{\"endpoints\":[{\"interval\":\"15s\",\"path\":\"/apisix/prometheus/metrics\",\"scheme\":\"http\",\"targetPort\":\"prometheus\"}],\"namespaceSelector\":{\"matchNames\":[\"ingress-apisix\"]},\"selector\":{\"matchLabels\":{\"app.kubernetes.io/instance\":\"dev-apisix\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"apisix\",\"app.kubernetes.io/service\":\"apisix-gateway\",\"app.kubernetes.io/version\":\"3.1.0\",\"helm.sh/chart\":\"ap isix-1.1.0\"}}}}\n" }, "creationTimestamp": "2023-01-30T08:01:53Z", "generation": 6, "labels": { "app.kubernetes.io/instance": "dev-apisix", "app.kubernetes.io/managed-by": "Helm", "app.kubernetes.io/name": "apisix", "app.kubernetes.io/version": "3.1.0", "argocd.argoproj.io/instance": "dev-apisix", "helm.sh/chart": "apisix-1.1.0", "release": "kube-prometheus-stack" }, "name": "dev-apisix", "namespace": "ingress-apisix", "resourceVersion": "525298855", "uid": "3199f803-fc45-4450-88bc-300068e05cce" }, "spec": { "endpoints": [ { "interval": "15s", "path": "/apisix/prometheus/metrics", "scheme": "http", "targetPort": "prometheus" } ], "namespaceSelector": { "matchNames": [ "ingress-apisix" ] }, "selector": { "matchLabels": { "app.kubernetes.io/instance": "dev-apisix", "app.kubernetes.io/managed-by": "Helm", "app.kubernetes.io/name": "apisix", "app.kubernetes.io/service": "apisix-gateway", "app.kubernetes.io/version": "3.1.0", "helm.sh/chart": "apisix-1.1.0" } } } } ``` # Prometheus ```json kubectl get prometheus kube-prometheus-stack-prometheus -o json { "apiVersion": "monitoring.coreos.com/v1", "kind": "Prometheus", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"monitoring.coreos.com/v1\",\"kind\":\"Prometheus\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"kube-prometheus-stack-prometheus\",\"app.kubernetes.io/instance\":\"dev-kube-prometheus-stack\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/part-of\":\"kube-prometheus-stack\",\"app.kubernetes.io/version\":\"44.2.1\",\"argocd.argoproj.io/instance\":\"dev-kube-prometheus-stack\",\"chart\":\"kube-prometheus-stack-44.2.1\",\"heritage\":\"Helm\",\"release\":\"dev-kube-prometheus-stack\"},\"name\":\"kube-prometheus-stack-prometheus\",\"namespace\":\"monitor\"},\"spec\":{\"additionalScrapeConfigs\":{\"key\":\"additional-scrape-configs.yaml\",\"name\":\"kube-prometheus-stack-prometheus-scrape-confg\"},\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"spec\",\"operator\":\"In\",\"values\":[\"4xla rge\"]}]}]}}},\"alerting\":{\"alertmanagers\":[{\"apiVersion\":\"v2\",\"name\":\"kube-prometheus-stack-alertmanager\",\"namespace\":\"monitor\",\"pathPrefix\":\"/\",\"port\":\"http-web\"}]},\"configMaps\":[\"bucket\"],\"enableAdminAPI\":false,\"externalUrl\":\"http://kube-prometheus-stack-prometheus.monitor:9090\",\"hostNetwork\":false,\"image\":\"quay.io/prometheus/prometheus:v2.41.0\",\"listenLocal\":false,\"logFormat\":\"logfmt\",\"logLevel\":\"info\",\"paused\":false,\"podMonitorNamespaceSelector\":{},\"podMonitorSelector\":{\"matchLabels\":{\"release\":\"dev-kube-prometheus-stack\"}},\"portName\":\"http-web\",\"probeNamespaceSelector\":{},\"probeSelector\":{\"matchLabels\":{\"release\":\"dev-kube-prometheus-stack\"}},\"replicas\":1,\"resources\":{\"limits\":{\"cpu\":8,\"memory\":\"8G\"},\"requests\":{\"cpu\":8,\"memory\":\"8G\"}},\"retention\":\"10d\",\"routePrefix\":\"/\",\"ruleNamespaceSelector\":{},\"ruleSelector\":{\"matchLabels\":{\"release\":\"dev-kube-prometheus-stack\"} },\"scrapeInterval\":\"1m\",\"scrapeTimeout\":\"5s\",\"securityContext\":{\"fsGroup\":2000,\"runAsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":1000},\"serviceAccountName\":\"kube-prometheus-stack-prometheus\",\"serviceMonitorNamespaceSelector\":{},\"serviceMonitorSelector\":{\"matchExpressions\":[{\"key\":\"release\",\"operator\":\"In\",\"values\":[\"kube-prometheus-stack\",\"dev-kube-prometheus-stack\"]}]},\"shards\":3,\"storage\":{\"volumeClaimTemplate\":{\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"50Gi\"}},\"storageClassName\":\"efs\"}}},\"thanos\":{\"baseImage\":\"quay.io/thanos/thanos\",\"objectStorageConfigFile\":\"/etc/prometheus/configmaps/bucket/bucket.yaml\",\"version\":\"v0.30.1\",\"volumeMounts\":[{\"mountPath\":\"/etc/prometheus/configmaps/bucket\",\"name\":\"configmap-bucket\"}]},\"version\":\"v2.41.0\",\"walCompression\":true}}\n" }, "creationTimestamp": "2022-04-14T05:37:13Z", "generation": 24, "labels": { "app": "kube-prometheus-stack-prometheus", "app.kubernetes.io/instance": "dev-kube-prometheus-stack", "app.kubernetes.io/managed-by": "Helm", "app.kubernetes.io/part-of": "kube-prometheus-stack", "app.kubernetes.io/version": "44.2.1", "argocd.argoproj.io/instance": "dev-kube-prometheus-stack", "chart": "kube-prometheus-stack-44.2.1", "heritage": "Helm", "release": "dev-kube-prometheus-stack" }, "name": "kube-prometheus-stack-prometheus", "namespace": "monitor", "resourceVersion": "525328723", "uid": "a89b443b-7c98-4594-8ffe-66d963cfbb29" }, "spec": { "additionalScrapeConfigs": { "key": "additional-scrape-configs.yaml", "name": "kube-prometheus-stack-prometheus-scrape-confg" }, "affinity": { "nodeAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": { "nodeSelectorTerms": [ { "matchExpressions": [ { "key": "spec", "operator": "In", "values": [ "4xlarge" ] } ] } ] } } }, "alerting": { "alertmanagers": [ { "apiVersion": "v2", "name": "kube-prometheus-stack-alertmanager", "namespace": "monitor", "pathPrefix": "/", "port": "http-web" } ] }, "configMaps": [ "bucket" ], "enableAdminAPI": false, "evaluationInterval": "30s", "externalUrl": "http://kube-prometheus-stack-prometheus.monitor:9090";, "hostNetwork": false, "image": "quay.io/prometheus/prometheus:v2.41.0", "listenLocal": false, "logFormat": "logfmt", "logLevel": "debug", "paused": false, "podMonitorNamespaceSelector": {}, "podMonitorSelector": { "matchLabels": { "release": "dev-kube-prometheus-stack" } }, "portName": "http-web", "probeNamespaceSelector": {}, "probeSelector": { "matchLabels": { "release": "dev-kube-prometheus-stack" } }, "replicas": 1, "resources": { "limits": { "cpu": 8, "memory": "8G" }, "requests": { "cpu": 8, "memory": "8G" } }, "retention": "10d", "routePrefix": "/", "ruleNamespaceSelector": {}, "ruleSelector": { "matchLabels": { "release": "dev-kube-prometheus-stack" } }, "scrapeInterval": "1m", "scrapeTimeout": "5s", "securityContext": { "fsGroup": 2000, "runAsGroup": 2000, "runAsNonRoot": true, "runAsUser": 1000 }, "serviceAccountName": "kube-prometheus-stack-prometheus", "serviceMonitorNamespaceSelector": {}, "serviceMonitorSelector": { "matchExpressions": [ { "key": "release", "operator": "In", "values": [ "kube-prometheus-stack", "dev-kube-prometheus-stack" ] } ] }, "shards": 3, "storage": { "volumeClaimTemplate": { "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "50Gi" } }, "storageClassName": "efs" } } }, "thanos": { "baseImage": "quay.io/thanos/thanos", "objectStorageConfigFile": "/etc/prometheus/configmaps/bucket/bucket.yaml", "version": "v0.30.1", "volumeMounts": [ { "mountPath": "/etc/prometheus/configmaps/bucket", "name": "configmap-bucket" } ] }, "version": "v2.41.0", "walCompression": true }, "status": { "availableReplicas": 3, "conditions": [ { "lastTransitionTime": "2023-02-28T08:09:33Z", "observedGeneration": 24, "status": "True", "type": "Available" }, { "lastTransitionTime": "2023-01-31T08:00:15Z", "observedGeneration": 24, "status": "True", "type": "Reconciled" } ], "paused": false, "replicas": 3, "shardStatuses": [ { "availableReplicas": 1, "replicas": 1, "shardID": "0", "unavailableReplicas": 0, "updatedReplicas": 1 }, { "availableReplicas": 1, "replicas": 1, "shardID": "1", "unavailableReplicas": 0, "updatedReplicas": 1 }, { "availableReplicas": 1, "replicas": 1, "shardID": "2", "unavailableReplicas": 0, "updatedReplicas": 1 } ], "unavailableReplicas": 0, "updatedReplicas": 3 } } ``` Thank you in advance :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/30385786/221808212-b3b0d4b8-e037-4c29-82d7-7afaae6cc06f.png"](https://user-images.githubusercontent.com/30385786/221808212-b3b0d4b8-e037-4c29-82d7-7afaae6cc06f.png"){kind=link}
