Hello Geler,
I have same issue with v2.19.3 of Prometheus, do you have already fix this
issue?
On Tuesday, July 7, 2020 at 2:05:37 AM UTC+8, dinor geler wrote:
>
> Hello all , i was redirected from github to here , appreciate the help on
> the subject below :
>
> What did you do? upgraded my cluster to 1.16.8
>
> What did you expect to see? to see graphs of the rules i have.
>
> What did you see instead? Under which circumstances? am not able to see
> some metrics like :
> sum by(pod_name)
> (rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name!=""}[5m]))
>
> * 1000
> i get no datapoints found ,with 1.15.11 cluster ,it is working.
>
> aks on azure 1.16.8
>
> -
>
> System information:
>
> insert output of uname -srm here
> -
>
> Prometheus version:
>
> insert output of prom/prometheus:v2.19.2prom/prometheus:v2.19.2 here
> -
>
> Alertmanager version:
>
> insert output of alertmanager --version here (if relevant to the issue)
> -
>
> Prometheus configuration file:
>
> # Please edit the object below. Lines beginning with a '#' will be ignored,
> # and an empty file will abort the edit. If an error occurs while saving this
> file will be
> # reopened with the relevant failures.
> #
> apiVersion: v1
> data:
> alerting_rules.yml: |
> {}
> alerts: |
> {}
> prometheus.yml: |
> global:
> evaluation_interval: 1m
> scrape_interval: 1m
> scrape_timeout: 10s
> remote_write:
> - url: http://prom-mdm-converter/receive
>
> rule_files:
> - /etc/config/recording_rules.yml
> - /etc/config/alerting_rules.yml
> - /etc/config/rules
> - /etc/config/alerts
> scrape_configs:
> - job_name: prometheus
> static_configs:
> - targets:
> - localhost:9090
> - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
> job_name: kubernetes-apiservers
> kubernetes_sd_configs:
> - role: endpoints
> relabel_configs:
> - action: keep
> regex: default;kubernetes;https
> source_labels:
> - __meta_kubernetes_namespace
> - __meta_kubernetes_service_name
> - __meta_kubernetes_endpoint_port_name
> scheme: https
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> insecure_skip_verify: true
> - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
> job_name: kubernetes-nodes
> kubernetes_sd_configs:
> - role: node
> relabel_configs:
> - action: labelmap
> regex: __meta_kubernetes_node_label_(.+)
> - replacement: kubernetes.default.svc:443
> target_label: __address__
> - regex: (.+)
> replacement: /api/v1/nodes/$1/proxy/metrics
> source_labels:
> - __meta_kubernetes_node_name
> target_label: __metrics_path__
> scheme: https
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> insecure_skip_verify: true
> - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
> job_name: kubernetes-nodes-cadvisor
> kubernetes_sd_configs:
> - role: node
> relabel_configs:
> - action: labelmap
> regex: __meta_kubernetes_node_label_(.+)
> - replacement: kubernetes.default.svc:443
> target_label: __address__
> - regex: (.+)
> replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
> source_labels:
> - __meta_kubernetes_node_name
> target_label: __metrics_path__
> scheme: https
> tls_config:
> ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
> insecure_skip_verify: true
> - job_name: kubernetes-service-endpoints
> kubernetes_sd_configs:
> - role: endpoints
> relabel_configs:
> - action: keep
> regex: true
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_scrape
> - action: replace
> regex: (https?)
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_scheme
> target_label: __scheme__
> - action: replace
> regex: (.+)
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_path
> target_label: __metrics_path__
> - action: replace
> regex: ([^:]+)(?::\d+)?;(\d+)
> replacement: $1:$2
> source_labels:
> - __address__
> - __meta_kubernetes_service_annotation_prometheus_io_port
> target_label: __address__
> - action: labelmap
> regex: __meta_kubernetes_service_label_(.+)
> - action: replace
> source_labels:
> - __meta_kubernetes_namespace
> target_label: kubernetes_namespace
> - action: replace
> source_labels:
> - __meta_kubernetes_service_name
> target_label: kubernetes_name
> - action: replace
> source_labels:
> - __meta_kubernetes_pod_node_name
> target_label: kubernetes_node
> - job_name: kubernetes-service-endpoints-slow
> kubernetes_sd_configs:
> - role: endpoints
> relabel_configs:
> - action: keep
> regex: true
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
> - action: replace
> regex: (https?)
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_scheme
> target_label: __scheme__
> - action: replace
> regex: (.+)
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_path
> target_label: __metrics_path__
> - action: replace
> regex: ([^:]+)(?::\d+)?;(\d+)
> replacement: $1:$2
> source_labels:
> - __address__
> - __meta_kubernetes_service_annotation_prometheus_io_port
> target_label: __address__
> - action: labelmap
> regex: __meta_kubernetes_service_label_(.+)
> - action: replace
> source_labels:
> - __meta_kubernetes_namespace
> target_label: kubernetes_namespace
> - action: replace
> source_labels:
> - __meta_kubernetes_service_name
> target_label: kubernetes_name
> - action: replace
> source_labels:
> - __meta_kubernetes_pod_node_name
> target_label: kubernetes_node
> scrape_interval: 5m
> scrape_timeout: 30s
> - honor_labels: true
> job_name: prometheus-pushgateway
> kubernetes_sd_configs:
> - role: service
> relabel_configs:
> - action: keep
> regex: pushgateway
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_probe
> - job_name: kubernetes-services
> kubernetes_sd_configs:
> - role: service
> metrics_path: /probe
> params:
> module:
> - http_2xx
> relabel_configs:
> - action: keep
> regex: true
> source_labels:
> - __meta_kubernetes_service_annotation_prometheus_io_probe
> - source_labels:
> - __address__
> target_label: __param_target
> - replacement: blackbox
> target_label: __address__
> - source_labels:
> - __param_target
> target_label: instance
> - action: labelmap
> regex: __meta_kubernetes_service_label_(.+)
> - source_labels:
> - __meta_kubernetes_namespace
> target_label: kubernetes_namespace
> - source_labels:
> - __meta_kubernetes_service_name
> target_label: kubernetes_name
> - job_name: kubernetes-pods
> kubernetes_sd_configs:
> - role: pod
> relabel_configs:
> - action: keep
> regex: true
> source_labels:
> - __meta_kubernetes_pod_annotation_prometheus_io_scrape
> - action: replace
> regex: (.+)
> source_labels:
> - __meta_kubernetes_pod_annotation_prometheus_io_path
> target_label: __metrics_path__
> - action: replace
> regex: ([^:]+)(?::\d+)?;(\d+)
> replacement: $1:$2
> source_labels:
> - __address__
> - __meta_kubernetes_pod_annotation_prometheus_io_port
> target_label: __address__
> - action: labelmap
> regex: __meta_kubernetes_pod_label_(.+)
> - action: replace
> source_labels:
> - __meta_kubernetes_namespace
> target_label: kubernetes_namespace
> - action: replace
> source_labels:
> - __meta_kubernetes_pod_name
> target_label: kubernetes_pod_name
> - job_name: kubernetes-pods-slow
> kubernetes_sd_configs:
> - role: pod
> relabel_configs:
> - action: keep
> regex: true
> source_labels:
> - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
> - action: replace
> regex: (.+)
> source_labels:
> - __meta_kubernetes_pod_annotation_prometheus_io_path
> target_label: __metrics_path__
> - action: replace
> regex: ([^:]+)(?::\d+)?;(\d+)
> replacement: $1:$2
> source_labels:
> - __address__
> - __meta_kubernetes_pod_annotation_prometheus_io_port
> target_label: __address__
> - action: labelmap
> regex: __meta_kubernetes_pod_label_(.+)
> - action: replace
> source_labels:
> - __meta_kubernetes_namespace
> target_label: kubernetes_namespace
> - action: replace
> source_labels:
> - __meta_kubernetes_pod_name
> target_label: kubernetes_pod_name
> scrape_interval: 5m
> scrape_timeout: 30s
> recording_rules.yml: |
> {}
> rules: |
> groups:
> - interval: 1m
> name: custom_counter
> rules:
> - expr: 100 - (avg by (nodename)
> (irate(node_cpu_seconds_total{mode="idle"}[5m])
> * on(instance) group_left(nodename) node_uname_info * 100))
> record: node_cpu_usage
> - expr: sum by (device, nodename)
> (irate(node_disk_read_bytes_total[5m]) * 60 *
> on(instance) group_left(nodename) node_uname_info)
> record: node_disk_read_bytes_total_rate
> - expr:
> sum(rate(container_cpu_usage_seconds_total{container_name!='POD',pod_name!=''}[5m]))
> by (pod_name) * 1000
> record: container_cpu_usage
> - expr:
> sum(container_memory_usage_bytes{container_name!='POD',container_name!=''})
> by (pod_name)
> record: pod_memory_usage
> - expr: kube_node_status_condition{condition='Ready',status='true'}
> record: node_status
> - expr: sum by (node)
> (kube_pod_container_resource_requests_memory_bytes) / 1024
> / 1024
> record: node_memory_requests
> - expr: increase(kube_pod_container_status_restarts_total[1h])
> record: container_restarts
> kind: ConfigMap
> metadata:
> creationTimestamp: "2020-07-05T21:37:16Z"
> labels:
> app: prometheus
> chart: prometheus-10.4.0
> component: server
> heritage: Tiller
> release: prometheus
> name: prometheus-server
> namespace: monitor
> resourceVersion: "82618"
> selfLink: /api/v1/namespaces/monitor/configmaps/prometheus-server
> uid: 30b94a08-f056-4ab0-a2a3-9ac2e81d9afd
>
>
>
> - Alertmanager configuration file:
>
> insert configuration here (if relevant to the issue)
>
>
> - Logs:
>
> insert Prometheus and Alertmanager logs relevant to the issue here
>
> Did something changed in promql language ,do i have to do any modification
> to get it working and showing the graphs :
>
> [image: image]
> <https://user-images.githubusercontent.com/4949529/86543797-74f58300-bf2a-11ea-8ff5-5d5c89781cf6.png>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8da2b499-b567-4271-949a-ac3fe50f7ba7o%40googlegroups.com.
