wangchao732 commented on issue #12449: URL: https://github.com/apache/apisix/issues/12449#issuecomment-6731476741
> [@wangchao732](https://github.com/wangchao732) Based on your screenshots, this is not an APISIX counting bug. The scrape target `172.16.1.180:30215` is a NodePort that load-balances across APISIX pods, so scrapes can hit different pods and their counters get mixed into the same series. When the value drops after a switch, `rate()` treats it as a counter reset and counts the whole counter value (~73M) as new requests, which produces the multi-million RPS spikes. You can confirm this with `apisix_node_info{instance="172.16.1.180:30215"}`: its `hostname` label switches between pods at the spike times. > > To fix it, scrape each APISIX pod individually instead of going through a Service/NodePort: > > * With Prometheus Operator, enable `apisix.prometheus.enabled` and `metrics.serviceMonitor.enabled` in the Helm chart. > * With plain Prometheus inside the cluster, use pod discovery: > - job_name: apisix > metrics_path: /apisix/prometheus/metrics > kubernetes_sd_configs: > - role: pod > relabel_configs: > - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] > regex: apisix > action: keep > - source_labels: [__meta_kubernetes_pod_container_port_name] > regex: prometheus # declared when apisix.prometheus.enabled is true > action: keep > * If Prometheus runs outside the cluster, run an in-cluster scraper (e.g. Prometheus in agent mode) and `remote_write` to your Prometheus. > > Then remove the NodePort scrape job. Very professional reply! I know what to do now. Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
