AlinsRan commented on issue #12449:
URL: https://github.com/apache/apisix/issues/12449#issuecomment-6619050421
@wangchao732 Based on your screenshots, this is not an APISIX counting bug.
The scrape target `172.16.1.180:30215` is a NodePort that load-balances across
APISIX pods, so scrapes can hit different pods and their counters get mixed
into the same series. When the value drops after a switch, `rate()` treats it
as a counter reset and counts the whole counter value (~73M) as new requests,
which produces the multi-million RPS spikes. You can confirm this with
`apisix_node_info{instance="172.16.1.180:30215"}`: its `hostname` label
switches between pods at the spike times.
To fix it, scrape each APISIX pod individually instead of going through a
Service/NodePort:
- With Prometheus Operator, enable `apisix.prometheus.enabled` and
`metrics.serviceMonitor.enabled` in the Helm chart.
- With plain Prometheus inside the cluster, use pod discovery:
```yaml
- job_name: apisix
metrics_path: /apisix/prometheus/metrics
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
regex: apisix
action: keep
- source_labels: [__meta_kubernetes_pod_container_port_name]
regex: prometheus # declared when apisix.prometheus.enabled is true
action: keep
```
- If Prometheus runs outside the cluster, run an in-cluster scraper (e.g.
Prometheus in agent mode) and `remote_write` to your Prometheus.
Then remove the NodePort scrape job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
