hanzhenfang commented on issue #12449: URL: https://github.com/apache/apisix/issues/12449#issuecomment-6598807935
I ran a local experiment based on the single scrape target in the provided configuration. It produced approximately **663 RPS with zero new business requests**, while a direct-scrape control stayed at **0 RPS**. Could the maintainers help review the setup and confirm whether the interpretation below is correct? I followed the reported APISIX 3.11/Helm setup, 30-second scrape interval, and fixed `[5m]` query from the [configuration shared above](https://github.com/apache/apisix/issues/12449#issuecomment-4601225171). For this experiment, **I introduced two APISIX pods behind one NodePort and deliberately recreated the scrape connections** to test whether different per-pod counters could enter the same time series. Those additional conditions have not been established for the reported deployment: `xxx:30215` alone does not prove a NodePort topology, and the posted `replicas: 2` belongs to the upstream application. I am sharing this as a candidate mechanism to verify, not as confirmation of the original issue's root cause. #### Test environment - APISIX: `apache/apisix:3.11.0-debian`, Helm chart `2.10.0`, two pods with one worker per pod. - Kubernetes: OrbStack `v1.34.8+orb1`, arm64. - Prometheus: `prom/prometheus:v2.48.1`, 30-second scrape interval, `[5m]` rate window. - etcd: `v3.5.11`. Upstream: Nginx `1.28.1-alpine`, returning HTTP 200 through one shared Kubernetes Service IP. #### Steps used in the local experiment **1. Configured two APISIX pods to serve the same route.** I used the global rule posted above (`proxy-rewrite`, `prometheus` with `prefer_name: false`, and `cors`) and the second route's POST configuration. For the test, the route ID was `repro-route`, the host was `repro.local`, and the service ID was `420094949904614258`. Both pods forwarded `POST /info/query` to the same upstream Service IP. The upstream returned HTTP 200. The Java external plugin from the first route and the real Dapr application were not part of this test. I enabled the exporter on `0.0.0.0:9091`, at `/apisix/prometheus/metrics`, and exposed it through one NodePort Service selecting **both APISIX pods**, with `sessionAffinity: None` and port `31249`. **2. Generated different historical counter values, then stopped all business requests.** I sent requests directly to each pod's gateway port (`9080`), using: ```http POST /info/query Host: repro.local Content-Type: application/json {} ``` In this run, Pod A received 60,000 seed requests and Pod B received 59,000, plus one initial verification request per pod. All responses were HTTP 200. The resulting counters were: ```text Pod A: apisix_http_status{code="200",route="repro-route",matched_uri="/info/query",matched_host="repro.local",service="420094949904614258",consumer="",node="<shared-upstream-IP>"} 60001 Pod B: apisix_http_status{code="200",route="repro-route",matched_uri="/info/query",matched_host="repro.local",service="420094949904614258",consumer="",node="<shared-upstream-IP>"} 59001 ``` I checked that the complete exporter label sets were identical and that each pod's counter stayed constant after traffic stopped. In this setup, `node` identified the shared upstream address, so it did not distinguish the APISIX pods. **3. Started Prometheus with a direct-scrape control and a NodePort scrape.** The configuration was as follows, with the actual pod and node IPs substituted for the placeholders. Prometheus could reach all three targets. ```yaml global: scrape_interval: 30s scrape_configs: - job_name: apisix-direct metrics_path: /apisix/prometheus/metrics static_configs: - targets: - "<pod-A-IP>:9091" - "<pod-B-IP>:9091" - job_name: apisix-nodeport metrics_path: /apisix/prometheus/metrics static_configs: - targets: - "<node-IP>:31249" ``` The direct job has a separate `instance` for each pod. The NodePort job has only one `instance`, even when its connection is routed to a different pod. **4. Confirmed a stable zero-RPS baseline.** With business traffic stopped, I waited for all three targets to be `up` and checked the following query: ```promql sum by (service) ( rate(apisix_http_status{ job="apisix-nodeport", service="420094949904614258", route="repro-route" }[5m]) ) ``` I used the same query with `job="apisix-direct"` for the control and recorded the raw `apisix_http_status` samples for both jobs. Both queries returned 0 before the connection changes. The job filter kept the two sets of scrapes separate. **5. Recreated the scrape connections while keeping business traffic stopped.** To trigger backend selection without restarting APISIX or Prometheus, I alternated `enable_http2: false` and `enable_http2: true` in **both** scrape jobs and reloaded Prometheus after each change. With the configuration mounted from a ConfigMap, I first waited until the updated file was visible inside the container, then sent `SIGHUP` to the Prometheus process: ```bash kubectl -n issue12449-lab exec prometheus -c prometheus -- sh -c 'kill -HUP 1' ``` In this deployment, Prometheus was PID 1. The exporters continued serving plaintext HTTP/1.1; changing this setting was a way to recreate the scrape clients, not a test of HTTP/2 traffic. Target addresses, labels, and scrape intervals remained unchanged, and both jobs went through the same reloads. After each reload, I waited for a new scrape sample and repeated the process to observe several backend switches, including downward counter transitions. A new connection could select the same pod again, so not every reload produced a switch. The NodePort raw sequence changed between `60001` and `59001`, while the two direct sequences remained constant. I also matched `apisix_node_info`'s `hostname` label to the counter at the same scrape timestamp to verify which pod answered each NodePort scrape. **6. Stopped reloading and observed recovery.** I kept business traffic stopped and collected another 11 scrape samples (approximately 5.5 minutes). I also checked both pods' access logs and restart counts over the observation period. The NodePort RPS returned to 0; the measured recovery time is included below. #### Observed result | Measurement | Result in this run | |---|---| | New business requests during observation | **0**, verified from both pods' access logs | | Direct pod counters | Constant at **60,001 / 59,001** | | Direct-scrape RPS | **0 throughout** | | NodePort counter | **7 value transitions, including 4 decreases** | | NodePort RPS peak | **662.99 RPS** | | Recovery | First observed 0 RPS **270 seconds after the final decrease** | | APISIX restarts during observation | **No increase in restart counts** | #### Interpretation for confirmation My interpretation is that the shared target kept the same `instance` and business labels while switching between independent per-pod counters. The resulting downward transitions were interpreted by `rate()` as counter resets, producing an apparent increase despite no new business requests. Keeping each pod as a separate scrape target avoided that discontinuity in the control. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
