nic-6443 opened a new pull request, #14092: URL: https://github.com/apache/apisix/pull/14092
`t/stream-plugin/prometheus-metrics.t` has been failing intermittently in the `t/stream-plugin t/plugin/[l-z]*.t` shard (TEST 3, 12, 18, 22, 25, and occasionally 15). Every failure is the same thing: the scrape right after a block that ran a stream session doesn't contain that session's `apisix_stream_status` series yet. This is a test assumption problem, not a plugin bug. A finished session reaches `/apisix/prometheus/metrics` in two hops. The stream worker keeps its counter increments locally and syncs them into the `prometheus-metrics` dict every second (or once more when it exits), and the privileged agent rebuilds the cached exposition every `refresh_interval`. These assertion blocks scrape in the block right after the session, so in between there's a reload. On reload the new privileged agent fills the cache from `init_exporter_timer()` as soon as it starts, which races with the old stream worker's exit-time flush, and the next refill only comes `refresh_interval` (0.5s here) later. Test::Nginx sends the scrape a fraction of a second after the HUP, so when the new agent wins that race the scrape gets a cache without the session. Bandwidth and active connections are read from the stream zone during the refill itself, which is why those series are always there and only the counters go missing. So the blocks that scrape right after a session block (TEST 3, 12, 15, 18, 22, 25) now poll the endpoint until the expected series shows up, with a 5s deadline, and on timeout print the last exposition so the failure still shows what was there. The other scrape blocks follow a block that already waited (TEST 4 sleeps after closing its session, and TEST 19/26 come after a polled block), so they're left alone. `prometheus-metrics-service.t` TEST 10 has the same gap in a narrower form: it closes the session and scrapes once after 1.2s, but the worst case is up to 1s for the worker sync plus up to 0.5s for the refill. It now rescrapes until the status series shows up. To reproduce it deterministically, I added a delay to the worker counter sync in `prometheus_resty_counter.lua` (0.3s on the exit flush, 0.8s on the periodic one). With that delay, all 5 runs of the current tests failed: TEST 3, 12, 15, 22 and 25 in `prometheus-metrics.t` and TEST 10 in `prometheus-metrics-service.t` failed every time, and TEST 18 failed in 4 of the 5. With this change, all 5 runs pass with the same delay. Without any injected delay, 3 of 15 runs of the two files failed before this change, and 0 of 15 failed after. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
