nic-6443 opened a new pull request, #14034:
URL: https://github.com/apache/apisix/pull/14034
### Description
This adds Prometheus gauges for a few OpenResty internals that can run out
per process, so exhaustion becomes visible before it turns into errors:
- `apisix_nginx_timers{process, subsystem, state}`:
`ngx.timer.running_count()` / `pending_count()` of each Lua VM
- `apisix_nginx_timers_max{state}`: the configured `max_running_timers` /
`max_pending_timers`
- `apisix_nginx_process_connections{process}`: connection slots in use in
each process (`connection_n - free_connection_n`)
- `apisix_nginx_process_connections_max{process}`: connection slots of each
process (`worker_connections` for workers, 512 for the privileged agent)
- `apisix_nginx_lua_memory_bytes{process, subsystem}`:
`collectgarbage("count")` of each Lua VM
`process` is `worker-<id>` or `privileged-agent`. The motivating case is a
privileged agent that slowly leaks cosockets until its 512 connection slots are
gone: the workers look perfectly healthy, and `nginx_http_current_connections`
(stub_status) only counts accepted client connections, so nothing in the
current exposition moves until the agent starts failing.
None of these values can be read from another process, so each process
writes its own gauges into the metric dict on a timer (every
`refresh_interval`), and the privileged agent refreshes its own values right
before it renders the cache. The collection lives in its own module,
`apisix/plugins/prometheus/process_metrics.lua`; the exporter only registers it
and starts/stops it together with the plugin, so nothing new runs when
`prometheus` is disabled. The http and the stream VM report separately (they
have their own timers and memory), and the per-process values (connections,
timer limits) come from the http VM only.
The connection numbers are read through FFI from the exported `ngx_cycle`.
The struct prefix used is identical in every nginx core shipped in recent
runtimes, but a layout change would silently produce garbage, so there are two
guards: every process checks that the first connection slot is wired to the
first read/write events, and a worker checks that the number of slots it reads
equals `worker_connections` from the config. If that check fails, it logs one
warning and the two connection gauges are dropped for every process, including
the privileged agent, instead of exporting a wrong number.
Tests cover all five gauges for the worker and the privileged agent in both
subsystems, a worker's connection gauge going up by the number of cosockets
held open and back down when they are closed, the mismatch case, and the
stream-only mode in `t/cli/test_prometheus_stream.sh`. Docs are updated in both
languages.
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
