x1117 opened a new issue, #13658:
URL: https://github.com/apache/apisix/issues/13658
### Current Behavior
While using APISIX with the Prometheus plugin enabled, we observed that the
free space of the `prometheus-metrics` shared dict decreases periodically in a
step-like pattern.
The metric we monitored is:
```text
apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}
```
The pattern looks like this:
```text
free_space stays stable for a while
↓
drops by one step
↓
stays stable again
↓
drops by another step
```
In our environment, this happened approximately once per hour. The active
time series count was not very high, but the shared dict free space kept
decreasing.
We also dumped the `prometheus-metrics` shared dict and found many expired
KeyIndex entries still present, for example:
```text
__ngx_prom__key_N
```
One dump showed:
```text
key_count=751496
free=148631552
cap=536870912
phys_used=388239360
```
Entry summary:
```text
IDX PERM 76
IDX alive 28
IDX STALE 747970
VAL PERM 75
VAL alive 28
VAL STALE 4124
VAL GONE 77
```
Most stale entries were `__ngx_prom__key_N` index entries.
We reproduced the same type of behavior locally with APISIX `3.17.0`.
During reproduction, `prometheus-metrics` free space dropped multiple times:
```text
11:43:26 10395648 -> 10330112 drop 65536
11:50:08 10330112 -> 10289152 drop 40960
12:01:27 10289152 -> 8794112 drop 1495040
12:01:42 8794112 -> 8638464 drop 155648
12:08:08 8638464 -> 7581696 drop 1056768
```
It looks like expired `__ngx_prom__key_N` entries may remain in the
`prometheus-metrics` shared dict and continue occupying memory, while later
traffic creates new KeyIndex slots.
### Expected Behavior
When Prometheus metrics are configured with `expire > 0`, the expired metric
value entries and their related KeyIndex entries are expected to be reclaimed
effectively, or at least not keep consuming additional physical memory in the
`prometheus-metrics` shared dict.
If the number of active time series is bounded, the free space of
`prometheus-metrics` should eventually become stable instead of continuously
dropping step by step.
From our observation, the expired entries may not be physically reclaimed
because OpenResty shared dict expiration is lazy. The passive expiration path
only checks a small number of entries from the LRU tail, and it can be blocked
when permanent entries are at the tail. As a result, expired Prometheus entries
can remain in the shared dict and continue occupying slab memory.
If this is expected behavior of the current implementation, could you please
suggest the recommended way to avoid continuous `prometheus-metrics` shared
dict memory consumption by expired entries?
### Error Logs
There were no fatal APISIX errors during reproduction.
We temporarily added logs around Prometheus KeyIndex operations to
understand the behavior.
The KeyIndex slot number kept increasing:
```text
03:43:17 idx=48..161
03:49:57 idx=162..275
04:01:22~04:01:28 idx=276..2555
04:08:04~04:08:07 idx=2556..4721
```
`remove_expired_keys()` also reported worker-local cleanup:
```text
remove_expired_keys checked=2280 removed=2280 last=2555
```
After that, later traffic created new KeyIndex slots.
### Steps to Reproduce
1. Run APISIX `3.17.0` via Docker:
```text
apache/apisix:3.17.0-debian
```
2. Enable the Prometheus plugin and configure metric expiration:
```yaml
plugin_attr:
prometheus:
metrics:
http_status:
expire: 120
http_latency:
expire: 120
bandwidth:
expire: 120
upstream_status:
expire: 120
```
3. Create one or more routes with the Prometheus plugin enabled.
4. Send intermittent traffic. For example:
```text
active traffic for 60s
idle for 330s
active traffic for 60s
idle for 330s
active traffic for 60s
```
5. Monitor the following metric:
```text
apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}
```
6. In our local reproduction, to shorten the reproduction time, we
temporarily changed Prometheus initialization from:
```lua
prometheus = base_prometheus.init("prometheus-metrics", metric_prefix)
```
to:
```lua
prometheus = base_prometheus.init("prometheus-metrics", {
prefix = metric_prefix,
remove_expired_keys_interval = 300,
})
```
This made the local cleanup interval shorter than the default `3600s`.
7. Observe that `prometheus-metrics` free space drops in steps. In our
reproduction, we observed:
```text
11:43:26 10395648 -> 10330112 drop 65536
11:50:08 10330112 -> 10289152 drop 40960
12:01:27 10289152 -> 8794112 drop 1495040
12:01:42 8794112 -> 8638464 drop 155648
12:08:08 8638464 -> 7581696 drop 1056768
```
### Environment
- APISIX version: `3.17.0`
- Docker image: `apache/apisix:3.17.0-debian`
- Prometheus plugin: enabled
- Prometheus dependency in APISIX `3.17.0`:
```text
nginx-lua-prometheus-api7 = 0.20250302-1
```
- Operating system for local reproduction:
```text
Linux 6.6.87.2-microsoft-standard-WSL2
```
- etcd version used in local Docker compose:
```text
bitnamilegacy/etcd:3.5.11
```
- APISIX Dashboard version: not used
- Plugin runner: not used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
