x1117 commented on issue #13658: URL: https://github.com/apache/apisix/issues/13658#issuecomment-5139081617
While the fix in PR #13754 (bumping to `nginx-lua-prometheus-api7 1.0.0`) correctly addresses the root cause, I'd like to raise a potential performance concern about the implementation. The C implementation of `ngx.shared.DICT.flush_expired()` holds the shared dict mutex for the **entire duration** of the scan — there is no yielding, no batching, and no lock release-and-reacquire. It iterates the LRU queue from the tail, deletes expired entries (rbtree + slab free), and only releases the lock when finished. With the current design, `remove_expired_keys_interval` is hardcoded to 3600s (since APISIX passes a string prefix to `Prometheus.init()`, the code takes the string branch which locks the interval to `MAX_REMOVE_EXPIRED_KEYS_INTERVAL`). This means expired entries accumulate for up to 1 hour before cleanup. In a scenario with 100 routes, 4 metric types, and ~30 label combinations each, that's roughly `100 × 4 × 30 × 2` (key index + value) = **24,000 expired entries per hour**. With higher cardinality or more routes, this can easily reach 100,000+. The original reporter's dump showed **747,970 stale entries** in a single dictionary. Each expired entry takes ~1–5 µs to process in the C layer. So: - 24,000 entries → 24–120ms hold time → noticeable P99 latency spikes - 100,000 entries → 100–500ms → potential request timeouts - 750,000 entries → 0.75–3.75s → catastrophic During the lock hold, **all other workers are blocked** from reading or writing `prometheus-metrics`. Since nginx workers are single-threaded, the worker running `flush_expired()` cannot process any requests until it completes. In short, the current fix trades a memory leak for a potential latency spike. For production environments with high label churn, this could be equally problematic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
