nic-6443 opened a new pull request, #13826: URL: https://github.com/apache/apisix/pull/13826
`max_pending_entries` is only honoured when the plugin's metadata configures it, so out of the box the batch processor keeps every undelivered entry in worker memory. A log server that is slow or unreachable therefore grows the worker's memory with the request rate until the process hits its limit. It's most visible with `include_req_body` / `include_resp_body` on, since each entry then carries a copy of both bodies. The limit now defaults to 16384, and the fallback lives in `batch-processor-manager` rather than in each plugin, so a logger can't be wired up without it. That also fixes the other half of the problem: `datadog`, `lago`, `loggly`, `sls-logger` and `syslog` all go through the same manager but never exposed the knob at all. They do now, as does the stream subsystem's `syslog`. Two things come along with it: - the manager does the metadata lookup itself, using the plugin name that `new()` now takes, instead of twelve plugins repeating the same three lines and passing the value down; - discards are reported at most once per second with a running count. One line per discarded entry would turn the outage that causes the discards into a log flood — at 10k req/s that's 10k error lines a second. ### Picking the default I measured what one pending entry costs by parking entries in the buffer (batch never flushes) against a log endpoint that accepts the connection and never answers, with an equal-size control run over a route with no logger subtracted out. Single worker, `http-logger`, both bodies logged: | Body size, each side | Entries parked | Net RSS | Per entry | Entries in 128 MB | |---:|---:|---:|---:|---:| | bodies not logged | 119,974 | 230 MB | ~2.0 KB | ~66,800 | | 1 KB | 89,981 | 380 MB | ~4.3 KB | ~30,300 | | 4 KB | 29,996 | 313 MB | ~10.7 KB | ~12,270 | | 16 KB | 5,804 | 214 MB | ~37.7 KB | ~3,480 | | 64 KB | 1,808 | 282 MB | ~160 KB | ~820 | | 256 KB | 456 | 207 MB | ~464 KB | ~282 | 16384 is the largest power of two that stays inside a 128 MB budget for a representative body-logging setup (1 KB request + 1 KB response). Two things worth noting if you'd rather have a different number: it's ~16x the backlog a healthy system carries (measured below), and with bodies not logged — the default for every one of these plugins — it costs about 60 MB. ### What the default actually costs Re-run with the default in place and otherwise stock batch processor settings, against the same dead log endpoint. `pending` plateaus at exactly 16385 every time, which is the cap doing its job: | Body logged per request | Peak worker memory | Backlog with a log server that answers | |---|---:|---:| | bodies not logged | 63 MB | 982 entries @ 3000 req/s | | 1 KB + 1 KB | 172 MB | 960 entries @ 2000 req/s | | 4 KB + 4 KB | 469 MB | 986 entries @ 1200 req/s | | 16 KB + 16 KB | 1636 MB | 817 entries @ 600 req/s | These are higher than the entries alone account for, because batches already handed to the sender hold both their entries and the serialized payload built from them. That table is in `batch-processor.md`, along with a note to lower the limit when logging bodies larger than a few KB. The right column is the reason I'm comfortable with a default this large: with a log server that keeps up, the backlog sits under 1000 entries regardless of rate, so the limit stays well out of the way in healthy operation. ### Tests `t/utils/batch-processor-manager.t` covers the cap applying with no metadata configured, metadata still overriding it, the once-per-second discard reporting, and every batch-processor logger exposing the knob. All four fail on master and pass here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
