membphis commented on issue #12928: URL: https://github.com/apache/apisix/issues/12928#issuecomment-6408143517
We reproduced a specific CLI memory-growth scenario in both **APISIX 3.13.0 and 3.19.0** (the latest stable release checked on October 26, 2026). The affected process is the standalone LuaJIT process running `apisix/cli/apisix.lua start`; worker memory remains relatively stable after warm-up. **Important distinction from the default Kubernetes deployment** | Startup path | Access log destination | Exercises the affected CLI capture path? | | --- | --- | --- | | Official image / Helm default: `docker-start` | Worker stdout → container logging system | No | | Custom entrypoint invoking `apisix start` | Worker stdout → pipe captured by the CLI → CLI memory | Yes | Writing access logs to `/dev/stdout` alone is therefore insufficient to trigger this issue. The official entrypoint initializes APISIX and then directly executes OpenResty with `daemon off;`, bypassing the affected startup path. The official Helm chart also uses this entrypoint. See the [image entrypoint](https://github.com/apache/apisix-docker/blob/master/utils/docker-entrypoint.sh) and [Helm startup configuration](https://github.com/apache/apisix-helm-chart/blob/apisix-2.18.2/charts/apisix/templates/deployment.yaml#L65-L76). **Minimal reproduction** 1. Use an official `apache/apisix:3.13.0-debian` or `apache/apisix:3.19.0-debian` image with standalone YAML configuration, one worker, and a route using the `mocking` plugin to return HTTP 200 with body `ok`. 2. Enable access logging and set `nginx_config.http.access_log: /dev/stdout`. Override the default entrypoint to invoke `apisix start`. 3. Apply a container memory limit with swap disabled, and continuously request the route from outside the container using 20 concurrent HTTP/1.1 keep-alive connections. Monitor the CLI RSS separately from worker RSS. 4. CLI memory grows with cumulative traffic and can reach the container limit. In our test wrapper, PID 1 waits for the CLI and exits with its status. When the kernel OOM-kills the CLI, the wrapper exits, the container stops, and requests fail. **Observed results** Tests ran on Linux x86_64 with Docker and cgroup v2, using the default access log format. | Version | Container memory limit | Runs | Result | | --- | --- | --- | --- | | 3.13.0 | 128 MiB | 4 | CLI OOM-killed in every run; container exit and request failures after approximately 31–34 seconds | | 3.19.0 | 128 MiB | 2 | CLI OOM-killed in both runs; container exit and request failures after approximately 21–22 seconds | | 3.19.0 | 1 GiB | 1 | CLI OOM-killed after approximately 325 seconds, followed by container exit and request failures | The 128 MiB limit was used to accelerate repeated reproduction. In the separate **1 GiB** run: - CLI RSS increased from approximately **7.5 MiB to 966 MiB**; worker RSS remained approximately **40 MiB** after warm-up. - Approximately **6.47 million requests** completed successfully; **19 in-flight responses** were interrupted during shutdown. - All **120 independent probes over the following 30 seconds** failed with connection refused. - OOM was confirmed by kernel memory-cgroup records, not inferred solely from exit code 137. With the default `docker-start` entrypoint, each version completed approximately 1.81 million requests over a 90-second control run without this growth or request failures. These timings depend on traffic and log volume. **Cause and scope** The CLI starts OpenResty through `io.popen()` and reads its stdout using `read("*all")`. NGINX opens `/dev/stdout` for access logging and workers retain the pipe's write end. LuaJIT keeps the already-read output in a growing temporary buffer while waiting for EOF, so access logs accumulate in the launcher process. See the [CLI output-capture implementation](https://github.com/apache/apisix/blob/3.19.0/apisix/cli/util.lua#L36-L51). The launcher should not retain the server's lifetime log output. Whether killing the CLI interrupts service depends on process supervision: keeping the parent alive allowed NGINX to continue serving in a separate test, although access-log writes reported `Broken pipe`. This establishes a reproducible defect in that specific startup path. To determine whether it explains the original report, we still need the actual Pod `command`/`args` and image entrypoint or wrapper script. If the deployment uses the official default entrypoint, this finding does not establish the cause of its memory growth. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
