hanzhenfang commented on issue #13368:
URL: https://github.com/apache/apisix/issues/13368#issuecomment-4630178943
Hi, @Baoyuantop , I have reproduced this issue with a minimal Docker Compose
setup on `apache/apisix:3.16.0-debian`.
The important condition is: rotate `error.log` successfully first, then make
rotating `access.log` fail. In that case `log-rotate` returns before sending
`USR1`, so nginx/openresty keeps writing to the old file descriptor. The
current `logs/error.log` path then appears to stop receiving custom plugin logs.
Relevant files for reproduction are listed below.
`docker-compose.yaml`
```yaml
services:
etcd:
image: bitnamilegacy/etcd:3.6.4
environment:
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_ADVERTISE_CLIENT_URLS=http://etcd:2379
init-logs:
image: apache/apisix:3.16.0-debian
user: "0:0"
entrypoint:
- /bin/sh
- -c
- mkdir -p /usr/local/apisix/logs/access && chown -R 636:636
/usr/local/apisix/logs
volumes:
- apisix-logs:/usr/local/apisix/logs
apisix:
image: apache/apisix:3.16.0-debian
depends_on:
init-logs:
condition: service_completed_successfully
etcd:
condition: service_started
upstream:
condition: service_started
volumes:
- ./config.yaml:/usr/local/apisix/conf/config.yaml:ro
-
./plugins/log-probe.lua:/usr/local/apisix/apisix/plugins/log-probe.lua:ro
- apisix-logs:/usr/local/apisix/logs
ports:
- "127.0.0.1:19080:9080"
- "127.0.0.1:19090:9090"
- "127.0.0.1:19180:9180"
upstream:
image: nginx:alpine
volumes:
apisix-logs:
```
`config.yaml`
```yaml
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
allow_admin:
- 0.0.0.0/0
admin_key:
- name: admin
key: "my-secret-admin-key"
role: admin
etcd:
host:
- "http://etcd:2379";
apisix:
node_listen: 9080
enable_control: true
control:
ip: "0.0.0.0"
port: 9090
nginx_config:
error_log: logs/error.log
error_log_level: info
http:
enable_access_log: true
access_log: logs/access/access.log
plugins:
- log-rotate
- log-probe
plugin_attr:
log-rotate:
interval: 30
max_kept: 3
max_size: -1
enable_compression: false
```
`plugins/log-probe.lua`
```lua
local core = require("apisix.core")
local plugin_name = "log-probe"
local schema = {
type = "object",
properties = {},
}
local _M = {
version = 0.1,
priority = 1000,
name = plugin_name,
schema = schema,
}
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
function _M.access(conf, ctx)
core.log.warn("issue-13368 log-probe access uri=", ngx.var.uri)
end
function _M.rewrite(conf, ctx)
core.log.warn("issue-13368 log-probe rewrite uri=", ngx.var.uri)
end
function _M.log(conf, ctx)
core.log.warn("issue-13368 log-probe log uri=", ngx.var.uri)
end
return _M
```
## Reproduction Steps
1. Start APISIX:
```bash
docker compose up -d
```
2. Create a route that enables the custom plugin:
```bash
docker compose exec -T upstream sh -lc 'wget -S -O- \
--header "X-API-KEY: my-secret-admin-key" \
--header "Content-Type: application/json" \
--post-data
"{\"uri\":\"/repro\",\"plugins\":{\"log-probe\":{}},\"upstream\":{\"type\":\"roundrobin\",\"nodes\":{\"upstream:80\":1}}}"
\
http://apisix:9180/apisix/admin/routes'
```
3. Send one request before breaking rotation. The upstream returns `404`,
which is OK here; the APISIX plugin phases still run.
```bash
docker compose exec -T upstream wget -qO- http://apisix:9080/repro || true
```
4. Confirm the custom plugin log exists:
```bash
docker compose exec -T apisix sh -lc \
'find /usr/local/apisix/logs -type f -name "*error.log" -exec grep -H
"issue-13368 log-probe" {} + | tail -10'
```
5. Now make the access log directory non-writable. This forces
`os.rename("/usr/local/apisix/logs/access/access.log", "...__access.log")` to
fail during the next rotation.
```bash
docker compose exec -T apisix sh -lc \
'chmod 555 /usr/local/apisix/logs/access && ls -ld
/usr/local/apisix/logs/access'
```
6. Send another request before the next rotation:
```bash
docker compose exec -T upstream wget -qO- http://apisix:9080/repro || true
```
7. Wait for the next `log-rotate` interval, then send one more request:
```bash
sleep 35
docker compose exec -T upstream wget -qO- http://apisix:9080/repro || true
```
8. Inspect the logs:
```bash
docker compose exec -T apisix sh -lc \
'find /usr/local/apisix/logs -type f -name "*error.log" -exec grep -H
"Permission denied\|send USR1 signal\|issue-13368 log-probe" {} + | tail -40'
```
## Observed Result
The normal rotations send `USR1`, for example:
```text
/usr/local/apisix/logs/2026-06-05_09-26-00__error.log:
2026/06/05 09:26:00 [warn] ... log-rotate.lua:230: rotate_file(): send USR1
signal to master process [1] for reopening log file
```
At the failed rotation, `access.log` cannot be renamed:
```text
/usr/local/apisix/logs/2026-06-05_09-26-30__error.log:
2026/06/05 09:26:30 [error] ... log-rotate.lua:150: rename_file(): move file
from /usr/local/apisix/logs/access/access.log to
/usr/local/apisix/logs/access//2026-06-05_09-26-30__access.log res:nil
msg:/usr/local/apisix/logs/access/access.log: Permission denied
```
There is no `send USR1 signal ... for reopening log file` message for that
failed rotation.
After the failed rotation, a new request still logs to the rotated file, not
the current `logs/error.log` path:
```text
/usr/local/apisix/logs/2026-06-05_09-26-30__error.log:
2026/06/05 09:26:48 [warn] ... log-probe.lua:26: phase_func(): issue-13368
log-probe rewrite uri=/repro
2026/06/05 09:26:48 [warn] ... log-probe.lua:22: phase_func(): issue-13368
log-probe access uri=/repro
2026/06/05 09:26:48 [warn] ... log-probe.lua:30: phase_func(): issue-13368
log-probe log uri=/repro while logging request
```
So the custom plugin is still executing and writing logs, but
nginx/openresty continues writing to the old file descriptor because the failed
`access.log` rotation causes `rotate_file()` to return before sending `USR1`.
This looks consistent with the current flow in
`apisix/plugins/log-rotate.lua`: if any `rename_file(...)` call returns `nil`,
`rotate_file()` returns immediately and skips the reopen signal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
