Re: [I] bug: APISIX 3.17.0 is unable to start containers built from the base image [apisix]
juzhiyuan closed issue #13775: bug: APISIX 3.17.0 is unable to start containers built from the base image URL: https://github.com/apache/apisix/issues/13775 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 is unable to start containers built from the base image [apisix]
janiussyafiq commented on issue #13775:
URL: https://github.com/apache/apisix/issues/13775#issuecomment-5422125208
Thanks for the screenshots. They show the pod has no `command`/`args`
override and exits with code 0 in the same second with no logs. That means the
image's `CMD` is no longer `docker-start`.
The cause is `docker commit`: it saves the container's runtime settings into
the image. If the container was started with a shell (for example `docker run
-it --user root apache/apisix:3.17.0-ubuntu /bin/bash`), the committed image
gets `CMD ["/bin/bash"]`, the same way it got `USER root`. Your Dockerfile does
not set `CMD`, so the kaniko build inherits it. In the pod,
`/docker-entrypoint.sh /bin/bash` runs `bash` with no stdin, which exits 0
immediately with no output. I reproduced this locally with the same steps.
You can confirm with:
```bash
docker inspect xxx/apisix-deploy:4708a41c --format '{{json .Config.Cmd}}'
```
Expected `["docker-start"]`; `["/bin/bash"]` means this is the cause.
**Fix: build from the official image and drop `docker commit`.** The commit
was only needed because `apt-get` fails as the `apisix` user. Switch user
inside the Dockerfile instead:
```dockerfile
FROM apache/apisix:3.17.0-ubuntu
USER root
RUN apt-get update && apt-get install -y --no-install-recommends
openjdk-21-jdk-headless \
&& rm -rf /var/lib/apt/lists/*
COPY target/*.jar /usr/local/apisix/
RUN chown -R apisix:root /usr/local/apisix/
USER apisix
```
This keeps `ENTRYPOINT`, `CMD ["docker-start"]` and `USER apisix` from the
official image, so no override is needed in the Deployment, and upgrading
APISIX later is just changing the `FROM` line. I verified this Dockerfile
against etcd: the container starts and serves `Server: APISIX/3.17.0`.
If you need to keep the current image for now, adding `CMD ["docker-start"]`
to the Dockerfile or `args: ["docker-start"]` in the Deployment also works.
Note: `bash /docker-entrypoint.sh` with no argument never starts APISIX,
even in the official image. Use `bash /docker-entrypoint.sh docker-start` for
manual tests.
Could u first confirm with above fix first? Let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix-dashboard]
Neilblaze commented on issue #3464: URL: https://github.com/apache/apisix-dashboard/issues/3464#issuecomment-5395454149 Dug into this and reproduced it against the exact dashboard bundled in APISIX 3.17.0, which was built from c8d3466d. At that commit, `PluginCardList` kept a MobX `useLocalObservable` store, and its computed captured the delete handler from the first render. That render happens before `form.reset()` fills in the fetched values, so the handler closed over an empty `plugins` object. Deleting any card then rewrote `plugins` from that stale empty snapshot and cleared everything, which lines up with the empty plugins payload you captured. Current master is not affected. ec0fce26 (#3446) replaced the store with a plain `useMemo`, which removed the stale capture. I opened #3469 with an end to end regression test so this scenario stays pinned. For 3.17.0 users, the fix ships once apache/apisix bumps `APISIX_DASHBOARD_COMMIT` past ec0fce26. One extra heads up: on routes and services the same wipe saves successfully because `plugins` is optional there, so plugin deletion on those pages is risky on 3.17.0 as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix-dashboard]
juzhiyuan commented on issue #3464: URL: https://github.com/apache/apisix-dashboard/issues/3464#issuecomment-5390906399 sure @Neilblaze -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix-dashboard]
Neilblaze commented on issue #3464: URL: https://github.com/apache/apisix-dashboard/issues/3464#issuecomment-5386991524 If this is still up for grabs, I'd love to take this up! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
passerbyabc commented on issue #13673: URL: https://github.com/apache/apisix/issues/13673#issuecomment-5350842783 ## Additional reproduction (production, K3s + APISIX Ingress) Same regression as this issue. Confirmed it is **not** an actually-oversized body. ### Environment - APISIX on K3s (Ingress Controller), TLS terminated at APISIX - `nginx.conf` has `client_max_body_size 0;` (unlimited, default) - HTTP/1.1 POST from a Spring `RestTemplate` webhook client - Route: third-party callback `POST /callback/...` → HTTP upstream ### Access log (redacted) ... "POST /callback_api/... HTTP/1.1" 413 255 0.018 "-" "" - - - `upstream_addr` / `upstream_status` / `upstream_response_time` are all `-` → rejected in APISIX, never forwarded. ### Error log (redacted) client intended to send too large chunked body: 0+311 bytes, ... request: "POST /callback/... HTTP/1.1", host: "xxx.xxx.com" First chunk is only **311 bytes**. The `0+` matches `client_max_body_size 0` treated as a literal 0-byte quota on the chunked path. ### Temporary workaround Leaving the global limit at `0` still 413s **any** non-empty chunked POST. Setting a **non-zero** limit unblocks it. **Option A (what we used): route-level `client-control`** — ApisixRoute / Admin API, no APISIX restart: ```yaml plugins: - name: client-control enable: true config: max_body_size: 10485760 # 10MiB; must be != 0 After this, the same webhook returned 200 and upstream_* was populated. Option B (global, as in this issue): non-zero client_max_body_size — requires rendering nginx.conf and restarting APISIX: nginx_config: http: client_max_body_size: 100m # must be != 0; do not use 0 We prefer A until a runtime image that includes api7/apisix-nginx-module#115 is available. ### Question Which released APISIX / apisix-runtime image includes api7/apisix-nginx-module#115 so we can drop the route workaround? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 is unable to start containers built from the base image [apisix]
wangchao732 commented on issue #13775: URL: https://github.com/apache/apisix/issues/13775#issuecomment-5312283244 @janiussyafiq Thanks for your reply. I'm attaching a few screenshots to give you a clearer picture: https://github.com/user-attachments/assets/7b5cd5bf-9c4a-41d6-96d0-b5d37e4ab4cc"; /> https://github.com/user-attachments/assets/5ec5a358-8e9c-455e-9b3e-fb972ce086bf"; /> https://github.com/user-attachments/assets/7d96c5f6-de81-4a16-ad29-5b5c9b4000fb"; /> https://github.com/user-attachments/assets/12ed7ee0-77c6-4386-bb4e-99a31010f65a"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 is unable to start containers built from the base image [apisix]
janiussyafiq commented on issue #13775: URL: https://github.com/apache/apisix/issues/13775#issuecomment-5290276530 Hi @wangchao732, I reproduced your exact chain (run `--user root`, `docker commit`, your Dockerfile with openjdk-21-jdk-headless, `chown -R apisix:root`, `USER apisix`) and the resulting image starts APISIX normally, so the commit-based image itself is fine. What you observed is expected behavior of the entrypoint. `/docker-entrypoint.sh` only starts APISIX when its first argument is `docker-start`: ```bash if [[ "$1" == "docker-start" ]]; then ... exec /usr/local/openresty/bin/openresty -p /usr/local/apisix -g 'daemon off;' fi exec "$@" ``` Running `bash /docker-entrypoint.sh` with no argument falls through to `exec "$@"`, which does nothing: it exits 0 with no output and no openresty process, exactly what you saw. The same command with the argument works: `bash /docker-entrypoint.sh docker-start`. The most common way to hit this in a real deployment is a Kubernetes `command:` override, which replaces the image CMD (`docker-start`), so the container exits immediately and silently. If they override the command, either remove them or use: ```yaml command: ["/docker-entrypoint.sh"] args: ["docker-start"] ``` Let me know if this explains the issue you encountered, and if not feel free to reply with more details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix-dashboard]
juzhiyuan commented on issue #3464: URL: https://github.com/apache/apisix-dashboard/issues/3464#issuecomment-5288357266 Hello @ipanocloud, do you have interest to submit a PR to fix it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix-dashboard]
ipanocloud opened a new issue, #3464:
URL: https://github.com/apache/apisix-dashboard/issues/3464
### Current Behavior
Current Behavior
In the built-in Dashboard of APISIX 3.17.0, when editing a Plugin Config
that contains multiple plugins:
- Click the **Delete** button on any single plugin card
- Click **Submit** to save
- **All plugins are cleared** (plugins: {} in the request payload)
This is a **frontend UI bug**, not an APISIX core bug.
### Expected Behavior
- Deleting one plugin should only remove that plugin
- All other plugins remain unchanged
- Submit and save correctly
### Error Logs
_No response_
### Steps to Reproduce
1.Open APISIX 3.17.0 built-in Dashboard
2.Go to Plugin Config
3.Create or edit a Plugin Config with two or more plugins (e.g., cors,
prometheus, limit-req)
4.Delete one plugin using the delete icon on the plugin card
5.Click Submit
6.Re-enter edit page: all plugins are gone
### Environment
- APISIX version: 3.17.0
- Dashboard: Built-in
- Browser: Chrome / Edge / Firefox (all reproduce)
- Deployment: Binary / Docker (both reproduce)
- OS: Linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
[I] bug: APISIX 3.17.0 Built-in Dashboard: Deleting one plugin in Plugin Config will remove all plugins [apisix]
ipanocloud opened a new issue, #13823:
URL: https://github.com/apache/apisix/issues/13823
### Current Behavior
Current Behavior
In the built-in Dashboard of APISIX 3.17.0, when editing a Plugin Config
that contains multiple plugins:
Click the Delete button on any single plugin card
Click Submit to save
All plugins are cleared (plugins: {} in the request payload)
This is a frontend UI bug, not an APISIX core bug.
### Expected Behavior
Deleting one plugin should only remove that plugin
All other plugins remain unchanged
Submit and save correctly
### Error Logs
_No response_
### Steps to Reproduce
1.Open APISIX 3.17.0 built-in Dashboard
2.Go to Plugin Config
3.Create or edit a Plugin Config with two or more plugins (e.g., cors,
prometheus, limit-req)
4.Delete one plugin using the delete icon on the plugin card
5.Click Submit
6.Re-enter edit page: all plugins are gone
### Environment
- APISIX version: 3.17.0
- Dashboard: Built-in (Next Gen)
- Browser: Chrome / Edge / Firefox (all reproduce)
- Deployment: Binary / Docker (both reproduce)
- OS: Linux
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
manueljishi commented on issue #2708:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-5265451571
We hit this and tracked it to a reproducible root cause. Posting in full
since the issue was just marked stale, and because I think #2689 and #2699 are
the same bug.
**TL;DR:** it is not the controller and not APISIX. It is the **adc sidecar
version boundary at v0.22.0**, combined with adc's *update* path being unable
to migrate a service from the old inline-upstream format to the new split
format.
## Environment
| | before | after |
|---|---|---|
| apisix helm chart | 2.12.4 | 2.16.0 |
| APISIX | 3.14.1 | 3.17.0 |
| apisix-ingress-controller | 2.0.0-rc5 | 2.1.0 |
| **adc** | **0.21.2** | **0.26.0** |
Provider is etcd/traditional (`deployment.role: traditional`,
`config_provider: etcd`), external etcd, ~218 services, Kubernetes 1.33 on EKS.
## Symptom
After the chart upgrade, services whose pods were replaced started returning
502 permanently. Everything else looked healthy:
- controller logs `"status":"success"`
- Argo shows Synced / Healthy
- `ApisixRoute` status is `Accepted=True`
- the controller sends the **correct** node IPs
Only services whose pods happened to cycle after the upgrade were affected,
so it degrades gradually rather than failing loudly.
## Root cause
adc changed how service config is stored:
- **[api7/adc#354](https://github.com/api7/adc/pull/354)** (v0.22.0)
`feat(apisix): separate inline upstream`
- **[api7/adc#384](https://github.com/api7/adc/pull/384)** (v0.23.0)
`fix(apisix): both inline and referenced upstreams coexist within the service`
— *"This PR will commit the removal of the inline upstream upon service
updates."*
So:
- adc **≤ 0.21.x** → service holds an embedded `"upstream": {nodes: [...]}`
- adc **≥ 0.22.0** → service holds `"upstream_id"`, nodes live in a separate
`/apisix/upstreams/`
The apisix helm chart defaults the adc sidecar to **0.21.2 in chart 2.12.x**
and **0.26.0 in chart 2.16.0**, so a routine chart upgrade crosses this
boundary. Neither the APISIX nor the apisix-ingress-controller release notes
mention it.
**Existing services are not migrated.** adc's update path reads a service
that still has an inline upstream, assumes the separate upstream object already
exists, and emits only the service write. APISIX rejects it:
```
PUT /apisix/admin/services/ -> 400 Bad Request
error_msg: failed to fetch upstream info by upstream id [], response
code: 404
```
One rejection fails the whole ADC batch, so unrelated services in the same
batch also stop converging. In our cluster this was 28 failures per sync.
## Evidence chain
Names below are redacted as `` / ``; IDs are the real
deterministic hashes for a single service.
Controller is sending the right thing — from the `prepared request body` log
line:
```json
"upstream":{"labels":{"managed-by":"apisix-ingress-controller"},
"nodes":[{"host":"10.0.17.10","port":3000,"weight":100}]}
```
etcd still holds the old value:
```console
$ etcdctl get /apisix/services/ --print-value-only
{"upstream":{"nodes":[{"weight":100,"port":3000,"host":"10.0.30.68"}],...}}
```
Real endpoint at the time was `10.0.17.10`; `10.0.30.68` belonged to no
running pod. Data plane confirms:
```
balancer.lua:376: run(): proxy request to 10.0.30.68:3000 while connecting
to upstream
connect() failed (113: No route to host) while connecting to upstream,
upstream: "http://10.0.30.68:3000/";
```
State was split roughly in half — **218 services, only 107 upstream
objects** — so over 100 services referenced an `upstream_id` that was never
created.
Notably, the controller payload contains `services`, `routes` and
`consumers` but **no `upstreams` array** — the inline→split conversion happens
entirely inside adc.
## What did not work
- **Restarting the controller / APISIX.** No effect — the stale values are
in etcd.
- **`ingressClassName: apisix` on the CRs** (the resolution on #2699).
Tested directly; the service stayed on the inline format. Those routes were
already `Accepted=True`, so that fix addresses a different failure.
- **Forcing a service rewrite** by appending a throwaway host to the route.
This *does* migrate a service — but only where the upstream object already
exists. It migrated 45 of 215 and then stalled with no error.
- **Upgrading adc forward.** Checked every release 0.26.0 → 0.29.0. The
differ v4 work in 0.27.1/0.28.0 is explicitly *"only a code refactoring; you
should expect no difference in behavior"*. Nothing addresses upstream creation
ordering.
## What did work
**adc's create path is fine — only the update path is broken.** Deleting the
service object makes adc treat
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
github-actions[bot] commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-5248001328 This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: APISIX 3.17.0 is unable to start containers built from the base image [apisix]
wangchao732 opened a new issue, #13775: URL: https://github.com/apache/apisix/issues/13775 ### Current Behavior baseimgae: apache/apisix:3.17.0-ubuntu Since the Apisix user lacks apt permissions, OpenJDK 21 is installed using docker --privileged --user root. ```dockerfile FROM xxxapisix:3.17.0-ubuntu-2026080415 RUN apt-get update && apt-get install -y openjdk-21-jdk-headless COPY target/*.jar /usr/local/apisix/ RUN chown -R apisix:root /usr/local/apisix/ USER apisix ``` ext-plugin It fails to start regardless of whether it's configured or not. I tried using sleep 3600 to keep the container running. When I manually executed /usr/local/openresty/bin/openresty -p /usr/local/apisix -g 'daemon off;' inside the container, it worked fine. However, running bash /docker-entrypoint.sh produces no output, and ps -aux shows no openresty process. ### Expected Behavior _No response_ ### Error Logs null ### Steps to Reproduce 1、pull image apache/apisix:3.17.0-ubuntu; 2、docker run --user root apache/apisix:3.17.0-ubuntu && docker commit xxx xxx/apisix:3.17.0-ubuntu-2026080415; 3、build images above Dockefile; 4、deployment; ### Environment - APISIX version: 3.17.0 - Operating system (run `uname -a`): Ubuntu 24.04.4 LTS - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): 1.29.2.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
nic-6443 closed issue #13673: bug: # APISIX 3.16 → 3.17 Chunked POST Regression URL: https://github.com/apache/apisix/issues/13673 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: [apisix]
cfitzw opened a new issue, #13685:
URL: https://github.com/apache/apisix/issues/13685
### Current Behavior
There is a bug that started happening in v3.17.0.
I have a few commented out environment variables using the syntax
`${{MY_ENV_VAR}}`. It appears something is attempting to resolve those, even
though commented out.
It should not be resolving env vars from commented out lines.
```ansii
/usr/local/openrestv//luajit/bin/luajit ./apisix/cli/apisix.lua init
failed to read local vaml config of apisix: failed to handle configuration:
can't find environment variable MY_ENV_VAR
```
The routes files contains:
```yaml
# ${{MY_ENV_VAR}}
```
### Expected Behavior
The app should start without weeping due to commented out environment
variables.
### Error Logs
_No response_
### Steps to Reproduce
See above.
### Environment
From helm chart v2.16.0 = APISix 3.17.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
lxbme commented on issue #13673: URL: https://github.com/apache/apisix/issues/13673#issuecomment-4921840034 Hi @mcupei, thanks for the issue. I've reproduced this bug and it is caused by a rebase mistake at api7/apisix-nginx-module repository. ## Root cause APISIX 3.17 bumped the runtime base from OpenResty 1.27.1.2 to 1.29.2.4. While porting the patches to the new base, the `&& max_body_size` zero-guard was lost in `patch/1.29.2.4/nginx-client_max_body_size.patch` (hunk for `ngx_http_request_body_chunked_filter()`): ```c /* old patch — correct *//* new patch — guard lost */ if (max_body_sizeif (max_body_size && max_body_size- r->headers_in.content_length_n < rb->chunked->size) - r->headers_in.content_length_n < rb->chunked->size) ``` Since APISIX defaults to `client_max_body_size 0` (unlimited), the check becomes `0 - 0 < chunk_size` → always true → **413 for every HTTP/1.1 chunked request**, regardless of size. Requests with `Content-Length` take a different code path whose guard is intact, which is why only chunked requests fail. The fix is already proposed in api7/apisix-nginx-module#115. After it's merged, a new apisix-runtime release and rebuilt 3.17.x images are still needed. ## Workaround Until a fixed image is released, set any non-zero limit in `config.yaml` (verified on 3.17.0): ```yaml nginx_config: http: client_max_body_size: 1000m ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
lxbme commented on issue #13673: URL: https://github.com/apache/apisix/issues/13673#issuecomment-4921754993 Hi @mcupei, thanks for the issue. I've reproduced this bug and it seems caused by a rebase mistake at api7/apisix-nginx-module repository. ## Reproduction Confirmed with byte-identical standalone configs, only swapping the image tag (`apache/apisix:3.16.0-debian` vs `3.17.0-debian`, default `client_max_body_size: 0`): | Test (11-byte body `hello=world`) | 3.16.0 | 3.17.0 | |---|---|---| | POST with `Content-Length` | 201 | 200 | | POST with `Transfer-Encoding: chunked` | 200 | **413** | Error log on 3.17: ``` [error] client intended to send too large chunked body: 0+11 bytes ``` An 11-byte body triggering "too large" shows this is not a real size overflow. ## Root cause APISIX 3.17 bumped the base from OpenResty **1.27.1.2** (apisix-runtime 1.3.3, apisix-nginx-module 1.19.3) to **1.29.2.4** (apisix-runtime 1.3.6, apisix-nginx-module 1.19.5). It is **not** an upstream nginx change — vanilla `nginx:1.29.2` with `client_max_body_size 0` proxies chunked requests fine, and so does a bare (Lua-free) nginx built from the 3.16 image. The regression is in the rebased patch in apisix-nginx-module: `patch/1.29.2.4/nginx-client_max_body_size.patch`, hunk for `ngx_http_request_body_chunked_filter()`: ```c /* old patch (1.27.1.1, module 1.19.3) — correct */ if (max_body_size && max_body_size - r->headers_in.content_length_n < rb->chunked->size) /* new patch (1.29.2.4, module 1.19.5 / 1.19.6 / master) — guard line lost in rebase */ if (max_body_size - r->headers_in.content_length_n < rb->chunked->size) ``` The `&& max_body_size` zero-guard (which implements nginx's "0 = unlimited" semantics) was dropped. With the default `client_max_body_size 0`, the check becomes `0 - 0 < 11` → true → 413 for **every** HTTP/1.1 chunked request, regardless of size. That's exactly the `0+11 bytes` in the log. Requests with `Content-Length` take a different code path whose guards are intact, which is why only chunked requests fail. The HTTP/2 hunk in the same patch kept its guard, so only the HTTP/1.1 chunked path is affected. Note this also breaks the `client-control` plugin's dynamic path: `set_client_max_body_size(0)` returns 413 on 3.17 as well. ## Fix Restore the lost guard line in [`patch/1.29.2.4/nginx-client_max_body_size.patch#L86`](https://github.com/api7/apisix-nginx-module/blob/1.19.5/patch/1.29.2.4/nginx-client_max_body_size.patch#L86) (still missing on `master` and tag `1.19.6`): ```diff -@@ -1139,8 +1165,14 @@ ngx_http_request_body_chunked_filter(ngx_http_request_t *r, ngx_chain_t *in) +@@ -1139,8 +1165,15 @@ ngx_http_request_body_chunked_filter(ngx_http_request_t *r, ngx_chain_t *in) ... +if (max_body_size ++&& max_body_size +#else ``` This makes the hunk identical in shape to the old `1.27.1.1` patch. For any non-zero limit `X`, `X && (X - n < s)` is equivalent to `X - n < s`, so the change only affects the `0` case — restoring the "0 = unlimited" behavior with no other behavior change. I verified the fix by building openresty-1.29.2.4 + apisix-nginx-module 1.19.5 twice (as shipped vs. with the guard restored): - As-shipped build reproduces the bug exactly (chunked + limit `0` → 413). - Fixed build: chunked with limit `0` → 200 (11B and 5MB bodies); non-zero limits still enforced (1m limit, 2MB chunked → 413); exact boundary preserved (20-byte limit: 20B → 200, 21B → 413); `client-control` dynamic path correct (`set_client_max_body_size(5)` → 413, `(0)` → 200). - The fixed patch applies cleanly on a pristine openresty-1.29.2.4 tree. Shipping it would need a new apisix-nginx-module tag, an apisix-runtime release, and rebuilt 3.17.x images. Until then, a verified workaround for users is setting any non-zero limit in `config.yaml`: ```yaml nginx_config: http: client_max_body_size: 1000m ``` @Baoyuantop What do you think of this fix? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
mcupei commented on issue #13673: URL: https://github.com/apache/apisix/issues/13673#issuecomment-4913152854 The nginx core jump from **1.27 → 1.29** changed how the chunked-body-size accounting path treats a `client_max_body_size` of `0`: for standard (`Content-Length`) requests the "0 disables the check" special case still applies, but for **chunked** request bodies the newer core enforces the limit literally (0 allowed bytes), rejecting the very first chunk regardless of size. This is why the failure is specific to chunked requests and appears immediately after the 3.16→3.17 upgrade with no config change on the user's side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: # APISIX 3.16 → 3.17 Chunked POST Regression [apisix]
mcupei opened a new issue, #13673: URL: https://github.com/apache/apisix/issues/13673 ### Current Behavior ## Summary Chunked-Transfer-Encoding POST requests are rejected with **`413 Request Entity Too Large`** by `apache/apisix:3.17.0-redhat`, while the identical request against `apache/apisix:3.16.0-redhat` succeeds. Requests using a `Content-Length` header (no chunked encoding) work correctly on **both** versions. ### Expected Behavior Chunked-Transfer-Encoding POST requests are working like before. ### Error Logs _No response_ ### Steps to Reproduce This was reproduced in a k3d (Kubernetes) cluster with both images running side-by-side with byte-for-byte identical APISIX route configuration — the only variable was the container image tag. ### Environment - k3d v5.9.0 cluster (`apisix-test`), local registry at `apisix-registry:5000` - APISIX deployed in **standalone mode** (`config_provider: yaml`, no etcd) so both versions run from an identical `apisix.yaml` / `config.yaml` - Upstream: `mendhak/http-https-echo:31`, echoes back method/headers/body as JSON - Route: `uri: /*` → `echo-upstream.apisix-test.svc.cluster.local:80` - Manifests: `k8s/namespace.yaml`, `k8s/apisix-configmap.yaml`, `k8s/echo-upstream.yaml`, `k8s/apisix-v316.yaml`, `k8s/apisix-v317.yaml` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
AlinsRan commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4888911574 Agree with @SkyeYoung / @juzhiyuan on fail-fast over auto-generation. Reasoning, tied to the actual behavior today: The real bug here is the **config.yaml rewrite** in `core/id.lua`: on an empty `admin_key` it generates a key, flips `changed`, and rewrites the file — which drops comments and can emit a trailing `...`, breaking the next start. That write-back has to go regardless of what we decide for the empty-key case. Auto-generating the key (even purely in memory, without the rewrite) is still the wrong default for two reasons: 1. **It changes on every restart.** A token that silently rotates each boot is unusable for any real deployment and confusing to debug — you set nothing, it "works", then breaks after a restart. 2. **It hides a security-relevant misconfiguration.** Silently inventing an admin credential is the opposite of fail-safe; the operator never learns they shipped without a key. We already have the right switch for the "I really don't want auth" case: **`admin_key_required`**. So the clean model is: - `admin_key_required: false` → run without an admin key (already supported, already prints a warning). This is the getting-started / local-dev path. - `admin_key_required: true` (the default) + empty/missing key → **fail fast at startup** with a clear message telling the user to set a key. No generation, no file mutation. Concretely that means: drop `autogenerate_admin_key` and the `write_file` back-write in `core/id.lua`, and change the current "empty key → will auto-generate" warning in `cli/ops.lua` into a hard error when `admin_key_required` is on. Secure by default, no surprise config edits, and the escape hatch for trials is one explicit flag. Behavior change to call out: anyone relying on the old auto-generate would now need to either set a key or set `admin_key_required: false`. That seems acceptable given it's the documented, safer default. I can take this if nobody's already on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix report duplicate metrics [apisix]
AlinsRan closed issue #11934: bug: apisix report duplicate metrics URL: https://github.com/apache/apisix/issues/11934 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX Ingress Controller v2.0.0 - Service Inline Upstreams Not Updated on Endpoint Changes [apisix-ingress-controller]
github-actions[bot] commented on issue #2689: URL: https://github.com/apache/apisix-ingress-controller/issues/2689#issuecomment-4700372172 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX Ingress Controller v2.0.0 - Service Inline Upstreams Not Updated on Endpoint Changes [apisix-ingress-controller]
github-actions[bot] closed issue #2689: bug: APISIX Ingress Controller v2.0.0 - Service Inline Upstreams Not Updated on Endpoint Changes URL: https://github.com/apache/apisix-ingress-controller/issues/2689 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. [apisix]
nic-6443 closed issue #12973: bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. URL: https://github.com/apache/apisix/issues/12973 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix dashboards main page [apisix]
nic-6443 closed issue #13497: bug: apisix dashboards main page URL: https://github.com/apache/apisix/issues/13497 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 needs anyuid [apisix]
nic-6443 closed issue #13075: bug: Apisix 3.15.0 needs anyuid URL: https://github.com/apache/apisix/issues/13075 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix may return http 504 when use dubbo-proxy [apisix]
nic-6443 closed issue #10429: bug: apisix may return http 504 when use dubbo-proxy URL: https://github.com/apache/apisix/issues/10429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix dashboards main page [apisix]
nic-6443 commented on issue #13497: URL: https://github.com/apache/apisix/issues/13497#issuecomment-4676897898 This is an apisix-dashboard issue rather than an APISIX one — the error in your second screenshot comes from the dashboard's web UI, which lives in the apache/apisix-dashboard repo. It's a known crash in the language menu's translation-progress component, already tracked as apache/apisix-dashboard#3300 and fixed by apache/apisix-dashboard#3380 (merged 2026-06-02). The fix isn't in a tagged dashboard release yet, so to get it now you'd need to build the dashboard from master or use an image built after that date. I'd suggest closing this one here since the gateway itself isn't involved; the dashboard repo is the place to follow for the release containing the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 needs anyuid [apisix]
nic-6443 commented on issue #13075: URL: https://github.com/apache/apisix/issues/13075#issuecomment-4676895409 The fix for this is merged now — your own apache/apisix-docker#617 updated the debian/redhat/ubuntu release Dockerfiles with the group-0 ownership and `g=u` permissions that OpenShift's arbitrary UIDs need. The reason 3.15.0/3.16.0 images still require anyuid is that PR #12824 only touched the `debian-dev` Dockerfile inside this repo, while release images are built from apache/apisix-docker, and its 3.16.0 release was cut a few days before #617 merged. So the first release image built after April 2026 will include it; until then the dev image or a locally rebuilt image with the patched Dockerfile is the workaround. I'd suggest closing this since the fix is merged in the right repo — if the next release image still misbehaves, apisix-docker is the place to reopen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix may return http 504 when use dubbo-proxy [apisix]
nic-6443 commented on issue #10429:
URL: https://github.com/apache/apisix/issues/10429#issuecomment-4676892526
I believe this was fixed on the runtime side rather than in the Lua code.
The dubbo-proxy path multiplexes requests onto a shared upstream connection via
a per-request "fake" upstream, and that fake upstream didn't inherit the
per-request connect/send/read timeouts APISIX sets, so they defaulted to 0 —
requests attached to a main connection that was still connecting (your `multi:
connect reuse unfinished` lines) timed out instantly, which matches the 504s
with ~0.000s upstream time right at the start of the benchmark.
This was fixed by api7/ngx_multi_upstream_module#16 ("fix: missing fake
upstream timeout"), shipped in module 1.3.2 and bundled into apisix-runtime
starting with APISIX 3.13.0. Could you retest on 3.13.0+ (ideally current
3.16)? If the startup 504s are gone I'd suggest closing this; if they're still
there, fresh logs would help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
[I] bug: apisix dashboards main page [apisix]
chlinwei opened a new issue, #13497: URL: https://github.com/apache/apisix/issues/13497 ### Current Behavior When I use cursor to hover language switch button, there is no languages show in the list https://github.com/user-attachments/assets/f8f6bfc6-80d8-4d83-9955-d5315ea7a5c3"; /> After I click this button. I got the error like below. https://github.com/user-attachments/assets/c4f1acca-c379-4a48-88ad-40ce3052747a"; /> **Apisix version is 3.16.0** ### Expected Behavior _No response_ ### Error Logs _No response_ ### Steps to Reproduce 1. Just install Apisix 3.16.0 in the ubuntu 22.04 throught apt After configured the etcd. 2.start apisix normally 3. You will get this error after click language switch button in Apisix dashboard ### Environment - APISIX version (run `apisix version`): - Operating system (run `uname -a`): - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller continuously attempts to delete non-existent global_rule [apisix-ingress-controller]
dovics commented on issue #2778: URL: https://github.com/apache/apisix-ingress-controller/issues/2778#issuecomment-4618288009 Thank you for your reply. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller continuously attempts to delete non-existent global_rule [apisix-ingress-controller]
dovics closed issue #2778: bug: apisix-ingress-controller continuously attempts to delete non-existent global_rule URL: https://github.com/apache/apisix-ingress-controller/issues/2778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller continuously attempts to delete non-existent global_rule [apisix-ingress-controller]
Baoyuantop commented on issue #2778: URL: https://github.com/apache/apisix-ingress-controller/issues/2778#issuecomment-4611407742 Please refer to https://apisix.apache.org/docs/ingress-controller/upgrade-guide/#controller-only-configuration-source Starting with APISIX Ingress Controller 2.0.0, the controller is the single source of truth. Manual Admin API changes will be overwritten on the next full sync. The prior approach, which allowed controller-managed and manually added configurations to coexist, was incorrect and is now deprecated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix promethues metrics service request much greater than the actual request [apisix]
wangchao732 commented on issue #12449:
URL: https://github.com/apache/apisix/issues/12449#issuecomment-4601225171
@Baoyuantop Very glad to receive your reply. Best wishes! Details：
Service info:
{
"name": "dapr-service-vehicle-realtime-data",
"upstream": {
"nodes": [
{
"host": "10.233.31.69",
"port": 80,
"weight": 1
}
],
"timeout": {
"connect": 60,
"send": 60,
"read": 60
},
"type": "roundrobin",
"hash_on": "vars",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
}
}
Routers info:
1、
{
"uris": [
"/dapr-service-vehicle-realtime-data/*",
"/realtime/*"
],
"name": "dapr-service-vehicle-realtime-data-https",
"host": "xxx",
"plugins": {
"ext-plugin-pre-req": {
"allow_degradation": false,
"conf": [
{
"name": "TokenValidator",
"value":
"{\"validate_header\":\"token\",\"rejected_code\":\"403\"}"
}
],
"disable": false
}
},
"service_id": "420094949904614258",
"status": 1
}
2、
{
"uris": [
"/dapr-service-vehicle-realtime-data/info/query",
"/info/query",
"/realtime/getLastestInfos",
"/dapr-service-vehicle-realtime-data/realtime/getLastestInfos",
"/realtime/getLastLocation",
"/dapr-service-vehicle-realtime-data/realtime/getLastLocation",
"/dapr-service-vehicle-realtime-data/info/updatePositioningSysMode",
"/info/updatePositioningSysMode",
"/info/updateParam",
"/info/delParam",
"/dapr-service-vehicle-realtime-data/info/updateParam",
"/dapr-service-vehicle-realtime-data/info/delParam"
],
"name": "dapr-service-vehicle-realtime-data-info-query",
"methods": [
"POST"
],
"host": "xxx",
"service_id": "420094949904614258",
"status": 1
}
https://github.com/user-attachments/assets/76656c0b-c312-43bd-9781-5dfb62c5035f";
/>
Promethues config :
- job_name: "apisix"
scrape_interval: 30s
metrics_path: "/apisix/prometheus/metrics"
static_configs:
- targets: ["xxx:30215"] # apisix promethues expose port number.
Deployment:
spec:
replicas: 2
selector:
matchLabels:
app: dapr-service-vehicle-realtime-data
template:
metadata:
creationTimestamp: null
labels:
app: dapr-service-vehicle-realtime-data
annotations:
dapr.io/app-id: dapr-service-vehicle-realtime-data
dapr.io/app-port: '8080'
dapr.io/config: zipkin
dapr.io/enabled: 'true'
Grafana $__rate_interval: last 6 hours . When I manually set the Grafana
query range to 5 minutes and also change the query expression to a fixed value
(e.g., “5 minutes”), the issue persists; it recovers automatically after about
10–15 seconds, and a spike occurs approximately every 1–2 hours.
sum(rate(apisix_http_status{service=~"$service",route=~"$route",instance=~"$instance"}[5m]))
by (service)
https://github.com/user-attachments/assets/804f9074-5ef8-4365-bc95-dc880079507b";
/>
https://github.com/user-attachments/assets/48d8957f-57fa-402d--2861f21f8b51";
/>
Alert config:
- name: apisix_total_requests
rules:
- alert: apisix_total_requests
expr: sum(rate(apisix_http_status [5m])) > 10
for: 3m
labels:
severity: "P1"
annotations:
summary: "Apisix requests above 10."
description: "Attention! Host={{ $labels.instance }},name={{
$labels.name }},env={{ $labels.env }},service={{ $labels.service }},process={{
$labels.groupname }} Apisix request above 1000 for 5 minutes (current value: {{
$value }})"
APISIX access log count ： sed -n '/2026-06-02 17:38:40/,/2026-06-02
17::39:39/p' apisix*.log The total number is far less than the peak.
Dapr/internal service calls ： internal
dns,例如：dapr-service-vehicle-realtime-data.dapr-application.svc.cluster.local:xxx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: Apisix promethues metrics service request much greater than the actual request [apisix]
Baoyuantop commented on issue #12449: URL: https://github.com/apache/apisix/issues/12449#issuecomment-4600122143 Thanks for the additional information @wangchao732 . I checked the current issue details and the APISIX Prometheus plugin code path. `apisix_http_status` is a counter in APISIX, and the plugin increments it once in the HTTP log phase for the matched route/service/status. The screenshots show that the Grafana `sum(rate(...))` query has an abnormal spike, but they are not enough to determine whether APISIX is over-counting, Prometheus/Grafana is producing a scrape/rate artifact, or some internal calls are still going through APISIX. Could you please provide the following details? 1. The full route/service/upstream configuration for the service or route that shows the spike, not only the global rule. 2. The number of APISIX pods/instances, the Prometheus scrape interval, the actual `$__rate_interval` value in Grafana, and whether APISIX pods restarted or scaled during the spike window. 3. Raw `apisix_http_status` samples around the spike, preferably grouped by `instance`, `route`, `service`, and `code`, instead of only the aggregated `sum(rate(...))` graph. 4. The APISIX access log count for the same time window and how that count was calculated. 5. Whether the Dapr/internal service calls from the upstream application also pass through APISIX. With those details, we can continue verifying this on APISIX 3.11 and determine whether this is an APISIX counting issue or a monitoring aggregation/scrape issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: apisix-ingress-controller continuously attempts to delete non-existent global_rule [apisix-ingress-controller]
dovics opened a new issue, #2778:
URL: https://github.com/apache/apisix-ingress-controller/issues/2778
### Current Behavior
The apisix-ingress-controller continuously attempts to DELETE a global_rule
resource with ID `prometheus`, but this resource does not exist in APISIX. This
results in a 404 "Key not found" error on every sync cycle (approximately every
60 seconds), causing persistent error logs.
**This bug persists even after restarting the apisix-ingress-controller**,
indicating the stale reference is stored in persistent storage, not in-memory
state.
### Expected Behavior
The controller should either:
- Only manage resources created through ApisixGlobalRule CRD, ignoring
pre-existing resources in APISIX, OR
- Handle the 404 response gracefully when deleting a resource that no longer
exists (idempotent DELETE)
### Error Logs
```
2026-06-02T06:49:29.563ZINFOprovider.client client/client.go:201
syncing resources for config{"service_number": 0}
2026-06-02T06:49:29.914ZERROR provider.executor
client/executor.go:328 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":1,"success_count":0,"failed_count":1,"success":[],"failed":[{"event":{"resourceType":"global_rule","type":"delete","resourceId":"prometheus","resourceName":"prometheus"},"failed_at":"2026-06-02T06:49:29Z","synced_at":"0001-01-01T00:00:00Z","reason":"DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}","response":{"status":404,"headers":{"access-control-allow-credentials":"true","access-control-allow-origin":"*","access-control-expose-headers":"*","access-control-max-age":"3600","connection":"keep-alive","content-type":"application/json","date":"Tue,
02 Jun 2026 06:49:29
GMT","server":"APISIX/3.14.1","transfer-encoding":"chunked","x-api-version":"v3"}}}]},
"error": "ADC Server sync failed: DELETE http://apisix-
admin.infra.svc:9180/apisix/admin/global_rules/prometheus, responded with
status 404 Not Found, response body: {\"message\":\"Key not found\"}"}
2026-06-02T06:49:29.914ZERROR provider.executor
client/executor.go:142 failed to run http sync for server {"server":
"http://apisix-admin.infra.svc:9180";, "error": "ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}"}
2026-06-02T06:49:29.914ZERROR provider.client client/client.go:269
failed to execute adc command {"config":
{"name":"GatewayProxy/infra/apisix-ingress-controller-config","serverAddrs":["http://apisix-admin.infra.svc:9180"],"tlsVerify":false},
"error": "ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]"}
2026-06-02T06:49:29.914ZERROR provider.client client/client.go:210
failed to sync resources{"name":
"GatewayProxy/infra/apisix-ingress-controller-config", "error": "ADC execution
errors: [ADC execution error for
GatewayProxy/infra/apisix-ingress-controller-config: [ServerAddr:
http://apisix-admin.infra.svc:9180, Err: DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
found\"}]]"}
2026-06-02T06:49:29.914ZERROR providerapisix/status.go:321
failed to get resource label{"configName":
"GatewayProxy/infra/apisix-ingress-controller-config", "resourceType":
"global_rule", "id": "prometheus", "error": "not found"}
2026-06-02T06:49:29.914ZERROR provider
apisix/provider.go:282 failed to sync {"error": "failed to sync 1 configs:
GatewayProxy/infra/apisix-ingress-controller-config"}
2026-06-02T06:50:29.562ZINFOprovider.client client/client.go:177
syncing all resources
2026-06-02T06:50:29.562ZINFOprovider.client client/client.go:201
syncing resources for config{"service_number": 0}
2026-06-02T06:50:29.914ZERROR provider.executor
client/executor.go:328 ADC Server sync failed {"result":
{"status":"all_failed","total_resources":1,"success_count":0,"failed_count":1,"success":[],"failed":[{"event":{"resourceType":"global_rule","type":"delete","resourceId":"prometheus","resourceName":"prometheus"},"failed_at":"2026-06-02T06:50:29Z","synced_at":"0001-01-01T00:00:00Z","reason":"DELETE
http://apisix-admin.infra.svc:9180/apisix/admin/global_rules/prometheus,
responded with status 404 Not Found, response body: {\"message\":\"Key not
Re: [I] bug: Apisix promethues metrics service request much greater than the actual request [apisix]
wangchao732 commented on issue #12449:
URL: https://github.com/apache/apisix/issues/12449#issuecomment-4495348383
The same issue has occurred again recently.
https://github.com/user-attachments/assets/7582b532-5258-482f-b721-354f09db7c31";
/>
https://github.com/user-attachments/assets/99fb8d1d-3fcc-4b6a-a4ac-002fa1f55b47";
/>
It is clearly observable that the metric
`sum(rate(apisix_http_status{service=~"$service",route=~"$route",instance=~"$instance"}[$__rate_interval]))
by (service)` spikes instantaneously to 2–3 million. Through analysis of
network packets, bandwidth, and boundary metrics, everything appears normal,
with no evidence of a sudden surge in requests. Therefore, I suspect there may
be certain issues within APISIX itself, such as:
apisix_http_status should theoretically exhibit steady growth; however, the
sharp increase followed by a sharp decrease clearly indicates an issue
originating from APISIX itself.
https://github.com/user-attachments/assets/1cc0715e-ec5e-4702-917b-98493122dc78";
/>
@SkyeYoung @Baoyuantop I need help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
[I] bug: Apisix promethues metrics service request much greater than the actual request [apisix]
wangchao732 opened a new issue, #12449:
URL: https://github.com/apache/apisix/issues/12449
### Current Behavior
```
sum(rate(apisix_http_status{service=~"$service",route=~"$route",instance=~"$instance"}[$__rate_interval]))
by (service)
sum(rate(apisix_http_status{service=~"$service",route=~"$route",instance=~"$instance"}[$__rate_interval]))
by (route)
```
view http://dev-minio.bcnyyun.com/download/apisix-dashbord.png
According to the log statistics, the number of interface calls is less than
1,000.
### Expected Behavior
null
### Error Logs
null
### Steps to Reproduce
null
### Environment
- APISIX version 3.11
- Operating system: Centos 7.9.x linux
- Apisix install by helm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX Ingress Controller v2.0.0 - Service Inline Upstreams Not Updated on Endpoint Changes [apisix-ingress-controller]
github-actions[bot] commented on issue #2689: URL: https://github.com/apache/apisix-ingress-controller/issues/2689#issuecomment-4456104150 This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
vortegatorres commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4435006610 Following up on your earlier suggestion (@AlinsRan): > Using health check ups can alleviate this issue. We enabled active health checks on the affected upstream about 3 weeks ago. The same underlying behavior described in this issue persists, and the health check has introduced an additional failure mode. ## Environment APISIX Helm chart 2.13.0, standalone mode, backends listening on port 8080. ## Example Example of the situation that keeps happening with` IP-A`: - APISIX access logs show `IP-A:8080` serving successful responses (status 200) up to a specific minute. - Shortly after, the IP stopped appearing in any backend workload's logs. - The Pod was terminated as part of the rolling deploy. - APISIX continued active health-checking `IP-A:8080` for approximately 85 minutes after the Pod no longer existed. Any update about this issue? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix 3.16.0 comprehensive tracing breaks with HTTPS keepalive connections [apisix]
nic-6443 closed issue #13200: bug: apisix 3.16.0 comprehensive tracing breaks with HTTPS keepalive connections URL: https://github.com/apache/apisix/issues/13200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
vortegatorres commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4263259344 Many thanks for sharing your findings, @jotasixto. In our case, that does not appear to be the issue, since we are using port 8080 in all of the cases mentioned. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix 3.16.0 comprehensive tracing breaks with HTTPS keepalive connections [apisix]
janiussyafiq commented on issue #13200:
URL: https://github.com/apache/apisix/issues/13200#issuecomment-4240824069
Hi @jens-skribble, thanks for reporting the bug.
I was able to reproduce the bug in both scenarios:
1. Using provided reproduce.sh script.
2. MRE from local APISIX (traditional mode)
a. Configure `config.yaml`
```
apisix:
tracing: true
ssl:
enable: true
listen:
- port: 9443
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_key:
- name: admin
key:
role: admin
```
b. Run `make run`
c. Configure SSL cert and key (can generate thru openssl, i just copied
the one provided in reproduce.zip)
```
curl http://127.0.0.1:9180/apisix/admin/ssls/1 -H 'X-API-KEY: ' \
-X PUT -d "{
\"snis\": [\"localhost\"],
\"cert\": \"$(cat /path/to/server.crt)\",
\"key\": \"$(cat /path/to/server.key)\"
}"
```
d. Create route
```
curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: ' \
-X PUT -d '{
"uri": "/*",
"upstream": {
"type": "roundrobin",
"nodes": {"httpbin.org:80": 1}
}
}'
```
e. Trigger the crash (two requests on same TLS connection):
```
curl -sk \
-o /dev/null -w "%{http_code} " https://localhost:9443/get \
-o /dev/null -w "%{http_code}" https://localhost:9443/get
```
f. Check `error.log`
```
2026/04/13 17:23:55 [error] 179311#179311: *45028 lua entry thread aborted:
runtime error: /home/janiussyafiq/GitHub/apisix/apisix/utils/span.lua:62: bad
argument #1 to 'insert' (table expected, got nil)
stack traceback:
coroutine 0:
[C]: in function 'insert'
/home/janiussyafiq/GitHub/apisix/apisix/utils/span.lua:62: in function
'new'
/home/janiussyafiq/GitHub/apisix/apisix/tracer.lua:53: in function
'start'
/home/janiussyafiq/GitHub/apisix/apisix/init.lua:698: in function
'http_access_phase'
access_by_lua(nginx.conf:370):2: in main chunk, client: 127.0.0.1,
server: _, request: "GET /get HTTP/2.0", host: "localhost:9443", request_id:
"ebf545017724304f723182a819c476ec"
2026/04/13 17:23:55 [error] 179311#179311: *45028 failed to run
header_filter_by_lua*:
/home/janiussyafiq/GitHub/apisix/apisix/utils/span.lua:62: bad argument #1 to
'insert' (table expected, got nil)
stack traceback:
[C]: in function 'insert'
/home/janiussyafiq/GitHub/apisix/apisix/utils/span.lua:62: in function
'new'
/home/janiussyafiq/GitHub/apisix/apisix/tracer.lua:53: in function
'start'
/home/janiussyafiq/GitHub/apisix/apisix/init.lua:904: in function
'http_header_filter_phase'
header_filter_by_lua(nginx.conf:414):2: in main chunk, client:
127.0.0.1, server: _, request: "GET /get HTTP/2.0", host: "localhost:9443",
request_id: "ebf545017724304f723182a819c476ec"
2026/04/13 17:23:55 [error] 179311#179311: *45028 failed to run log_by_lua*:
/home/janiussyafiq/GitHub/apisix/apisix/tracer.lua:80: bad argument #1 to
'ipairs' (table expected, got nil)
stack traceback:
[C]: in function 'ipairs'
/home/janiussyafiq/GitHub/apisix/apisix/tracer.lua:80: in function
'release'
/home/janiussyafiq/GitHub/apisix/apisix/init.lua:: in function
'http_log_phase'
log_by_lua(nginx.conf:422):2: in main chunk while logging request,
client: 127.0.0.1, server: _, request: "GET /get HTTP/2.0", host:
"localhost:9443", request_id: "ebf545017724304f723182a819c476ec"
```
However the proposed solution didn't seem to fix the issue in my case. I
still get the same errors.
```
tablepool.release("tracing", tracing)
ctx.tracing = nil -- add this
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress [apisix-ingress-controller]
Baoyuantop closed issue #2730: bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress URL: https://github.com/apache/apisix-ingress-controller/issues/2730 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: apisix 3.16.0 comprehensive tracing breaks with HTTPS keepalive connections [apisix]
jens-skribble opened a new issue, #13200:
URL: https://github.com/apache/apisix/issues/13200
### Current Behavior
After upgrading to apisix 3.16.0 in our staging environment and activating
the new [comprehensive tracing
feature](https://github.com/apache/apisix/pull/12686), by setting
`apisix.tracing: true` in `config.yaml` apisix shows `500` errors on HTTPS
connections instead of serving the correct pages.
We are using apisix in api driven mode in k8s with tls termination within
apisix behind our cloud providers load balancer with proxy protocol. Hence the
HTTPS connections are kept open. The bug however is easier to reproduce without
any plugins as described below.
---
Claude Code helped me with the local reproduction of the bug and already
created a root cause analysis and a proposed fix that I'm not able to verify on
my own. Here is its output.
Use it at your own discretion:
`tracer.release()` in `apisix/tracer.lua` returns the tracing table to a
memory pool (which zeroes all its fields, including spans) but never sets
`ctx.tracing = nil`:
```lua
function _M.release(ctx)
local tracing = ctx.tracing
if not tracing then return end
for _, sp in ipairs(tracing.spans) do sp:release() end
tablepool.release("tracing_spans", tracing.spans) -- zeroes
tracing.spans
tablepool.release("tracing", tracing) -- zeroes entire
table
-- BUG: ctx.tracing is not set to nil
end
```
`tracer.start()` guards initialisation with if not tracing then — which
evaluates to false for a stale non-nil pointer — so it skips setup and
immediately crashes at `table.insert(tracing.spans, self) (span.lua:62)`
because spans is nil.
The trigger is that `ngx.ctx` is shared across all HTTP requests on the same
TLS keepalive connection in OpenResty. APISIX calls `tracer.start(ngx_ctx)` in
`ssl_client_hello_phase()` (confirmed by `ngx.ctx.matched_ssl` and
`ngx.ctx.client_hello_sni` being written there and read later in
`http_access_phase` without repopulation). After the first HTTP request's
http_log_phase calls `tracer.release()`, `ngx_ctx.tracing` is left stale. The
second HTTP request on the same connection inherits it and crashes at
`init.lua:693`.
Plain HTTP is not affected because there is no SSL handshake phase — tracing
is initialised fresh inside http_access_phase on every request and the
stale-pointer path is never reached.
Fix — one line in `tracer.release()`:
```lua
tablepool.release("tracing", tracing)
Re: [I] bug: apisix-ingress-controller cannot close the gateway API [apisix-ingress-controller]
Baoyuantop commented on issue #2744: URL: https://github.com/apache/apisix-ingress-controller/issues/2744#issuecomment-4221253175 fixed in https://github.com/apache/apisix-ingress-controller/pull/2672 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller cannot close the gateway API [apisix-ingress-controller]
Baoyuantop closed issue #2744: bug: apisix-ingress-controller cannot close the gateway API URL: https://github.com/apache/apisix-ingress-controller/issues/2744 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: apisix-ingress-controller cannot close the gateway API [apisix-ingress-controller]
FIREcup opened a new issue, #2744:
URL: https://github.com/apache/apisix-ingress-controller/issues/2744
### Current Behavior
I deployed Apisix using Helm, and the GKE cluster doesn't have GRPCRoute and
other CRD resources installed.
But I close the gateway api:
```
ingress-controller:
config:
disableGatewayAPI: true
```
Ingress-controller startup is throwing the following error:
`2026-04-09T09:43:37.502Z ERROR
controller-runtime.controller-runtime.source.EventHandler source/kind.go:71 if
kind is a CRD, it should be installed before calling Start {"kind":
"GRPCRoute.gateway.networking.k8s.io", "error": "no matches for kind
\"GRPCRoute\" in version \"gateway.networking.k8s.io/v1\""}`
### Expected Behavior
_No response_
### Error Logs
_No response_
### Steps to Reproduce
1. APISIX version: 3.15.0
2. apisix-ingress-controller version: 2.0.1
### Environment
1. APISIX version: 3.15.0
2. apisix-ingress-controller version: 2.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
jotasixto commented on issue #2708: URL: https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-4218338000 We experienced a very similar issue in our environment and wanted to share our findings, as the root cause in our case turned out to be related to **EKS security group rules blocking low-port traffic between worker nodes**, rather than a bug in APISIX or the ingress controller itself. ## Our Environment - **EKS cluster** deployed via the official [terraform-aws-eks module](https://github.com/terraform-aws-modules/terraform-aws-eks) - **APISIX standalone** deployed with Helm chart **v2.13.0** - Backend services exposing **port 80** on their pods (front-end Nginx containers with `containerPort: 80`) ## Observed Behavior We saw the same symptoms described in this issue: after pod rescheduling or scaling events, APISIX gateways would intermittently fail to reach backend pods with `(111: Connection refused)` errors, particularly when pods were placed on different nodes than the APISIX gateway pods. When we attempted a workaround of ensuring at least one replica of each backend service ran on every node, we discovered the actual underlying problem: **traffic on port 80 was being blocked between worker nodes by the EKS node security group rules**. ## Root Cause: EKS Security Group Default Rules and Low Ports The [official terraform-aws-eks documentation on network connectivity](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/network_connectivity.md) explains that the default node security group rules only allow traffic on **ephemeral ports (1025-65535)** between the cluster control plane and worker nodes, and between nodes themselves. This is by design — AWS considers it a best practice because **non-privileged pods should not bind to ports below 1024**. Looking at the [security group diagram](https://raw.githubusercontent.com/terraform-aws-modules/terraform-aws-eks/master/.github/images/security_groups.svg) from the module documentation, port 80 is simply not in the allowed range for node-to-node or cluster-to-node ingress traffic. This means: - When an APISIX gateway pod on **Node A** tries to reach a backend pod on **Node B** using port 80, the traffic is **silently dropped** by the node security group. - When both APISIX and the backend pod happen to be on the **same node**, traffic works fine (it stays within the node and doesn't cross the security group boundary). - This creates the **intermittent** behavior: it works or fails depending on pod placement, which changes with scaling events, node rotation, etc. ## Our Fix We added custom security group rules to explicitly allow port 80 traffic between worker nodes ([Automya/claims#185](https://github.com/Automya/claims/pull/185)): ```yaml node_security_group_additional_rules: ingress_node_ports_fronts: description : "Allow port 80 from cluster to worker nodes" protocol : "tcp" from_port : 80 to_port : 80 type : "ingress" source_cluster_security_group : true ingress_node_ports_fronts_self: description : "Allow port 80 between worker nodes" protocol : "tcp" from_port : 80 to_port : 80 type : "ingress" self : true ``` After applying these rules, the issue was **fully resolved** — APISIX gateways could reach backend pods on any node in the cluster without `Connection refused` errors, regardless of pod placement. ## Recommendation If you're running on EKS (especially with the terraform-aws-eks module) and your backend services expose pods on **port 80 or any port below 1024**, check your node security group rules. The default rules only allow ephemeral ports (1025-65535), and traffic to low ports between nodes will be silently dropped. The proper long-term fix is to **migrate backend services to listen on high ports (≥1024)**, which aligns with AWS best practices for non-privileged pods. The custom security group rules above are a valid workaround if migrating ports immediately is not feasible. **TL;DR**: In our case, APISIX was working correctly — it was the EKS node security groups blocking cross-node traffic on port 80 that caused the intermittent `Connection refused` errors after pod rescheduling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at:
Re: [I] bug: apisix-ingress-controller cannot close the gateway API [apisix]
FIREcup commented on issue #13193:
URL: https://github.com/apache/apisix/issues/13193#issuecomment-4213340741
The AI analysis suggests that the GRPCRoute resource appears to be not
being processed correctly：
GatewayReconciler unconditionally watches GRPCRoute.
File:
`[internal/controller/gateway_controller.go](https://github.com/apache/apisix-ingress-controller/blob/2.0.1/internal/controller/gateway_controller.go)`
The SetupWithManager() function contains this:
```
bdr := ctrl.NewControllerManagedBy(mgr).
For(&gatewayv1.Gateway{}, ...).
Watches(&gatewayv1.GatewayClass{}, ...).
Watches(&gatewayv1.HTTPRoute{}, ...).
Watches(&gatewayv1.GRPCRoute{}, ...).
Watches(&v1alpha1.GatewayProxy{}, ...).
Watches(&corev1.Secret{}, ...)
```
Here, both HTTPRoute and GRPCRoute are unconditionally watched (...).
The existence checks are only performed on the following:
```
if pkgutils.HasAPIResource(mgr, &gatewayv1alpha2.TCPRoute{}) { ... }
if pkgutils.HasAPIResource(mgr, &gatewayv1alpha2.TLSRoute{}) { ... }
if pkgutils.HasAPIResource(mgr, &gatewayv1alpha2.UDPRoute{}) { ... }
```
In other words:
- TCPRoute / TLSRoute / UDPRoute are protected
- GRPCRoute is not protected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
[I] bug: apisix-ingress-controller cannot close the gateway API [apisix]
FIREcup opened a new issue, #13193:
URL: https://github.com/apache/apisix/issues/13193
### Current Behavior
I deployed Apisix using Helm, and the GKE cluster doesn't have GRPCRoute and
other CRD resources installed.
But I close the gateway api:
```
ingress-controller:
config:
disableGatewayAPI: true
```
Ingress-controller startup is throwing the following error:
`2026-04-09T09:43:37.502Z ERROR
controller-runtime.controller-runtime.source.EventHandler source/kind.go:71 if
kind is a CRD, it should be installed before calling Start {"kind":
"GRPCRoute.gateway.networking.k8s.io", "error": "no matches for kind
\"GRPCRoute\" in version \"gateway.networking.k8s.io/v1\""}`
### Expected Behavior
_No response_
### Error Logs
_No response_
### Steps to Reproduce
1. APISIX version: 3.15.0
2. apisix-ingress-controller version: 2.0.1
### Environment
1. APISIX version: 3.15.0
2. apisix-ingress-controller version: 2.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
janiussyafiq commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4212206412 another approach for this problem would be: ``` 1. config.yaml has explicit non-empty key? → use it, done 2. sidecar file exists? → read key from it, done 3. neither? → generate key write to sidecar file (not rewrite to config) ``` essentially sidecar file is a file that would get generated for the sole purpose of putting the auto-generated admin key, hence prevent config.yaml from getting rewritten whenever admin key is empty. More input from community would be greatly appreciated! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [bug] apisix helm charts README.md some config params is wrong [apisix-helm-chart]
Baoyuantop closed issue #258: [bug] apisix helm charts README.md some config params is wrong URL: https://github.com/apache/apisix-helm-chart/issues/258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] [bug] apisix helm charts README.md some config params is wrong [apisix-helm-chart]
Baoyuantop commented on issue #258: URL: https://github.com/apache/apisix-helm-chart/issues/258#issuecomment-4205176475 The README documentation referenced in this issue has been rewritten and updated multiple times. The old commit hash references no longer exist, and the parameter table has been regenerated. Closing as stale. If you still find documentation issues, please feel free to open a new issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix control service is not installed on k8s by helm chart [apisix-helm-chart]
Baoyuantop closed issue #474: bug: apisix control service is not installed on k8s by helm chart URL: https://github.com/apache/apisix-helm-chart/issues/474 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix control service is not installed on k8s by helm chart [apisix-helm-chart]
Baoyuantop commented on issue #474: URL: https://github.com/apache/apisix-helm-chart/issues/474#issuecomment-4205053525 The current chart supports the Control API service configuration (`control.enabled: true`, port 9090) with a dedicated `service-control.yaml` template. Closing as the feature is available. If you still encounter issues with this, please feel free to reopen. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 tries to connect to etcd in standalone api-driven mode (non crashing) [apisix]
shreemaan-abhishek closed issue #12989: bug: Apisix 3.15.0 tries to connect to etcd in standalone api-driven mode (non crashing) URL: https://github.com/apache/apisix/issues/12989 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.0+ OpenID Connect redirect drops request port (regression vs 3.13.0) [apisix]
shreemaan-abhishek closed issue #12970: bug: APISIX 3.14.0+ OpenID Connect redirect drops request port (regression vs 3.13.0) URL: https://github.com/apache/apisix/issues/12970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX incorrectly attempts to decompress uncompressed response bodies in log plugins [apisix]
Baoyuantop closed issue #13093: bug: APISIX incorrectly attempts to decompress uncompressed response bodies in log plugins URL: https://github.com/apache/apisix/issues/13093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. [apisix]
rg2011 commented on issue #12973: URL: https://github.com/apache/apisix/issues/12973#issuecomment-4163954389 thanks @Baoyuantop , I submitted https://github.com/apache/apisix/pull/13137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. [apisix]
Baoyuantop commented on issue #12973: URL: https://github.com/apache/apisix/issues/12973#issuecomment-4162959917 Of course @rg2011 , if you find the cause of the problem and can fix it, feel free to submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. [apisix]
rg2011 commented on issue #12973: URL: https://github.com/apache/apisix/issues/12973#issuecomment-4161026502 Hi @Baoyuantop , I was affected by this bug too and I am currently using a minimally patched version of `utils/upstream.lua`, to fix it. I can make an unit test too. Would you accept a PR for this? My current patch: ```diff --- instrumentation/base-image/apisix/utils/upstream.lua +++ instrumentation/fix-src/apisix/utils/upstream.lua @@ -100,8 +100,10 @@ if not new_nodes then return nil, err end +local new_nodes_empty = #new_nodes == 0 +local current_nodes_empty = not up.value.nodes or #up.value.nodes == 0 local ok = compare_upstream_node(up.value, new_nodes) -if ok then +if ok and (not current_nodes_empty or new_nodes_empty) then return up end ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
janiussyafiq commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4139889432 https://github.com/apache/apisix/pull/13091#discussion_r2951315476 - reference for those working with this issue moving forward -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 tries to connect to etcd in standalone api-driven mode (non crashing) [apisix]
vortegatorres commented on issue #12989: URL: https://github.com/apache/apisix/issues/12989#issuecomment-4134202056 Same issue for us using APISIX 3.15 and standalone mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 needs anyuid [apisix]
Baoyuantop commented on issue #13075: URL: https://github.com/apache/apisix/issues/13075#issuecomment-4116906419 Thank you for your contribution. I left some comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 needs anyuid [apisix]
sebgott commented on issue #13075: URL: https://github.com/apache/apisix/issues/13075#issuecomment-4116268969 I created PR to fix this since the one you referred to seemed abandoned, if you want to accept it https://github.com/apache/apisix-docker/pull/617 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: APISIX incorrectly attempts to decompress uncompressed response bodies in log plugins [apisix]
hachi029 opened a new issue, #13093:
URL: https://github.com/apache/apisix/issues/13093
### Current Behavior
When using a logging plugin to record the response body, APISIX attempts to
decompress uncompressed data, causing an warning messages in the error.log
### Expected Behavior
APISIX only decompress compressed response bodies.
### Error Logs
2026/03/17 09:17:47 [warn] 8028#8028: *41269 [lua] log-util.lua:417:
collect_body(): try decode compressed data err: inflate gzip err: INFLATE: data
error while sending to client, client: 127.0.0.1, server: _, request: "GET
/file-logger HTTP/1.1", upstream: "http://127.0.0.1:9001/file-logger";, host:
"127.0.0.1:9080", request_id: "68725bcdd1aff0fad2f7a907daa9ac06"
### Steps to Reproduce
1. Run apisix with following router conf:
```json
{
"uri": "/file-logger",
"upstream": {
"nodes": {
"127.0.0.1:9001": 1
},
"type": "roundrobin"
},
"id": "1"
"plugins": {
"file-logger": {
"path": "logs/file.log",
"include_resp_body":true
},
"gzip": {
"types": ["application/json"],
"min_length": 10,
"compression_level": 5
}
}
}
```
2. Start another nginx server 127.0.0.1:9001 with following conf :
```nginx
location /file-logger {
gzip off;
content_by_lua_block {
ngx.header.content_type = "application/json; charset=utf-8"
ngx.say('{"code":"hello 123", "status":200}')
}
}
```
3. Send request to APISIX:
```bash
curl -H 'Accept-Encoding: gzip' http://127.0.0.1:9080/file-logger
```
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
Baoyuantop commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4072845772 Hi @janiussyafiq, I have two questions regarding the current proposal: 1. We should not write automatically generated keys to the logs; any action that involves writing keys to the logs should be avoided. 2. `admin_key` is an array, and theoretically, there may be multiple entries with empty keys. Please ensure that all of them can be matched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
janiussyafiq commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4072688034 Hi @Baoyuantop, here's my proposed fix for this issue: Approach: 1. Remove `generate_yaml()` entirely — this eliminates any possibility of config.yaml being rewritten destructively 2. Keep the auto-generated key behaviour (agree with @fekitibi that the key generation security could be improved, but that can be a separate discussion) - When admin_key is empty, a warning log is emitted notifying the user that a key has been auto-generated, along with the key value - Instead of rewriting the entire file, only the key field is written back via targeted string substitution — all comments, formatting, and structure in config.yaml are fully preserved Impact on existing tests: Some test cases rely on reading the generated key from config.yaml after startup: https://github.com/apache/apisix/blob/4990927937280037602e81bb1b9554a784afa076/t/cli/test_admin.sh#L78 Since the targeted substitution only works when `key: ''` is explicitly present in the raw file, test cases that write a minimal config without the admin_key block (relying on merged defaults to supply it) will need to be updated to explicitly include the admin_key block so the substitution can find and replace it. E.g. https://github.com/apache/apisix/blob/4990927937280037602e81bb1b9554a784afa076/t/cli/test_admin.sh#L26-L35 If this approach looks good, I will proceed with the full implementation and test updates in this PR #13091 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress [apisix-ingress-controller]
Hemanth1205 commented on issue #2730: URL: https://github.com/apache/apisix-ingress-controller/issues/2730#issuecomment-4066563450 @Baoyuantop , Please go through the attached MR #2732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX rewrites yaml configuration file to cause restarting errors [apisix]
janiussyafiq commented on issue #12170: URL: https://github.com/apache/apisix/issues/12170#issuecomment-4052249800 I would like to work on this issue @Baoyuantop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix forwarded traffic to the wrong enpodints in gRPC mode. [apisix]
Baoyuantop closed issue #12792: bug: Apisix forwarded traffic to the wrong enpodints in gRPC mode. URL: https://github.com/apache/apisix/issues/12792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix forwarded traffic to the wrong enpodints in gRPC mode. [apisix]
Baoyuantop commented on issue #12792: URL: https://github.com/apache/apisix/issues/12792#issuecomment-4037526768 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress [apisix-ingress-controller]
Hemanth1205 commented on issue #2730: URL: https://github.com/apache/apisix-ingress-controller/issues/2730#issuecomment-4023813926 @Baoyuantop , sure. Currently working on the change. Will submit a PR soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress [apisix-ingress-controller]
Baoyuantop commented on issue #2730: URL: https://github.com/apache/apisix-ingress-controller/issues/2730#issuecomment-4023481394 Hi @Hemanth1205, welcome to submit a PR to fix this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX Core Cookie Parser Fails with Spaces or Quoted Values in Cookie Header [apisix]
Baoyuantop closed issue #12452: bug: APISIX Core Cookie Parser Fails with Spaces or Quoted Values in Cookie Header URL: https://github.com/apache/apisix/issues/12452 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX Core Cookie Parser Fails with Spaces or Quoted Values in Cookie Header [apisix]
Baoyuantop commented on issue #12452: URL: https://github.com/apache/apisix/issues/12452#issuecomment-4021749189 Closing this due to inactivity. If there’s still interest in this feature, please let us know! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: apisix-ingress-controller not supporting Hostnames when specified in gatewayproxy.spec.statusAddress [apisix-ingress-controller]
Hemanth1205 opened a new issue, #2730: URL: https://github.com/apache/apisix-ingress-controller/issues/2730 ### Current Behavior I'm trying to setup APISIX 3.15 in standalone api-driven modes (suitable for Ingress controller as per docs) using helm in AWS EKS. When deploying in EKS, we are deploying the apisix-gateway service as CLusterIP and enabling Ingress to create ALB in front of APISIX deployments. As per the docs, we to use `statusAddress` or `publishService` to make the ingress controller attach the address to the ingress/Gateway objects referencing the ingress class of type `apisix` . But when we pass the ingress hostname we got from cloud providers, it's throwing an error saying provided statusAddress is not valid IPAddress. ### Expected Behavior When deploying APISIX in cloud environments Case 1: PublishService Using ClusterIP service with Ingress enabled and `PublishService` flag is passed with a service name, We should support both `IPAddress` and `Hostname` Types and populate the `status.ingress.hostname` or `ingress.status.ip` based on the values from Ingress that's referencing the service provided and fetch the `status.ingress.hostname/ip` . Case 2: StatusAddress When `gatewayproxy.spec.statusAddress` is provided, check if the address is `ip` or `hostname` and update the gateway/ingress resources depending on the type of address provided. ### Error Logs provided statusAddress is not valid IPAddress. ### Steps to Reproduce 1. Install APISIX in standalone -apidriven mode with ingress controller enabled. 2. Enable ingress for APISIX and make the apisix-gateway service as ClusterIP 3. Pass the hostname returned in the status,ingress.hostname gatewayproxy.spec.statusAddress 4. look in the apisix-ingress-controller pods for the error ### Environment - APISIX Ingress controller version : 2.0.1 - Kubernetes cluster version : v1.34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 tries to connect to etcd in standalone api-driven mode (non crashing) [apisix]
pbaranow commented on issue #12989: URL: https://github.com/apache/apisix/issues/12989#issuecomment-4011004684 I can confirm this issue happens with `ApisixGlobalRule` objects configured. I found following log in `apisix` pod logs at the time where `apisix-ingress-controller` complained about issue with synchronizing the configuration: ``` [lua] health_check.lua:114: report_failure(): update endpoint: http://127.0.0.1:2379 to unhealthy, client: 10.103.12.70, server: , request: "PUT /apisix/admin/configs HTTP/1.1 ``` It might come from `reload_plugins` which calls `sync_local_conf_to_etcd` unconditionally: https://github.com/apache/apisix/blob/3.15.0/apisix/admin/init.lua#L375-L382 as opposed to `_init`, where there's a check if apisix is configured to run in `standalone` mode: https://github.com/apache/apisix/blob/3.15.0/apisix/admin/init.lua#L506-L508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix route will not sync [apisix]
Baoyuantop commented on issue #12283: URL: https://github.com/apache/apisix/issues/12283#issuecomment-4010919504 This issue has been resolved in the new version of the ingress controller. Please refer to https://apisix.apache.org/docs/ingress-controller/upgrade-guide/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX keeps sending requests to old pod IP after it changes in Kubernetes [apisix]
Baoyuantop commented on issue #12294: URL: https://github.com/apache/apisix/issues/12294#issuecomment-4010917746 This issue has been resolved in the new version of the ingress controller. Please refer to https://apisix.apache.org/docs/ingress-controller/upgrade-guide/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix route will not sync [apisix]
Baoyuantop closed issue #12283: bug: apisix route will not sync URL: https://github.com/apache/apisix/issues/12283 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX keeps sending requests to old pod IP after it changes in Kubernetes [apisix]
Baoyuantop closed issue #12294: bug: APISIX keeps sending requests to old pod IP after it changes in Kubernetes URL: https://github.com/apache/apisix/issues/12294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: Apisix 3.15.0 needs anyuid [apisix]
Baoyuantop commented on issue #13075: URL: https://github.com/apache/apisix/issues/13075#issuecomment-4008516591 The APISIX release image was pushed by another project and appears to require completion: https://github.com/apache/apisix-docker/pull/612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: Apisix 3.15.0 needs anyuid [apisix]
sebgott opened a new issue, #13075: URL: https://github.com/apache/apisix/issues/13075 ### Current Behavior In the APISIX 3.15.0 release notes it says "Adjust directory permissions to allow APISIX to run on OpenShift without the anyuid command (PR [#12824](https://github.com/apache/apisix/pull/12824))". But when using the 3.15.0 image, APISIX still needs anyuid perms as it gets blocked from running ```ln -s /apisix-config/apisix.yaml /usr/local/apisix/conf/apisix.yaml``` for example. When using the dev image however, APISIX is able to start. ### Expected Behavior APISIX should be able to start on Openshift without anyuid permissions on version 3.15.0 ### Error Logs _No response_ ### Steps to Reproduce 1. Deploy APISIX on OpenShift with tag "3.15.0-ubuntu/debian/redhat" 2. Do not give anyuid permissions ### Environment - APISIX version (run `apisix version`): 3.15.0 - Operating system (run `uname -a`): - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix not allow request, when user-agent set to "Go-http-client/2.0" [apisix]
Baoyuantop commented on issue #11625: URL: https://github.com/apache/apisix/issues/11625#issuecomment-3995991580 fixed by https://github.com/apache/apisix/pull/11651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix not allow request, when user-agent set to "Go-http-client/2.0" [apisix]
Baoyuantop closed issue #11625: bug: apisix not allow request, when user-agent set to "Go-http-client/2.0" URL: https://github.com/apache/apisix/issues/11625 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix missing Transfer-Encoding header when client http body size is not zero [apisix]
Baoyuantop commented on issue #11703: URL: https://github.com/apache/apisix/issues/11703#issuecomment-3995819243 This is a design flaw in nginx, not an issue with APISIX. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: apisix missing Transfer-Encoding header when client http body size is not zero [apisix]
Baoyuantop closed issue #11703: bug: apisix missing Transfer-Encoding header when client http body size is not zero URL: https://github.com/apache/apisix/issues/11703 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX sends wrong redirect to browser when upstream is HTTP (SSL termination) [apisix]
Baoyuantop commented on issue #11408: URL: https://github.com/apache/apisix/issues/11408#issuecomment-3995217366 If there is still a problem, please open it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX sends wrong redirect to browser when upstream is HTTP (SSL termination) [apisix]
Baoyuantop closed issue #11408: bug: APISIX sends wrong redirect to browser when upstream is HTTP (SSL termination) URL: https://github.com/apache/apisix/issues/11408 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX sends wrong redirect to browser when upstream is HTTP (SSL termination) [apisix]
Baoyuantop commented on issue #11408:
URL: https://github.com/apache/apisix/issues/11408#issuecomment-3995210171
You can use the following plugin configuration to achieve similar
functionality.
```
{
plugins: {
response-rewrite: {
headers: {
set: {
Location: $regex_replace($http_location, '^http://', 'https://')
}
},
vars: [
[status, ~~, ^30[12378]$],
[http_location, ~~, ^http://]
]
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: Apisix should reject configurations with missing plugins in standalone mode [apisix]
shreemaan-abhishek closed issue #12189: bug: Apisix should reject configurations with missing plugins in standalone mode URL: https://github.com/apache/apisix/issues/12189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.1 After the DNS service recovered temporarily, Apisix returned a 503 error. The issue was resolved only after restarting APISIX or changing the service IP. [apisix]
Baoyuantop commented on issue #12973: URL: https://github.com/apache/apisix/issues/12973#issuecomment-3956311515 Hi @zbfzn, thank you for providing such a detailed and insightful bug report. From your description and the logs, it seems your diagnosis is correct. The issue likely lies in the upstream node comparison logic, where APISIX fails to update the `nodes` list after DNS resolution recovers. When the DNS service fails, the list of resolved IPs becomes empty. After the DNS service is restored, even though new IPs are resolved, the `compare_upstream_node` function might incorrectly determine that no changes have occurred because the original domain name configuration in `original_nodes` hasn't changed. This prevents the empty `nodes` list from being updated with the newly resolved, valid IP addresses, leading to the persistent 503 errors. We have marked this as a bug and will prioritize its investigation. Our team will follow your reproduction steps to verify the behavior and work on a fix based on your findings. The fix will likely involve adjusting the comparison logic to correctly identify a change when the resolved IPs are updated, especially when recovering from an empty state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
[I] bug: [apisix-java-plugin-runner]
escseszemely-cell opened a new issue, #327: URL: https://github.com/apache/apisix-java-plugin-runner/issues/327 ### Issue description ### Environment * your apisix-java-plugin-runner version ### Minimal test code / Steps to reproduce the issue 1. 2. 3. ### What's the actual result? (including assertion message & call stack if applicable) ### What's the expected result? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX 3.14.0+ OpenID Connect redirect drops request port (regression vs 3.13.0) [apisix]
czegi90 commented on issue #12970: URL: https://github.com/apache/apisix/issues/12970#issuecomment-3923190855 I can confirm this regression. My deployment worked correctly with 3.13.0 but broke after upgrading to 3.14.0+. The port is missing from the redirect URL - initiating the login flow at https://localhost:9443/login redirects the browser to https://localhost/oidc/login/.apisix/redirect?state=... (port 9443 dropped), making APISIX unreachable via the callback. Manually correcting the port in the address bar allows the auth flow to complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] bug: APISIX keeps stale (deleted) Pod IP in upstream after scale-down, causing `111: Connection refused` [apisix-ingress-controller]
AlinsRan commented on issue #2708:
URL:
https://github.com/apache/apisix-ingress-controller/issues/2708#issuecomment-3888337417
# Issue Investigation Checklist
## 1. Logs and Runtime Information
### 1.1 Complete Logs
Please provide the full logs during the time when the issue occurred.
Ingress Pod
- ingress container logs
- adc-server container logs
- If possible, please enable debug logging and reproduce the issue.
Debug configuration references:
- Ingress debug:
https://github.com/apache/apisix-helm-chart/blob/9d5ad2a28f7e75490e05f9dd977cd29791f13629/charts/apisix-ingress-controller/values.yaml#L80
- ADC debug:
https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix-ingress-controller/values.yaml#L77
APISIX Pod
- `error.log`
- `access.log` entries that include `/apisix/admin/configs` API calls
---
## 2. Restart Verification
Please confirm:
- Does restarting `ingress-apisix` resolve the issue?
This helps determine whether the problem originates from the ingress/ADC
side or the APISIX data plane.
---
## 3. Standalone API Configuration Comparison
When the issue occurs:
- Call the standalone API
- Retrieve the data plane configuration **before and after restart**
- Compare the configuration differences
Standalone API reference:
https://apisix.apache.org/zh/docs/apisix/deployment-modes/#example
This step is critical for root cause analysis.
---
## 4. Resource Limits and Usage
Please provide:
- Resource requests/limits for ingress pods
- Resource requests/limits for APISIX pods
- CPU and memory usage when the issue occurs
Insufficient resources may cause sync failures or abnormal behavior.
---
# 2. Configuration Conflict Investigation
## 2.1 GatewayProxy Conflict
Please check whether multiple `GatewayProxy` resources are pointing to the
same data plane.
You can run:
```bash
kubectl get gatewayproxy -A -o yaml
Multiple `GatewayProxy` resources sharing the same data plane may cause
configuration conflicts.
---
## 2.2 Multiple Ingress Controllers
Please confirm:
* Whether multiple `apisix-ingress-controller` instances are deployed in the
same Kubernetes cluster
* If yes, whether they are properly isolated (e.g., ingressClass, namespace
scope, etc.)
Improper isolation may also lead to configuration conflicts.
---
# 3. Investigation Goals
The goal is to determine:
* Whether the issue is on the ingress/ADC side or the APISIX data plane side
* Whether there is a configuration conflict
* Whether improper resource limits are causing synchronization failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] bug: Apisix 3.15.0 tries to connect to etcd in standalone api-driven mode (non crashing) [apisix]
Baoyuantop commented on issue #12989: URL: https://github.com/apache/apisix/issues/12989#issuecomment-3888290803 After adding the ApisixGlobalRule, I did indeed see the warning logs. ``` 2026/02/12 02:16:42 [warn] 50#50: *41990 [lua] health_check.lua:114: report_failure(): update endpoint: http://etcd.host:2379 to unhealthy, client: 192.168.194.39, server: , request: "PUT /apisix/admin/configs HTTP/1.1", host: "192.168.194.47:9180", request_id: "f77176a7e38b3d0a2359b04285d356fd" 2026/02/12 02:16:42 [warn] 50#50: *41990 [lua] v3.lua:247: _request_uri(): http://etcd.host:2379: timeout. Retrying, client: 192.168.194.39, server: , request: "PUT /apisix/admin/configs HTTP/1.1", host: "192.168.194.47:9180", request_id: "f77176a7e38b3d0a2359b04285d356fd" 192.168.194.39 - - [12/Feb/2026:02:16:42 +] 192.168.194.47:9180 "PUT /apisix/admin/configs HTTP/1.1" 202 5 30.034 "-" "axios/1.13.2" - - - "http://192.168.194.47:9180"; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
