shareinto opened a new issue, #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378
### Issue description
my k8s cluster has about 15k namespaces
```
k get ns | wc -l
14846
```
and log dispaly that every namespace stuck 10 seconds, so the provider start
took a long time
```
I1012 21:39:58.101271 1 request.go:665] Waited for 1.000058582s due to
client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-disagreement42/apisixroutes
[GIN] 2022/10/12 - 21:39:59 | 200 | 66.378µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:02 | 200 | 48.148µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:08.101818 1 request.go:665] Waited for 11.000514374s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-negligee64/apisixroutes
[GIN] 2022/10/12 - 21:40:09 | 200 | 37.35µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:12 | 200 | 26.873µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:18.301545 1 request.go:665] Waited for 21.20015681s due to
client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-earum75/apisixroutes
[GIN] 2022/10/12 - 21:40:19 | 200 | 24.12µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:22 | 200 | 27.596µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:28.301788 1 request.go:665] Waited for 31.200280131s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-leader63/apisixroutes
[GIN] 2022/10/12 - 21:40:29 | 200 | 40.844µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:32 | 200 | 26.052µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:38.501964 1 request.go:665] Waited for 41.400382216s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-alcoholism27/apisixroutes
[GIN] 2022/10/12 - 21:40:39 | 200 | 23.659µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:42 | 200 | 29.537µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:48.701165 1 request.go:665] Waited for 51.599499253s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-misconception18/apisixroutes
[GIN] 2022/10/12 - 21:40:49 | 200 | 24.521µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:52 | 200 | 24.267µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:58.701926 1 request.go:665] Waited for 1m1.600208075s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-caroling45/apisixroutes
[GIN] 2022/10/12 - 21:40:59 | 200 | 24.771µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:41:02 | 200 | 39.746µs | 10.0.254.118 | GET
"/healthz"
I1012 21:41:08.901096 1 request.go:665] Waited for 1m11.799295833s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-consulted24/apisixroutes
```
and i debug the code found that is the provider_init.go
https://github.com/apache/apisix-ingress-controller/blob/8a17eea26e96570dd1054f258e484bf7814627eb/pkg/providers/apisix/provider_init.go#L141
### Environment
- your apisix-ingress-controller version (output of
apisix-ingress-controller version --long):
- your Kubernetes cluster version (output of kubectl version):
- if you run apisix-ingress-controller in Bare-metal environment, also show
your OS version (uname -a):
Version: 1.4.1
Git SHA: no-git-module
Go Version: go1.16.15
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6",
GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean",
BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc",
Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+",
GitVersion:"v1.19.16-eks-a05fdea",
GitCommit:"a05fdea9a04815782f9c4b08eed73bb2d78b07e3", GitTreeState:"clean",
BuildDate:"2022-06-09T21:50:47Z", GoVersion:"go1.15.15", Compiler:"gc",
Platform:"linux/amd64"}
### Minimal test code / Steps to reproduce
1. create many namespaces
### Actual result
ingress controller stuck
### Error log
```
I1012 21:39:58.101271 1 request.go:665] Waited for 1.000058582s due to
client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-disagreement42/apisixroutes
[GIN] 2022/10/12 - 21:39:59 | 200 | 66.378µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:02 | 200 | 48.148µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:08.101818 1 request.go:665] Waited for 11.000514374s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-negligee64/apisixroutes
[GIN] 2022/10/12 - 21:40:09 | 200 | 37.35µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:12 | 200 | 26.873µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:18.301545 1 request.go:665] Waited for 21.20015681s due to
client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-earum75/apisixroutes
[GIN] 2022/10/12 - 21:40:19 | 200 | 24.12µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:22 | 200 | 27.596µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:28.301788 1 request.go:665] Waited for 31.200280131s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-leader63/apisixroutes
[GIN] 2022/10/12 - 21:40:29 | 200 | 40.844µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:32 | 200 | 26.052µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:38.501964 1 request.go:665] Waited for 41.400382216s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-alcoholism27/apisixroutes
[GIN] 2022/10/12 - 21:40:39 | 200 | 23.659µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:42 | 200 | 29.537µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:48.701165 1 request.go:665] Waited for 51.599499253s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-misconception18/apisixroutes
[GIN] 2022/10/12 - 21:40:49 | 200 | 24.521µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:40:52 | 200 | 24.267µs | 10.0.254.118 | GET
"/healthz"
I1012 21:40:58.701926 1 request.go:665] Waited for 1m1.600208075s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-caroling45/apisixroutes
[GIN] 2022/10/12 - 21:40:59 | 200 | 24.771µs | 10.0.254.118 | GET
"/healthz"
[GIN] 2022/10/12 - 21:41:02 | 200 | 39.746µs | 10.0.254.118 | GET
"/healthz"
I1012 21:41:08.901096 1 request.go:665] Waited for 1m11.799295833s due
to client-side throttling, not priority and fairness, request:
GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-consulted24/apisixroutes
```
### Expected result
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
