maslow opened a new issue, #2028: URL: https://github.com/apache/apisix-ingress-controller/issues/2028
### Current Behavior Enabling the APISIX ingress controller can lead to a surge in CPU usage on the APISIX gateway. Turning off the APISIX ingress controller will resolve the issue. <img width="347" alt="image" src="https://github.com/apache/apisix-ingress-controller/assets/7122237/27b621ef-90b5-4957-8345-801af28c29d2";> The above-mentioned issue can be consistently and reliably reproduced in my cluster. Each time, the memory usage increases for three minutes, then the CPU usage returns to a lower level for three minutes, and after three minutes, it will rise again, continuing for three minutes. It's almost unusable and severely affects operations, with the service delay increasing from 20ms to 2000ms. ### Expected Behavior _No response_ ### Error Logs 2023-10-27T11:11:29+08:00 error ingress/ingress.go:148 failed to translate ingress {"error": "endpoints: endpoints \"j3t46i\" not found", "ingress": {}} 2023-10-27T11:11:29+08:00 warn ingress/ingress.go:268 sync ingress failed, will retry {"object": {"Type":1,"Object":{"Key":"j3t46i/j3t46i","GroupVersion":"networking/v1","OldObject":null},"OldObject":null,"Tombstone":null}, "error": "endpoints: endpoints \"j3t46i\" not found"} 2023-10-27T11:11:29+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:29+08:00 error translation/translator.go:158 failed to translate ingress backend to upstream {"error": "endpoints: endpoints \"dz24qv\" not found", "ingress": "&Ingress{ObjectMeta:{dz24qv dz24qv 37023cf1-b564-46cc-8779-77ed62cf901b 295270932 1 2023-10-19 20:31:45 +0800 HKT <nil> <nil> map[] map[k8s.apisix.apache.org/cors-allow-credential:false k8s.apisix.apache.org/cors-allow-headers:* k8s.apisix.apache.org/cors-allow-methods:* k8s.apisix.apache.org/cors-allow-origin:* k8s.apisix.apache.org/cors-expose-headers:* k8s.apisix.apache.org/enable-cors:true k8s.apisix.apache.org/enable-websocket:true k8s.apisix.apache.org/svc-namespace:dz24qv laf.dev/appid:dz24qv laf.dev/ingress.type:runtime nginx.ingress.kubernetes.io/cors-allow-credentials:false nginx.ingress.kubernetes.io/cors-allow-headers:DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,x-laf-develop-token,x-laf-func-data nginx.ingress.kubernetes.io/cors-a llow-methods:* nginx.ingress.kubernetes.io/cors-allow-origin:* nginx.ingress.kubernetes.io/cors-expose-headers:* nginx.ingress.kubernetes.io/enable-cors:true nginx.ingress.kubernetes.io/server-snippet:client_header_buffer_size 4096k;\nlarge_client_header_buffers 8 512k;\n] [] [] [{unknown Update networking.k8s.io/v1 2023-10-19 20:31:45 +0800 HKT FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:k8s.apisix.apache.org/cors-allow-credential\":{},\"f:k8s.apisix.apache.org/cors-allow-headers\":{},\"f:k8s.apisix.apache.org/cors-allow-methods\":{},\"f:k8s.apisix.apache.org/cors-allow-origin\":{},\"f:k8s.apisix.apache.org/cors-expose-headers\":{},\"f:k8s.apisix.apache.org/enable-cors\":{},\"f:k8s.apisix.apache.org/enable-websocket\":{},\"f:k8s.apisix.apache.org/svc-namespace\":{},\"f:laf.dev/appid\":{},\"f:laf.dev/ingress.type\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-credentials\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-headers\":{},\"f:nginx.ingress.kubernetes.io/cors- allow-methods\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-origin\":{},\"f:nginx.ingress.kubernetes.io/cors-expose-headers\":{},\"f:nginx.ingress.kubernetes.io/enable-cors\":{},\"f:nginx.ingress.kubernetes.io/server-snippet\":{}}},\"f:spec\":{\"f:ingressClassName\":{},\"f:rules\":{}}} }]},Spec:IngressSpec{DefaultBackend:nil,TLS:[]IngressTLS{},Rules:[]IngressRule{IngressRule{Host:dz24qv.laf.dev,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:dz24qv,Port:ServiceBackendPort{Name:,Number:8000,},},},PathType:*Prefix,},},},},},},IngressClassName:*apisix,},Status:IngressStatus{LoadBalancer:{[]},},}"} 2023-10-27T11:11:29+08:00 error ingress/ingress.go:148 failed to translate ingress {"error": "endpoints: endpoints \"dz24qv\" not found", "ingress": {}} 2023-10-27T11:11:29+08:00 warn ingress/ingress.go:268 sync ingress failed, will retry {"object": {"Type":1,"Object":{"Key":"dz24qv/dz24qv","GroupVersion":"networking/v1","OldObject":null},"OldObject":null,"Tombstone":null}, "error": "endpoints: endpoints \"dz24qv\" not found"} 2023-10-27T11:11:29+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:30+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:31+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:31+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:31+08:00 error translation/translator.go:158 failed to translate ingress backend to upstream {"error": "service.spec.ports: port not defined", "ingress": "&Ingress{ObjectMeta:{6529093daa9e4f74966f5192 h7j5vb 5b043fd8-19fc-43b8-aa5c-be0d2b54933d 288674296 1 2023-10-13 17:09:18 +0800 HKT <nil> <nil> map[] map[k8s.apisix.apache.org/cors-allow-credential:false k8s.apisix.apache.org/cors-allow-headers:* k8s.apisix.apache.org/cors-allow-methods:* k8s.apisix.apache.org/cors-allow-origin:* k8s.apisix.apache.org/cors-expose-headers:* k8s.apisix.apache.org/enable-cors:true k8s.apisix.apache.org/svc-namespace:h7j5vb laf.dev/appid:h7j5vb laf.dev/bucket.name:h7j5vb-test laf.dev/ingress.type:bucket nginx.ingress.kubernetes.io/cors-allow-credentials:false nginx.ingress.kubernetes.io/cors-allow-headers:DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization,x-laf-develop-token,x-laf-func-data,x-amz-content-sha256,x-amz-securit y-token,x-amz-user-agent,x-amz-date nginx.ingress.kubernetes.io/cors-allow-methods:* nginx.ingress.kubernetes.io/cors-allow-origin:* nginx.ingress.kubernetes.io/cors-expose-headers:* nginx.ingress.kubernetes.io/enable-cors:true] [] [] [{unknown Update networking.k8s.io/v1 2023-10-13 17:09:18 +0800 HKT FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:k8s.apisix.apache.org/cors-allow-credential\":{},\"f:k8s.apisix.apache.org/cors-allow-headers\":{},\"f:k8s.apisix.apache.org/cors-allow-methods\":{},\"f:k8s.apisix.apache.org/cors-allow-origin\":{},\"f:k8s.apisix.apache.org/cors-expose-headers\":{},\"f:k8s.apisix.apache.org/enable-cors\":{},\"f:k8s.apisix.apache.org/svc-namespace\":{},\"f:laf.dev/appid\":{},\"f:laf.dev/bucket.name\":{},\"f:laf.dev/ingress.type\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-credentials\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-headers\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-methods\":{},\"f:nginx.ingress.kubernetes.io/cors-allow-or igin\":{},\"f:nginx.ingress.kubernetes.io/cors-expose-headers\":{},\"f:nginx.ingress.kubernetes.io/enable-cors\":{}}},\"f:spec\":{\"f:ingressClassName\":{},\"f:rules\":{}}} }]},Spec:IngressSpec{DefaultBackend:nil,TLS:[]IngressTLS{},Rules:[]IngressRule{IngressRule{Host:oss.laf.dev,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/h7j5vb-test,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:h7j5vb,Port:ServiceBackendPort{Name:,Number:9000,},},},PathType:*Prefix,},},},},},IngressRule{Host:h7j5vb-test.oss.laf.dev,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:h7j5vb,Port:ServiceBackendPort{Name:,Number:9000,},},},PathType:*Prefix,},},},},},},IngressClassName:*apisix,},Status:IngressStatus{LoadBalancer:{[]},},}"} 2023-10-27T11:11:31+08:00 error ingress/ingress.go:148 failed to translate ingress {"error": "service.spec.ports: port not defined", "ingress": {}} 2023-10-27T11:11:31+08:00 warn ingress/ingress.go:268 sync ingress failed, will retry {"object": {"Type":1,"Object":{"Key":"h7j5vb/6529093daa9e4f74966f5192","GroupVersion":"networking/v1","OldObject":null},"OldObject":null,"Tombstone":null}, "error": "service.spec.ports: port not defined"} 2023-10-27T11:11:31+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:32+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:32+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:33+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:34+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} 2023-10-27T11:11:34+08:00 error ingress/ingress.go:472 failed to get APISIX gateway external IPs {"error": "resource name may not be empty"} ### Steps to Reproduce Only my cluster. ### Environment - APISIX Ingress controller version (run `apisix-ingress-controller version --long`) image: apache/apisix-ingress-controller:1.6.0 `apisix-ingress-controller` command not found in this docker image. <img width="463" alt="image" src="https://github.com/apache/apisix-ingress-controller/assets/7122237/9fe4fe8a-866d-4490-b7c8-4cee42b0ec63";> - Kubernetes cluster version (run `kubectl version`) ```sh kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:22:09Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:26Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"} ``` - OS version if running APISIX Ingress controller in a bare-metal environment (run `uname -a`) ```sh uname -a Linux laf-canary-master001 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@apisix.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
