slene opened a new issue #449: URL: https://github.com/apache/apisix-ingress-controller/issues/449
### Issue description ### Environment * apisix-ingress-controller, latest master branch * k8s v1.20.2 * apisix 2.5 ### Minimal test code / Steps to reproduce the issue 1. start apisix-ingress first 2. stop kube-apiserver 3. controller now is running as a candidate 4. start kube-apiserver 5. controller now is running as leader 6. panic: "close of closed channel" (close of closed channel) ### What's the actual result? (including assertion message & call stack if applicable) ``` E0513 17:38:10.666973 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v2alpha1.ApisixClusterConfig: failed to list *v2alpha1.ApisixClusterConfig: the server could not find the requested resource (get apisixclusterconfigs.apisix.apache.org) W0513 17:38:18.852172 1 reflector.go:436] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: watch of *v1.Secret ended with: very short watch: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Unexpected watch close - watch lasted less than a second and no items received E0513 17:38:19.692055 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:20.598540 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://172.24.78.235:8443/api/v1/secrets?resourceVersion=354595": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:21.692777 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:23.692718 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused 2021-05-13T17:38:24+08:00 info ingress/controller.go:246 LeaderElection {"message": " stopped leading", "event_type": "Normal"} I0513 17:38:24.691934 1 leaderelection.go:278] failed to renew lease default/ingress-apisix-leader: timed out waiting for the condition E0513 17:38:24.692023 1 leaderelection.go:301] Failed to release lock: resource name may not be empty 2021-05-13T17:38:24+08:00 info ingress/controller.go:293 controller now is running as a candidate {"namespace": "default", "pod": ""} I0513 17:38:24.692080 1 leaderelection.go:243] attempting to acquire leader lease default/ingress-apisix-leader... 2021-05-13T17:38:24+08:00 info ingress/apisix_tls.go:70 ApisixTls controller exited 2021-05-13T17:38:24+08:00 info ingress/apisix_route.go:70 ApisixRoute controller exited 2021-05-13T17:38:24+08:00 info ingress/endpoint.go:86 endpoints controller exited 2021-05-13T17:38:24+08:00 info ingress/apisix_upstream.go:70 ApisixUpstream controller exited 2021-05-13T17:38:24+08:00 info ingress/secret.go:74 secret controller exited 2021-05-13T17:38:24+08:00 info ingress/ingress.go:71 ingress controller exited 2021-05-13T17:38:24+08:00 error ingress/apisix_cluster_config.go:58 cache sync failed 2021-05-13T17:38:24+08:00 info ingress/apisix_cluster_config.go:59 ApisixClusterConfig controller exited E0513 17:38:24.692344 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:29.059509 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:31.265768 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:34.873517 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:37.442765 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:40.728304 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused E0513 17:38:43.178480 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused 2021-05-13T17:38:47+08:00 info ingress/controller.go:246 LeaderElection {"message": " became leader", "event_type": "Normal"} I0513 17:38:47.031012 1 leaderelection.go:253] successfully acquired lease default/ingress-apisix-leader 2021-05-13T17:38:47+08:00 info ingress/controller.go:321 controller now is running as leader {"namespace": "default", "pod": ""} 2021-05-13T17:38:47+08:00 info ingress/secret.go:60 secret controller started 2021-05-13T17:38:47+08:00 info ingress/endpoint.go:59 endpoints controller started 2021-05-13T17:38:47+08:00 info ingress/apisix_upstream.go:58 ApisixUpstream controller started 2021-05-13T17:38:47+08:00 info ingress/ingress.go:59 ingress controller started 2021-05-13T17:38:47+08:00 info ingress/apisix_route.go:57 ApisixRoute controller started 2021-05-13T17:38:47+08:00 info ingress/apisix_cluster_config.go:55 ApisixClusterConfig controller started 2021-05-13T17:38:47+08:00 info ingress/apisix_tls.go:58 ApisixTls controller started E0513 17:38:47.032363 1 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel) goroutine 678 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x16f8a20, 0x1b85f10) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x92 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x82 panic(0x16f8a20, 0x1b85f10) c:/go/src/runtime/panic.go:969 +0x166 k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002f0e00) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:752 +0x2ba k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001f8f40, 0xc00010d4d0) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x62 panic: close of closed channel [recovered] panic: close of closed channel goroutine 678 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x105 panic(0x16f8a20, 0x1b85f10) c:/go/src/runtime/panic.go:969 +0x166 k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002f0e00) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:752 +0x2ba k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001f8f40, 0xc00010d4d0) C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x51 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start C:/Users/slene/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:71 +0x62 ``` ### What's the expected result? https://github.com/apache/apisix-ingress-controller/blob/f199cdb5f5bfe3cb5acb19dc1903b1f5f426a353/pkg/ingress/controller.go#L190-L205 https://github.com/apache/apisix-ingress-controller/blob/f199cdb5f5bfe3cb5acb19dc1903b1f5f426a353/pkg/ingress/controller.go#L344-L367 When controller become to leader from candidate. That code use SharedIndexInformer not recycled. And reused in Controller.run. But context already canceled. So code panic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
