han6565 opened a new issue #769:
URL: https://github.com/apache/apisix-ingress-controller/issues/769
### Issue description
我在k8s上部署了一个apisix，一个apisix-controller，两台配置好后controller健康检查无问题，
当apisix pod被kill后就算pod重新拉起，controller仍然无法重新连接到apisix
但是我后续手动admin访问pod相同地址没有问题
是我配置问题，还是本来没有pod能访问时就应该再也无法连接呢
`
021-11-23T20:51:39+08:00 warn apisix/cluster.go:452 failed to check
health for cluster default: dial tcp 172.24.150.14:9180: connect: connection
refused, will retry
2021-11-23T20:51:39+08:00 warn ingress/controller.go:660 failed
to check health for default cluster: timed out waiting for the condition, give
up leader
2021-11-23T20:51:39+08:00 info ingress/endpoint.go:83 endpoints
controller exited
2021-11-23T20:51:39+08:00 info ingress/apisix_tls.go:71
ApisixTls controller exited
2021-11-23T20:51:39+08:00 error ingress/ingress.go:63 cache sync
failed
2021-11-23T20:51:39+08:00 info ingress/ingress.go:64 ingress
controller exited
2021-11-23T20:51:39+08:00 info ingress/apisix_upstream.go:71
ApisixUpstream controller exited
2021-11-23T20:51:39+08:00 info ingress/secret.go:76 secret
controller exited
2021-11-23T20:51:39+08:00 info ingress/service.go:61 svc controller
exited
2021-11-23T20:51:39+08:00 info ingress/namespace.go:82 namespace
controller exited
2021-11-23T20:51:39+08:00 info ingress/pod.go:56 pod controller
exited
2021-11-23T20:51:39+08:00 info ingress/apisix_route.go:71
ApisixRoute controller exited
2021-11-23T20:51:39+08:00 info ingress/apisix_consumer.go:67
ApisixConsumer controller exited
2021-11-23T20:51:39+08:00 info ingress/apisix_cluster_config.go:69
ApisixClusterConfig controller exited
2021-11-23T20:51:39+08:00 info ingress/controller.go:354
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
I1123 20:51:39.105127 1 leaderelection.go:243] attempting to acquire
leader lease apisix/ingress-apisix-leader...
2021-11-23T20:51:39+08:00 info ingress/controller.go:307
LeaderElection {"message": "apisix-ingress-controller-7994d7bb49-z5hms became
leader", "event_type": "Normal"}
I1123 20:51:39.111962 1 leaderelection.go:253] successfully acquired
lease apisix/ingress-apisix-leader
2021-11-23T20:51:39+08:00 info ingress/controller.go:387
controller tries to leading ... {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
2021-11-23T20:51:39+08:00 error ingress/controller.go:414 failed
to wait the default cluster to be ready: dial tcp 172.24.150.14:9180: connect:
connection refused
E1123 20:51:39.112199 1 leaderelection.go:325] error retrieving
resource lock apisix/ingress-apisix-leader: Get
https://172.24.144.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/ingress-apisix-leader:
context canceled
2021-11-23T20:51:39+08:00 info ingress/controller.go:307
LeaderElection {"message": "apisix-ingress-controller-7994d7bb49-z5hms stopped
leading", "event_type": "Normal"}
I1123 20:51:39.112223 1 leaderelection.go:278] failed to renew lease
apisix/ingress-apisix-leader: timed out waiting for the condition
2021-11-23T20:51:39+08:00 info ingress/controller.go:354
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
I1123 20:51:39.112241 1 leaderelection.go:243] attempting to acquire
leader lease apisix/ingress-apisix-leader...
2021-11-23T20:51:39+08:00 info apisix/cluster.go:156 syncing cache
{"cluster": "default"}
2021-11-23T20:51:39+08:00 info apisix/cluster.go:347 syncing schema
{"cluster": "default"}
2021-11-23T20:51:39+08:00 error apisix/plugin.go:46 failed to list
plugins' names: Get http://172.24.150.14:9180/apisix/admin/plugins/list:
context canceled
2021-11-23T20:51:39+08:00 error apisix/cluster.go:367 failed to list
plugin names in APISIX: Get
http://172.24.150.14:9180/apisix/admin/plugins/list: context canceled
2021-11-23T20:51:39+08:00 warn apisix/cluster.go:330 failed to sync
schema: Get http://172.24.150.14:9180/apisix/admin/plugins/list: context
canceled
2021-11-23T20:51:39+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:39+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
2021-11-23T20:51:39+08:00 info ingress/controller.go:307
LeaderElection {"message": "apisix-ingress-controller-7994d7bb49-z5hms became
leader", "event_type": "Normal"}
I1123 20:51:39.118355 1 leaderelection.go:253] successfully acquired
lease apisix/ingress-apisix-leader
2021-11-23T20:51:39+08:00 info ingress/controller.go:387
controller tries to leading ... {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
2021-11-23T20:51:39+08:00 warn apisix/cluster.go:307 waiting cluster
default to ready, it may takes a while
2021-11-23T20:51:41+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:41+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
[GIN] 2021/11/23 - 20:51:42 | 200 | 42.863µs | 172.24.248.6 | GET
"/healthz"
2021-11-23T20:51:43+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:43+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
2021-11-23T20:51:45+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:45+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
[GIN] 2021/11/23 - 20:51:46 | 200 | 41.607µs | 172.24.248.6 | GET
"/healthz"
2021-11-23T20:51:47+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:47+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
2021-11-23T20:51:47+08:00 error apisix/cluster.go:166 failed to sync
cache {"cost_time": "8.001080415s", "cluster": "default"}
2021-11-23T20:51:47+08:00 error ingress/controller.go:414 failed
to wait the default cluster to be ready: Get
http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:47+08:00 info ingress/controller.go:354
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
I1123 20:51:47.113474 1 leaderelection.go:243] attempting to acquire
leader lease apisix/ingress-apisix-leader...
2021-11-23T20:51:47+08:00 info apisix/cluster.go:347 syncing schema
{"cluster": "default"}
2021-11-23T20:51:47+08:00 error apisix/plugin.go:46 failed to list
plugins' names: Get http://172.24.150.14:9180/apisix/admin/plugins/list:
context canceled
2021-11-23T20:51:47+08:00 error apisix/cluster.go:367 failed to list
plugin names in APISIX: Get
http://172.24.150.14:9180/apisix/admin/plugins/list: context canceled
2021-11-23T20:51:47+08:00 info apisix/cluster.go:156 syncing cache
{"cluster": "default"}
2021-11-23T20:51:47+08:00 error apisix/route.go:119 failed to list
routes: Get http://172.24.150.14:9180/apisix/admin/routes: context canceled
2021-11-23T20:51:47+08:00 error apisix/cluster.go:200 failed to list
route in APISIX: Get http://172.24.150.14:9180/apisix/admin/routes: context
canceled
2021-11-23T20:51:47+08:00 warn apisix/cluster.go:330 failed to sync
schema: Get http://172.24.150.14:9180/apisix/admin/plugins/list: context
canceled
2021-11-23T20:51:47+08:00 info ingress/controller.go:307
LeaderElection {"message": "apisix-ingress-controller-7994d7bb49-z5hms became
leader", "event_type": "Normal"}
I1123 20:51:47.119539 1 leaderelection.go:253] successfully acquired
lease apisix/ingress-apisix-leader
2021-11-23T20:51:47+08:00 info ingress/controller.go:387
controller tries to leading ... {"namespace": "apisix", "pod":
"apisix-ingress-controller-7994d7bb49-z5hms"}
2021-11-23T20:51:47+08:00 warn apisix/cluster.go:307 waiting cluster
default to ready, it may takes a while
`
### Environment
* your apisix-ingress-controller version (output of
`apisix-ingress-controller version --long`);
* 1.3.0
* your Kubernetes cluster version (output of `kubectl version`);
* Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.14",
GitCommit:"89182bdd065fbcaffefec691908a739d161efc03", GitTreeState:"clean",
BuildDate:"2020-12-18T12:02:35Z", GoVersion:"go1.13.15", Compiler:"gc",
Platform:"linux/amd64"}
* if you run apisix-ingress-controller in Bare-metal environment, also show
your OS version (`uname -a`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
