slene opened a new issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448
### Issue description
### Environment
* apisix-ingress-controller, latest master branch
* k8s v1.20.2
* apisix 2.5
### Minimal test code / Steps to reproduce the issue
1. start apisix-ingress first
2. then start apisix
3. change some ingress resources
4. always sync ingress failed
### What's the actual result? (including assertion message & call stack if
applicable)
```
I0513 17:07:52.216990 1 leaderelection.go:243] attempting to acquire
leader lease default/ingress-apisix-leader...
2021-05-13T17:07:52+08:00 info ingress/controller.go:246
LeaderElection {"message": " became leader", "event_type": "Normal"}
I0513 17:07:52.271697 1 leaderelection.go:253] successfully acquired
lease default/ingress-apisix-leader
2021-05-13T17:07:52+08:00 info ingress/controller.go:321
controller now is running as leader {"namespace": "default", "pod": ""}
2021-05-13T17:07:52+08:00 warn apisix/cluster.go:252 waiting
cluster default to ready, it may takes a while
2021-05-13T17:07:52+08:00 info apisix/cluster.go:115 syncing
cache {"cluster": "default"}
2021-05-13T17:07:52+08:00 debug apisix/route.go:111 try to list
routes in APISIX {"cluster": "default", "url":
"http://127.0.0.1:9182/apisix/admin/routes"}
2021-05-13T17:07:52+08:00 error apisix/route.go:117 failed to
list routes: Get "http://127.0.0.1:9182/apisix/admin/routes": dial tcp
127.0.0.1:9182: connect: connection refused
2021-05-13T17:07:52+08:00 error apisix/cluster.go:159 failed to
list route in APISIX: Get "http://127.0.0.1:9182/apisix/admin/routes": dial tcp
127.0.0.1:9182: connect: connection refused
2021-05-13T17:07:52+08:00 error apisix/cluster.go:127 failed to
sync cache {"cost_time": "6.631413ms", "cluster": "default"}
2021-05-13T17:07:52+08:00 warn apisix/cluster.go:258 cluster
default now is ready, cost time 6.661213ms
2021-05-13T17:07:52+08:00 info ingress/secret.go:60 secret
controller started
2021-05-13T17:07:52+08:00 info ingress/endpoint.go:59 endpoints
controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_tls.go:58
ApisixTls controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_upstream.go:58
ApisixUpstream controller started
2021-05-13T17:07:52+08:00 info ingress/ingress.go:59 ingress
controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_route.go:57
ApisixRoute controller started
...
2021-05-13T17:09:57+08:00 error ingress/ingress.go:176 failed to
sync ingress artifacts {"error": "2 errors occurred:\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\n"}
...
// start apisix
// change some ingress resources
// like kubectl create ing httpbinxx --rule="/*=httpbin:80" --class=apisix
...
2021-05-13T17:11:19+08:00 debug ingress/ingress.go:138 translated
ingress resource to a couple of routes and upstreams {"ingress": {}, "routes":
[{"id":"95d36b91","name":"ingress__/","uris":["/","/*"],"upstream_id":"5ce57b8e"}],
"upstreams": [{"id":"5ce57b8e","name":"default_httpbin_80","desc":"Created by
apisix-ingress-controller, DO NOT modify it
manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.2.0.4","port":80,"weight":100}],"scheme":"http"}]}
2021-05-13T17:11:19+08:00 debug apisix/route.go:177 try to
delete route {"id": "95d36b91", "name": "ingress__/", "cluster": "default",
"url": "http://127.0.0.1:9182/apisix/admin/routes"}
2021-05-13T17:11:19+08:00 debug apisix/upstream.go:169 try to
delete upstream {"id": "5ce57b8e", "name": "default_httpbin_80", "cluster":
"default", "url": "http://127.0.0.1:9182/apisix/admin/upstreams"}
2021-05-13T17:11:19+08:00 error ingress/ingress.go:176 failed to
sync ingress artifacts {"error": "2 errors occurred:\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\n"}
2021-05-13T17:11:19+08:00 warn ingress/ingress.go:189 sync ingress
failed, will retry {"object":
{"Type":3,"Object":{"Key":"default/httpbinxx","GroupVersion":"extensions/v1beta1","OldObject":null},"Tombstone":{}},
"error": "2 errors occurred:\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\t* Get
\"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182:
connect: connection refused\n\n"}
```
### What's the expected result?
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/cluster.go#L133-L142
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/cluster.go#L242-L244
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/route.go#L141-L151
https://github.com/apache/apisix-ingress-controller/pull/176 say.
> Also, controller will try to sync (with an exponential back off strategy)
to the APISIX cluster when the cluster was just added. All operations will fail
if the cluster is not ready (the initial sync is failure), we'll add a recover
mechanism once the APISIX cluster recovered, the controller shall detect it.
I have not found any recover code if cluster is not ready.
ExponentialBackoff will return if syncCacheOnce return an error. Can it be use
PollImmediateInfinite instead ? It can wait for success and return. If
possible, I will submit the code.
```
err := wait.PollImmediateInfinite(time.Second*3, func() (bool, error) {
// wait until success
if done, _ := c.syncCacheOnce(); done {
return true, nil
}
return false, nil
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
