acuteaura commented on issue #1980:
URL:
https://github.com/apache/apisix-ingress-controller/issues/1980#issuecomment-1910041686
This is a *criticial* issue and checking if the TCP port is available is
absolutely not sufficient, @tao12345666333. We had yet another ingress
controller that simply got stuck. Frankly, the entire leader election code
seems broken (e.g reports as "running as leader" in the callback, where
*nothing else* happens, starts controller loop once *any* controller is
elected), and I'll look into getting a PR in for this as soon as possible, this
is simply making us look bad as an infra team for adopting the ingress.
```
2024-01-19T21:29:00+08:00 [34minfo[0m ingress/ingress.go:121 init
apisix ingress controller
2024-01-19T21:29:00+08:00 [34minfo[0m ingress/ingress.go:123 version:
Version: 1.7.0
Git SHA: no-git-module
Go Version: go1.20.8
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
2024-01-19T21:29:00+08:00 [34minfo[0m ingress/ingress.go:133 use
configuration
{
"cert_file": "/etc/webhook/certs/cert.pem",
"key_file": "/etc/webhook/certs/key.pem",
"log_level": "info",
"log_output": "stderr",
"log_rotate_output_path": "",
"log_rotation_max_size": 100,
"log_rotation_max_age": 0,
"log_rotation_max_backups": 0,
"http_listen": ":8080",
"https_listen": ":8443",
"ingress_publish_service": "apisix/apisix-internal-gateway",
"ingress_status_address": [],
"enable_profiling": true,
"kubernetes": {
"kubeconfig": "",
"resync_interval": "6h0m0s",
"namespace_selector": [],
"election_id": "ingress-apisix-internal-leader",
"ingress_class": "apisix-internal",
"ingress_version": "networking/v1",
"watch_endpoint_slices": false,
"api_version": "apisix.apache.org/v2",
"enable_gateway_api": false,
"disable_status_updates": false,
"enable_admission": false
},
"apisix": {
"admin_api_version": "v3",
"default_cluster_name": "default",
"default_cluster_base_url":
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin";,
"default_cluster_admin_key": "******"
},
"apisix_resource_sync_interval": "1h0m0s",
"apisix_resource_sync_comparison": true,
"plugin_metadata_cm": "",
"etcdserver": {
"enabled": false,
"prefix": "/apisix",
"listen_address": ":12379",
"ssl_key_encrypt_salt": "edd1c9f0985e76a2"
}
}
2024-01-19T21:29:00+08:00 [34minfo[0m ingress/ingress.go:145 start
ingress controller
2024-01-19T21:29:00+08:00 [34minfo[0m providers/controller.go:170
start leader election
2024-01-19T21:29:00+08:00 [34minfo[0m providers/controller.go:155
start api server
I0119 21:29:00.286860 1 leaderelection.go:250] attempting to acquire
leader lease apisix/ingress-apisix-internal-leader...
2024-01-19T21:29:00+08:00 [33mwarn[0m providers/controller.go:220
found a new leader apisix-internal-ingress-controller-578bf88584-jvs59
2024-01-19T21:29:00+08:00 [34minfo[0m providers/controller.go:222
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:00+08:00 [34minfo[0m providers/controller.go:387
controller tries to leading ... {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:00+08:00 [34minfo[0m apisix/stream_route.go:38
resource stream_routes is disabled
2024-01-19T21:29:00+08:00 [34minfo[0m apisix/cluster.go:248 syncing
cache {"cluster": "default"}
2024-01-19T21:29:00+08:00 [33mwarn[0m apisix/cluster.go:423 waiting
cluster default to ready, it may takes a while
2024-01-19T21:29:00+08:00 [34minfo[0m apisix/cluster.go:463 syncing
schema {"cluster": "default"}
2024-01-19T21:29:00+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:00+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:02+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:02+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:03+08:00 [34minfo[0m providers/controller.go:141
LeaderElection {"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l
became leader", "event_type": "Normal"}
I0119 21:29:03.113259 1 leaderelection.go:260] successfully acquired
lease apisix/ingress-apisix-internal-leader
2024-01-19T21:29:03+08:00 [33mwarn[0m providers/controller.go:220
found a new leader apisix-internal-ingress-controller-9d96f45cb-4kg8l
2024-01-19T21:29:03+08:00 [34minfo[0m providers/controller.go:212
controller now is running as leader {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
2024-01-19T21:29:04+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:04+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:06+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:06+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:08+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: unexpected status code 503; error message:
{"error_msg":"invalid response code: 502"}
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:258 failed
to sync cache {"cost_time": "8.016969049s", "cluster": "default"}
2024-01-19T21:29:08+08:00 [31merror[0m providers/controller.go:419
failed to wait the default cluster to be ready: unexpected status code 503;
error message: {"error_msg":"invalid response code: 502"}
2024-01-19T21:29:08+08:00 [34minfo[0m apisix/stream_route.go:38
resource stream_routes is disabled
2024-01-19T21:29:08+08:00 [34minfo[0m apisix/cluster.go:248 syncing
cache {"cluster": "default"}
2024-01-19T21:29:08+08:00 [34minfo[0m apisix/cluster.go:463 syncing
schema {"cluster": "default"}
2024-01-19T21:29:08+08:00 [31merror[0m apisix/route.go:90 failed
to list routes: Get
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/routes":
context canceled
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:298 failed
to list routes in APISIX: Get
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/routes":
context canceled
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:258 failed
to sync cache {"cost_time": "60.702µs", "cluster": "default"}
2024-01-19T21:29:08+08:00 [31merror[0m apisix/plugin.go:46 failed
to list plugins' names: Get
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true":
context canceled
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:483 failed
to list plugin names in APISIX: Get
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true":
context canceled
2024-01-19T21:29:08+08:00 [31merror[0m apisix/cluster.go:446 failed
to sync schema: Get
"http://apisix-internal-admin.apisix.svc.cluster.local:9180/apisix/admin/plugins?all=true":
context canceled
E0122 23:07:05.784720 1 leaderelection.go:369] Failed to update lock:
Put
"https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/ingress-apisix-internal-leader":
context deadline exceeded
I0122 23:07:05.784783 1 leaderelection.go:285] failed to renew lease
apisix/ingress-apisix-internal-leader: timed out waiting for the condition
2024-01-22T23:07:09+08:00 [34minfo[0m providers/controller.go:141
LeaderElection {"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l
stopped leading", "event_type": "Normal"}
2024-01-22T23:07:09+08:00 [34minfo[0m providers/controller.go:231
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
I0122 23:07:09.152585 1 leaderelection.go:250] attempting to acquire
leader lease apisix/ingress-apisix-internal-leader...
2024-01-22T23:07:09+08:00 [34minfo[0m providers/controller.go:141
LeaderElection {"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l
became leader", "event_type": "Normal"}
I0122 23:07:09.984780 1 leaderelection.go:260] successfully acquired
lease apisix/ingress-apisix-internal-leader
2024-01-22T23:07:09+08:00 [34minfo[0m providers/controller.go:212
controller now is running as leader {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
E0124 13:07:40.830633 1 leaderelection.go:369] Failed to update lock:
Put
"https://10.0.0.1:443/apis/coordination.k8s.io/v1/namespaces/apisix/leases/ingress-apisix-internal-leader":
context deadline exceeded
I0124 13:07:40.830667 1 leaderelection.go:285] failed to renew lease
apisix/ingress-apisix-internal-leader: timed out waiting for the condition
2024-01-24T13:07:40+08:00 [34minfo[0m providers/controller.go:141
LeaderElection {"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l
stopped leading", "event_type": "Normal"}
2024-01-24T13:07:40+08:00 [34minfo[0m providers/controller.go:231
controller now is running as a candidate {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
I0124 13:07:40.843705 1 leaderelection.go:250] attempting to acquire
leader lease apisix/ingress-apisix-internal-leader...
2024-01-24T13:07:40+08:00 [34minfo[0m providers/controller.go:141
LeaderElection {"message": "apisix-internal-ingress-controller-9d96f45cb-4kg8l
became leader", "event_type": "Normal"}
I0124 13:07:40.864098 1 leaderelection.go:260] successfully acquired
lease apisix/ingress-apisix-internal-leader
2024-01-24T13:07:40+08:00 [34minfo[0m providers/controller.go:212
controller now is running as leader {"namespace": "apisix", "pod":
"apisix-internal-ingress-controller-9d96f45cb-4kg8l"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
