tuanhoangth1603 commented on issue #7517:
URL: https://github.com/apache/cloudstack/issues/7517#issuecomment-2302392513
> any logs on cloudstack management server and in the pod
`cloud-controller-manager` ?
Thanks for your reply. Here is logs of CCM
```I0821 15:10:39.921244 1 serving.go:313] Generated self-signed cert
in-memory
W0821 15:11:05.221926 1 client_config.go:552] Neither --kubeconfig nor
--master was specified. Using the inClusterConfig. This might not work.
I0821 15:11:05.420413 1 controllermanager.go:120] Version:
v0.0.0-master+$Format:%h$
I0821 15:11:05.422112 1 configmap_cafile_content.go:202] Starting
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422169 1 shared_informer.go:223] Waiting for caches to
sync for
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422223 1 configmap_cafile_content.go:202] Starting
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.422240 1 shared_informer.go:223] Waiting for caches to
sync for
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.618715 1 secure_serving.go:178] Serving securely on
[::]:10258
I0821 15:11:05.618813 1 leaderelection.go:242] attempting to acquire
leader lease kube-system/cloud-controller-manager...
I0821 15:11:05.619351 1 tlsconfig.go:240] Starting
DynamicServingCertificateController
I0821 15:11:05.918307 1 shared_informer.go:230] Caches are synced for
client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.918307 1 shared_informer.go:230] Caches are synced for
client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:06.018794 1 event.go:278]
Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system",
Name:"cloud-controller-manager", UID:"79917607-9b73-462e-bf49-76a5cb88c24b",
APIVersion:"v1", ResourceVersion:"785", FieldPath:""}): type: 'Normal' reason:
'LeaderElection'
cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572
became leader
E0821 15:11:06.018661 1 event.go:316] Could not construct reference
to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""},
ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"",
Namespace:"kube-system", SelfLink:"",
UID:"33584655-e620-4127-9ce5-e1ac08e95df8", ResourceVersion:"786",
Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0,
ext:63859849865, loc:(*time.Location)(0x30c2040)}},
DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil),
Labels:map[string]string(nil), Annotations:map[string]string(nil),
OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil),
ClusterName:"",
ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm",
Operation:"Update", APIVersion:"coordination.k8s.io/v1",
Time:(*v1.Time)(0xc0001a6340), FieldsType:"FieldsV1",
FieldsV1:(*v1.FieldsV1)(0xc0001a6380)}}},
Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil),
LeaseDurationSeconds:(*int32)(nil), AcquireTime
:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil),
LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type
v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event:
'Normal' 'LeaderElection'
'cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572
became leader'
I0821 15:11:06.118337 1 leaderelection.go:252] successfully acquired
lease kube-system/cloud-controller-manager
I0821 15:11:06.120413 1 node_controller.go:110] Sending events to api
server.
I0821 15:11:06.120516 1 controllermanager.go:247] Started "cloud-node"
I0821 15:11:06.220667 1 node_lifecycle_controller.go:78] Sending
events to api server
I0821 15:11:06.220758 1 controllermanager.go:247] Started
"cloud-node-lifecycle"
I0821 15:11:06.319991 1 controllermanager.go:247] Started "service"
I0821 15:11:06.320047 1 core.go:101] Will not configure cloud provider
routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0821 15:11:06.320064 1 controllermanager.go:244] Skipping "route"
I0821 15:11:06.418280 1 controller.go:208] Starting service controller
I0821 15:11:06.418313 1 shared_informer.go:223] Waiting for caches to
sync for service
I0821 15:11:06.918866 1 shared_informer.go:230] Caches are synced for
service
E0821 15:11:08.719857 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.721984 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:11:08.818466 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.821324 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:11:08.823046 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.920077 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.221094 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.319975 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.321029 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.323385 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:16:09.324513 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.327037 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:09.922209 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.019894 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:10.021178 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.118357 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:21:10.120241 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.122821 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
I0821 15:25:38.304728 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3023", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:38.519534 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:38.519646 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3023", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:25:43.520304 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:43.521805 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:43.521878 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:25:53.522501 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:53.524087 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:53.524171 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
E0821 15:26:10.619873 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.622562 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:26:10.718288 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.721020 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:26:10.722174 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.724422 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
I0821 15:26:11.190439 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:11.191841 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:11.191884 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:26:13.524803 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:13.526348 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:13.526489 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:27:33.527261 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:27:33.721428 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:27:33.818172 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
I0821 15:30:13.723275 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason:
'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:30:13.921406 1 controller.go:244] error processing service
default/traefik (will retry): failed to ensure load balancer: error retrieving
load balancer rules: invalid character '<' looking for beginning of value
I0821 15:30:14.018213 1 event.go:278]
Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik",
UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1",
ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason:
'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load
balancer: error retrieving load balancer rules: invalid character '<' looking
for beginning of value
E0821 15:31:10.921484 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.119953 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:31:11.121277 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.123845 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
E0821 15:31:11.125066 1 node_controller.go:237] error retrieving
instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.127270 1 node_controller.go:245] Error getting node
addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by
provider ID: error retrieving node addresses: invalid character '<' looking for
beginning of value, and error by node name: error retrieving node addresses:
invalid character '<' looking for beginning of value
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
