amitmvyas commented on issue #11994:
URL: https://github.com/apache/cloudstack/issues/11994#issuecomment-3495267094
```
kubectl --insecure-skip-tls-verify=true --kubeconfig kube.conf get pods -A
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7bfdc5b57c-kj99k
1/1 Running 1 (19h ago) 23h
kube-system calico-node-d8dsj
1/1 Running 1 (19h ago) 23h
kube-system calico-node-hqlxb
1/1 Running 1 (19h ago) 23h
kube-system calico-node-lpnrl
1/1 Running 1 (19h ago) 23h
kube-system cloud-controller-manager-857cd68cf5-8xbfr
1/1 Running 2 (19h ago) 23h
kube-system cluster-autoscaler-744b9d65b-zzw2p
1/1 Running 2 (19h ago) 19h
kube-system coredns-674b8bbfcf-9wgdv
1/1 Running 1 (19h ago) 23h
kube-system coredns-674b8bbfcf-s5v82
1/1 Running 1 (19h ago) 23h
kube-system etcd-cl8s-control-19a5261efb6
1/1 Running 1 (19h ago) 23h
kube-system kube-apiserver-cl8s-control-19a5261efb6
1/1 Running 1 (19h ago) 23h
kube-system kube-controller-manager-cl8s-control-19a5261efb6
1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-g98w9
1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-kg5t4
1/1 Running 1 (19h ago) 23h
kube-system kube-proxy-prtkk
1/1 Running 1 (19h ago) 23h
kube-system kube-scheduler-cl8s-control-19a5261efb6
1/1 Running 1 (19h ago) 23h
kubernetes-dashboard dashboard-metrics-scraper-5bd45c9dd6-fm7wn
1/1 Running 1 (19h ago) 23h
kubernetes-dashboard kubernetes-dashboard-687457b9bd-pcdt7
1/1 Running 1 (19h ago) 23h
scale-test big-request-697b5d9654-hscsm
1/1 Running 1 (19h ago) 20h
scale-test big-request-697b5d9654-p224c
0/1 Pending 0 20h
scale-test big-request-697b5d9654-tw7fq
1/1 Running 1 (19h ago) 20h
```
Tried with CMK as well
```
root@innmi1vcsmgt-d001:~# cmk scale kubernetescluster
id=da626491-7457-4106-a165-de0d41d2417a size=4
{
"account": "admin",
"accountid": "9b2313ee-b4ae-11f0-914b-bc2411c796cd",
"cmd":
"org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2025-11-06T11:38:18+0530",
"created": "2025-11-06T11:38:18+0530",
"domainid": "abbdc016-b4ad-11f0-914b-bc2411c796cd",
"domainpath": "ROOT",
"jobid": "62e801e2-cc16-4014-9b50-d78b67e875e7",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Scaling Kubernetes cluster : CL8S failed, scaling
Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "9b23987d-b4ae-11f0-914b-bc2411c796cd"
}
🙈 Error: async API failed for job 62e801e2-cc16-4014-9b50-d78b67e875e7
```
This is the logs for cluster-autoscaler.
```
I1106 02:53:19.477668 1 client.go:169] NewAPIRequest API request
URL:http://142.79.253.232:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=da626491-7457-4106-a165-de0d41d2417a&response=json&size=3&signature=***
I1106 02:53:19.477890 1 event_sink_logging_wrapper.go:48]
Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system",
Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5",
APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Normal'
reason: 'ScaledUpGroup' Scale-up: setting group
da626491-7457-4106-a165-de0d41d2417a size to 3 instead of 2 (max: 4)
I1106 02:53:19.556058 1 client.go:175] NewAPIRequest response status
code:200
I1106 02:53:19.863209 1 reflector.go:255] Listing and watching
*v1beta1.PodDisruptionBudget from
/home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
W1106 02:53:19.865022 1 reflector.go:324]
/home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309:
failed to list *v1beta1.PodDisruptionBudget: the server could not find the
requested resource
E1106 02:53:19.865056 1 reflector.go:138]
/home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309:
Failed to watch *v1beta1.PodDisruptionBudget: failed to list
*v1beta1.PodDisruptionBudget: the server could not find the requested resource
I1106 02:53:29.557052 1 client.go:169] NewAPIRequest API request
URL:http://142.79.253.232:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=acee7bf9-c308-4929-87e5-42f74652d5eb&response=json&signature=***
I1106 02:53:29.597851 1 client.go:175] NewAPIRequest response status
code:200
E1106 02:53:29.598077 1 client.go:118] API failed for job
acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin
accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd
cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530
domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT
jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0
jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed,
scaling Kubernetes cluster with running VMs on hypervisor KVM is not
supported!] jobresultcode:530 jobresulttype:object jobstatus:2
userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
W1106 02:53:29.598146 1 clusterstate.go:277] Disabling scale-up for
node group da626491-7457-4106-a165-de0d41d2417a until 2025-11-06
03:23:19.477324222 +0000 UTC m=+64348.574537733; errorClass=Other;
errorCode=cloudProviderError
I1106 02:53:29.598178 1 metrics.go:380] Function scaleUp took
10.120850343s to complete
I1106 02:53:29.598196 1 eventing_scale_up_processor.go:47] Skipping
event processing for unschedulable pods since there is a ScaleUp attempt this
loop
E1106 02:53:29.598212 1 static_autoscaler.go:459] Failed to scale up:
failed to increase node group size: Unable to scale cluster : API failed for
job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin
accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd
cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530
domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT
jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0
jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed,
scaling Kubernetes cluster with running VMs on hypervisor KVM is not
supported!] jobresultcode:530 jobresulttype:object jobstatus:2
userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.598232 1 event_sink_logging_wrapper.go:48]
Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system",
Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5",
APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Warning'
reason: 'FailedToScaleUpGroup' Scale-up failed for group
da626491-7457-4106-a165-de0d41d2417a: Unable to scale cluster : API failed for
job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin
accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd
cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530
domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT
jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0
jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed,
scaling Kubernetes cluster with running VMs on hypervisor KVM is not
supported!] jobresultcode:530 jobresultt
ype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.613991 1 metrics.go:380] Function main took
10.194459637s to complete
I1106 02:53:30.318148 1 reflector.go:536]
k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StatefulSet total
8 items received
I1106 02:53:36.016083 1 reflector.go:255] Listing and watching
*v1beta1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:134
W1106 02:53:36.018008 1 reflector.go:324]
k8s.io/client-go/informers/factory.go:134: failed to list
*v1beta1.CSIStorageCapacity: the server could not find the requested resource
E1106 02:53:36.018094 1 reflector.go:138]
k8s.io/client-go/informers/factory.go:134: Failed to watch
*v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the
server could not find the requested resource
I1106 02:53:36.329876 1 reflector.go:536]
k8s.io/client-go/informers/factory.go:134: Watch close -
*v1.PodDisruptionBudget total 11 items received
I1106 02:53:39.614541 1 static_autoscaler.go:235] Starting main loop
I1106 02:53:39.614764 1 client.go:169] NewAPIRequest API request
URL:http://142.79.253.232:8080/client/api?apiKey=***&command=listKubernetesClusters&id=da626491-7457-4106-a165-de0d41d2417a&response=json&signature=***
I1106 02:53:39.679225 1 client.go:175] NewAPIRequest response status
code:200
```
I have attached entire logs as well.
[cluster-autoscaler-744b9d65b-zzw2p.log](https://github.com/user-attachments/files/23382972/cluster-autoscaler-744b9d65b-zzw2p.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
