jhemarcos opened a new issue, #7530:
URL: https://github.com/apache/cloudstack/issues/7530
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
SystemVMs
~~~
##### CLOUDSTACK VERSION
~~~
4.18.0.0
~~~
##### CONFIGURATION
A `fresh install` of CloudStack, adding a `first zone with advanced
networks` (with security groups). Exporting two `NFS` paths from management
server to use as `primary` and `secondary` storage, following [these
docs](https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html#using-the-management-server-as-the-nfs-server).
CloudStack handle the systemVM template registration.
##### OS / ENVIRONMENT
**Cloudstack Management Server:** `Ubuntu 20.04 (focal)`
**Host:** `KVM` with `Ubuntu 22.04 (jammy)`
##### SUMMARY
After configuring the first zone as described above, the system VMs do not
boot.
I can see the following alerts being generated every minute:
And in the management server logs I can see some exceptions:
```
2023-05-12 23:32:35,214 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-16:ctx-b88db058 job-28/job-1267 ctx-84fa0752)
(logid:ba649f5e) Unable to orchestrate start VM instance
{"id":11,"instanceName":"s-11-VM","type":"SecondaryStorageVm","uuid":"dff4e720-b2ee-4083-8eb5-30054d3e0b33"}
due to [Unable to create deployment, no usable volumes found for the VM: 11].
com.cloud.utils.exception.CloudRuntimeException: Unable to create
deployment, no usable volumes found for the VM: 11
at
com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1635)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1247)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:543)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1152)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5351)
at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5475)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-05-12 23:33:05,211 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-17:ctx-f74f0581 job-919/job-1268 ctx-25907c80)
(logid:7edf35c0) VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
state transited from [Starting] to [Stopped] with event [OperationFailed].
VM's original host: null, new host: null, host before state transition: null
2023-05-12 23:33:05,211 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-17:ctx-f74f0581 job-919/job-1268 ctx-25907c80)
(logid:7edf35c0) Unable to orchestrate start VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
due to [Unable to create deployment, no usable volumes found for the VM: 12].
com.cloud.utils.exception.CloudRuntimeException: Unable to create
deployment, no usable volumes found for the VM: 12
at
com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1635)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1247)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:543)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1152)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5351)
at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5475)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
```
2023-05-12 23:33:05,341 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-b64486e7) (logid:12474ac6) received console proxy alert
2023-05-12 23:33:05,343 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-b64486e7) (logid:12474ac6) Console proxy creation failure,
zone: 01.sc
2023-05-12 23:33:05,344 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-b64486e7) (logid:12474ac6) alertType=[10] dataCenterId=[4]
podId=[null] clusterId=[null] message=[Console proxy creation failure. zone:
01.sc, error details: Unable to orchestrate start VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
due to [Unable to create deployment, no usable volumes found for the VM: 12].].
2023-05-12 23:33:05,348 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-15221c2a) (logid:a2e4d955) received secondary storage vm alert
2023-05-12 23:33:05,349 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter]
(secstorage-1:ctx-15221c2a) (logid:a2e4d955) Secondary Storage Vm creation
failure, zone: 01.sc
2023-05-12 23:33:05,351 WARN [c.c.a.AlertManagerImpl]
(secstorage-1:ctx-15221c2a) (logid:a2e4d955) alertType=[19] dataCenterId=[4]
podId=[null] clusterId=[null] message=[Secondary Storage Vm creation failure.
zone: 01.sc, error details: Unable to allocate capacity on zone [4] due to
[null].].
2023-05-12 23:33:05,376 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-b64486e7) (logid:12474ac6) No recipients set in global
setting 'alert.email.addresses', skipping sending alert with subject [Console
proxy creation failure. zone: 01.sc, error details: Unable to orchestrate start
VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
due to [Unable to create deployment, no usable volumes found for the VM: 12].]
and content [Console proxy creation failure (zone 01.sc)].
2023-05-12 23:33:05,376 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-b64486e7) (logid:12474ac6) Unexpected exception Unable to
orchestrate start VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
due to [Unable to create deployment, no usable volumes found for the VM: 12].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate start
VM instance
{"id":12,"instanceName":"v-12-VM","type":"ConsoleProxy","uuid":"201b39c8-e89a-4ad5-bfcb-4c2197fd3559"}
due to [Unable to create deployment, no usable volumes found for the VM: 12].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5355)
at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5475)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-05-12 23:33:05,388 WARN [c.c.a.AlertManagerImpl]
(secstorage-1:ctx-15221c2a) (logid:a2e4d955) No recipients set in global
setting 'alert.email.addresses', skipping sending alert with subject [Secondary
Storage Vm creation failure. zone: 01.sc, error details: Unable to allocate
capacity on zone [4] due to [null].] and content [Secondary Storage Vm creation
failure (zone 01.sc)].
2023-05-12 23:33:05,390 WARN [c.c.v.SystemVmLoadScanner]
(secstorage-1:ctx-15221c2a) (logid:a2e4d955) Unexpected exception Unable to
orchestrate start VM instance
{"id":11,"instanceName":"s-11-VM","type":"SecondaryStorageVm","uuid":"dff4e720-b2ee-4083-8eb5-30054d3e0b33"}
due to [Unable to create deployment, no usable volumes found for the VM: 11].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate start
VM instance
{"id":11,"instanceName":"s-11-VM","type":"SecondaryStorageVm","uuid":"dff4e720-b2ee-4083-8eb5-30054d3e0b33"}
due to [Unable to create deployment, no usable volumes found for the VM: 11].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5355)
at jdk.internal.reflect.GeneratedMethodAccessor184.invoke(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5475)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
```
Analyzing the
[**code**](https://github.com/apache/cloudstack/blob/2f309b573bc70abea91af463a277dd91cc774f79/server/src/main/java/com/cloud/deploy/DeploymentPlanningManagerImpl.java#L1661)
for `findSuitablePoolsForVolumes` in stacktrace, I notice that in the
database, there is no record of volumes for the system VMs, but **I don't know
if this was expected**.
In the agent logs, I can see the following messages every minute:
```
2023-05-12 23:38:39,788 INFO [kvm.storage.LibvirtStorageAdaptor]
(agentRequest-Handler-3:null) (logid:4b681d10) Asking libvirt to refresh
storage pool 0f20f23f-9498-395b-b694-7577fd2b84992023-05-12 23:39:39,857 INFO
[kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null)
(logid:0220afd8) Trying to fetch storage pool
0f20f23f-9498-395b-b694-7577fd2b8499 from libvirt
```
The services for `libvirt` and `cloudstack-agent` in the host are running:
After the problems I tried to register the template manually
(https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html#prepare-the-system-vm-template),
but without error evolution.
##### STEPS TO REPRODUCE
* A fresh install of 4.18.0.0
* Using Management Server as NFS server
* Setup Advanced Zone with security groups from the UI (bellow some
screenshots)
>
>
>
>
>
>
>
>
##### EXPECTED RESULTS
A correct first zone added after setup with system VMs running.
##### ACTUAL RESULTS
SystemVMs cannot start.
> PS.: I'm new to CloudStack so I might be missing something :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
