Hi, On 4.15, this behavior occurred when Primary Storage was overflowing to the size of the 'disabled threshold'. Here the size of the indicator does not matter.
Asking ConfigDrive to release NicProfile[136-83-cd11d70e-92a1-4e93-be1a-be2bb6f308c6-192.168.40.243-null] Verifying image storage [1]. Capacity: total=[9 TB],used=[1 TB], threshold=[89.99999761581421%]. Deleting config drive ISO for vm: i-6-83-VM Regards, Piotr -----Original Message----- From: Piotr Pisz <pp...@post.pl> Sent: Wednesday, November 17, 2021 2:55 PM To: users@cloudstack.apache.org Subject: configdrive not working in 4.16 Hi, I have upgraded to 4.16 and am having trouble booting my VM on L2 network with ConfigDrive. I would have a request for help. Regards, Piotr 2021-11-17 14:47:27,229 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Hosts's actual total CPU: 31200 and CPU after applying overprovisioning: 31200 2021-11-17 14:47:27,230 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) We are allocating VM, increasing the used capacity of this host:1 2021-11-17 14:47:27,231 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Current Used CPU: 0 , Free CPU:27300 ,Requested CPU: 3900 2021-11-17 14:47:27,232 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Current Used RAM: (0 bytes) 0 , Free RAM:(57,66 GB) 61916983296 ,Requested RAM: (4,00 GB) 4294967296 2021-11-17 14:47:27,232 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) We are allocating VM to the last host again, so adjusting the reserved capacity if it is not less than required 2021-11-17 14:47:27,232 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Reserved CPU: 3900 , Requested CPU: 3900 2021-11-17 14:47:27,232 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Reserved RAM: (4,00 GB) 4294967296 , Requested RAM: (4,00 GB) 4294967296 2021-11-17 14:47:27,236 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) CPU STATS after allocation: for host: 1, old used: 0, old reserved: 3900, actual total: 31200, total with overprovisioning: 31200; new used:3900, reserved:0; requested cpu:3900,alloc_from_last:true 2021-11-17 14:47:27,241 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) RAM STATS after allocation: for host: 1, old used: (0 bytes) 0, old reserved: (4,00 GB) 4294967296, total: (61,66 GB) 66211950592; new used: (4,00 GB) 4294967296, reserved: (0 bytes) 0; requested mem: (4,00 GB) 4294967296,alloc_from_last:true 2021-11-17 14:47:27,243 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Host: 1 has cpu capability (cpu:8, speed:3900) to support requested CPU: 1 and requested speed: 3900 2021-11-17 14:47:27,243 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Checking if host: 1 has enough capacity for requested CPU: 3900 and requested RAM: (4,00 GB) 4294967296 , cpuOverprovisioningFactor: 1.0 2021-11-17 14:47:27,243 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Hosts's actual total CPU: 31200 and CPU after applying overprovisioning: 31200 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) We need to allocate to the last host again, so checking if there is enough reserved capacity 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Reserved CPU: 3900 , Requested CPU: 3900 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Reserved RAM: (4,00 GB) 4294967296 , Requested RAM: (4,00 GB) 4294967296 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Host has enough CPU and RAM available 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) STATS: Can alloc CPU from host: 1, used: 0, reserved: 3900, actual total: 31200, total with overprovisioning: 31200; requested cpu:3900,alloc_from_last_host?:true ,considerReservedCapacity?: true 2021-11-17 14:47:27,244 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) STATS: Can alloc MEM from host: 1, used: (0 bytes) 0, reserved: (4,00 GB) 4294967296, total: (61,66 GB) 66211950592; requested mem: (4,00 GB) 4294967296, alloc_from_last_host?: true , considerReservedCapacity?: true 2021-11-17 14:47:27,258 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Network id=212 is already implemented 2021-11-17 14:47:27,275 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Service SecurityGroup is not supported in the network id=212 2021-11-17 14:47:27,279 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Changing active number of nics for network id=212 on 1 2021-11-17 14:47:27,294 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Asking ConfigDrive to prepare for Nic[136-83-cd11d70e-92a1-4e93-be1a-be2bb6f308c6-192.168.40.243] 2021-11-17 14:47:27,303 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Verifying image storage [1]. Capacity: total=[9 TB], used=[1 TB], threshold=[89.99999761581421%]. 2021-11-17 14:47:27,304 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Verifying image storage [1]. Capacity: total=[9 TB], used=[1 TB], threshold=[89.99999761581421%]. 2021-11-17 14:47:27,306 DEBUG [c.c.n.e.ConfigDriveNetworkElement] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Creating config drive ISO for vm: i-6-83-VM 2021-11-17 14:47:27,309 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=service-offering, content=KVM-RBD-Custom 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=availability-zone, content=Piszki Lab 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=local-hostname, content=Brooklyn 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=local-ipv4, content=192.168.40.243 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=public-ipv4, content=192.168.40.243 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=public-hostname, content=null 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=instance-id, content=6b236c6c-4cae-465b-b1e7-dfeef70d34cb 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=vm-id, content=6b236c6c-4cae-465b-b1e7-dfeef70d34cb 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=public-keys, content=null 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=cloud-identifier, content=CloudStack-{f9bacf5f-af20-4116-8068-7d6a7fc92e26} 2021-11-17 14:47:27,310 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) [createConfigDriveIsoForVM] dataType=metadata, filename=hypervisor-host-name, content=psc1.piszki.lab 2021-11-17 14:47:27,311 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Executing config drive creation command: /usr/bin/genisoimage -o /tmp/configdrive12981467783887013707/i-6-83-VM.iso -ldots -allow-lowercase -allow-multidot -cache-inodes -l -quiet -J -r -V config-2 /tmp/configdrive12981467783887013707 2021-11-17 14:47:27,311 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Executing: /usr/bin/genisoimage -o /tmp/configdrive12981467783887013707/i-6-83-VM.iso -ldots -allow-lowercase -allow-multidot -cache-inodes -l -quiet -J -r -V config-2 /tmp/configdrive12981467783887013707 2021-11-17 14:47:27,312 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Executing while with timeout : 300000 2021-11-17 14:47:27,314 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Execution is successful. 2021-11-17 14:47:27,314 DEBUG [o.a.c.s.c.ConfigDriveBuilder] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Warning: creating filesystem that does not conform to ISO-9660. 2021-11-17 14:47:27,334 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 11-7692992588478939146: Sending { Cmd , MgmtId: 260497154630684, via: 11(s-163-VM), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.HandleConfigDriveIsoCommand":{"isoFile":"configdrive/ i-6-83-VM.iso","create":"true","destStore":{"com.cloud.agent.api.to.NfsTO":{ "_url":"nfs://nfs.piszki.lab/nfs","_role":"Image"}},"useHostCacheOnUnsupport edPool":"true","preferHostCache":"false","wait":"0","bypassHostMaintenance": "false"}}] } 2021-11-17 14:47:27,450 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) (logid:) Seq 11-7692992588478939146: Processing: { Ans: , MgmtId: 260497154630684, via: 11, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"Successfully saved config drive at secondary storage","wait":"0","bypassHostMaintenance":"false"}}] } 2021-11-17 14:47:27,450 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 11-7692992588478939146: Received: { Ans: , MgmtId: 260497154630684, via: 11(s-163-VM), Ver: v1, Flags: 10, { Answer } } 2021-11-17 14:47:27,450 DEBUG [c.c.a.m.AgentManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Details from executing class com.cloud.agent.api.HandleConfigDriveIsoCommand: Successfully saved config drive at secondary storage 2021-11-17 14:47:27,460 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Cleaning up resources for the vm VM instance {id: "83", name: "i-6-83-VM", uuid: "6b236c6c-4cae-465b-b1e7-dfeef70d34cb", type="User"} in Starting state 2021-11-17 14:47:27,467 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 1-8430738502437568545: Sending { Cmd , MgmtId: 260497154630684, via: 1(psc1.piszki.lab), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup": "false","forceStop":"false","vlanToPersistenceMap":{"40":"true"},"volumesToD isconnect":[],"vmName":"i-6-83-VM","executeInSequence":"false","wait":"0","b ypassHostMaintenance":"false"}}] } 2021-11-17 14:47:27,650 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) (logid:) Seq 1-843073850c2437568545: Processing: { Ans: , MgmtId: 260497154630684, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMa intenance":"false"}}] } 2021-11-17 14:47:27,650 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 1-8430738502437568545: Received: { Ans: , MgmtId: 260497154630684, via: 1(psc1.piszki.lab), Ver: v1, Flags: 10, { StopAnswer } } 2021-11-17 14:47:27,655 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Service SecurityGroup is not supported in the network id=212 2021-11-17 14:47:27,658 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Changing active number of nics for network id=212 on -1 2021-11-17 14:47:27,671 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Asking ConfigDrive to release NicProfile[136-83-cd11d70e-92a1-4e93-be1a-be2bb6f308c6-192.168.40.243-null] 2021-11-17 14:47:27,673 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Verifying image storage [1]. Capacity: total=[9 TB], used=[1 TB], threshold=[89.99999761581421%]. 2021-11-17 14:47:27,673 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Verifying image storage [1]. Capacity: total=[9 TB], used=[1 TB], threshold=[89.99999761581421%]. 2021-11-17 14:47:27,675 DEBUG [c.c.n.e.ConfigDriveNetworkElement] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Deleting config drive ISO for vm: i-6-83-VM 2021-11-17 14:47:27,676 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 11-7692992588478939147: Sending { Cmd , MgmtId: 260497154630684, via: 11(s-163-VM), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.HandleConfigDriveIsoCommand":{"isoFile":"configdrive/ i-6-83-VM.iso","create":"false","destStore":{"com.cloud.agent.api.to.NfsTO": {"_url":"nfs://nfs.piszki.lab/nfs","_role":"Image"}},"useHostCacheOnUnsuppor tedPool":"false","preferHostCache":"false","wait":"0","bypassHostMaintenance ":"false"}}] } 2021-11-17 14:47:27,687 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) (logid:) Seq 11-7692992588478939147: Processing: { Ans: , MgmtId: 260497154630684, via: 11, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","wait":"0","bypassHostMainte nance":"false"}}] } 2021-11-17 14:47:27,687 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Seq 11-7692992588478939147: Received: { Ans: , MgmtId: 260497154630684, via: 11(s-163-VM), Ver: v1, Flags: 10, { Answer } } 2021-11-17 14:47:27,687 WARN [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Unable to release some network resources for the VM VM instance {id: "83", name: "i-6-83-VM", uuid: "6b236c6c-4cae-465b-b1e7-dfeef70d34cb", type="User"} in Starting state java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app') at com.cloud.network.element.ConfigDriveNetworkElement.deleteConfigDriveIso(Con figDriveNetworkElement.java:661) at com.cloud.network.element.ConfigDriveNetworkElement.release(ConfigDriveNetwo rkElement.java:167) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.releaseNic(Ne tworkOrchestrator.java:2238) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.release(Netwo rkOrchestrator.java:2164) at com.cloud.vm.VirtualMachineManagerImpl.releaseVmResources(VirtualMachineMana gerImpl.java:1836) at com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.jav a:1826) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:1373) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:5802) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodA ccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.jav a:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManager Impl.java:5967) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext (AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo ntextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa ultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon text(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont ext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont extRunnable.java:45) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJob ManagerImpl.java:568) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java :515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut or.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu tor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-11-17 14:47:27,692 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Successfully released storage resources for the VM VM instance {id: "83", name: "i-6-83-VM", uuid: "6b236c6c-4cae-465b-b1e7-dfeef70d34cb", type="User"} in Starting state 2021-11-17 14:47:27,692 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Successfully cleaned up resources for the VM VM instance {id: "83", name: "i-6-83-VM", uuid: "6b236c6c-4cae-465b-b1e7-dfeef70d34cb", type="User"} in Starting state 2021-11-17 14:47:27,715 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) VM instance {id: "83", name: "i-6-83-VM", uuid: "6b236c6c-4cae-465b-b1e7-dfeef70d34cb", type="User"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: Host {"id": "1", "name": "psc1.piszki.lab", "uuid": "173b2490-2c07-426c-abef-022cd45c3066", "type"="Routing"}, new host: null, host before state transition: Host {"id": "1", "name": "psc1.piszki.lab", "uuid": "173b2490-2c07-426c-abef-022cd45c3066", "type"="Routing"} 2021-11-17 14:47:27,719 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Hosts's actual total CPU: 31200 and CPU after applying overprovisioning: 31200 2021-11-17 14:47:27,719 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Hosts's actual total RAM: (61,66 GB) 66211950592 and RAM after applying overprovisioning: (61,66 GB) 66211950592 2021-11-17 14:47:27,723 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) release cpu from host: 1, old used: 3900,reserved: 0, actual total: 31200, total with overprovisioning: 31200; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse 2021-11-17 14:47:27,727 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) release mem from host: 1, old used: (4,00 GB) 4294967296,reserved: (0 bytes) 0, total: (61,66 GB) 66211950592; new used: (0 bytes) 0,reserved:(0 bytes) 0; movedfromreserved: false,moveToReserveredfalse 2021-11-17 14:47:27,760 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Invocation exception, caused by: java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app') 2021-11-17 14:47:27,760 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915 ctx-e883b663) (logid:b59e4ee2) Rethrow exception java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app') 2021-11-17 14:47:27,761 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 4912 2021-11-17 14:47:27,761 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Unable to complete AsyncJobVO {id:4915, userId: 4, accountId: 4, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB- AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAE AAAAAAAAAAQAAAAAAAAAU3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAO amF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUd C5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHD FmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1Q YXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 260497154630684, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 17 14:47:26 CET 2021, removed: null}, job origin:4912 java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app') at com.cloud.network.element.ConfigDriveNetworkElement.createConfigDriveIso(Con figDriveNetworkElement.java:597) at com.cloud.network.element.ConfigDriveNetworkElement.addPasswordAndUserdata(C onfigDriveNetworkElement.java:225) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElemen t(NetworkOrchestrator.java:1610) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(Ne tworkOrchestrator.java:1936) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo rkOrchestrator.java:1870) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:1201) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:5802) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodA ccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.jav a:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManager Impl.java:5967) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext (AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo ntextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa ultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon text(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont ext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont extRunnable.java:45) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJob ManagerImpl.java:568) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java :515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut or.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu tor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-11-17 14:47:27,770 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Complete async job-4915, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyABxqYXZhLmxhbmcuQ2xhc3NDYXN0RXhjZXB0aW9ugAAFzs5n5VwCAAB4cgAaamF2YS5s YW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0f Pho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2 YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAK c3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRF eGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AN1jbGFzcyBjb20uY2xvdWQuYWdl bnQuYXBpLkFuc3dlciBjYW5ub3QgYmUgY2FzdCB0byBjbGFzcyBjb20uY2xvdWQuYWdlbnQuYXBp LkhhbmRsZUNvbmZpZ0RyaXZlSXNvQW5zd2VyIChjb20uY2xvdWQuYWdlbnQuYXBpLkFuc3dlciBh bmQgY29tLmNsb3VkLmFnZW50LmFwaS5IYW5kbGVDb25maWdEcml2ZUlzb0Fuc3dlciBhcmUgaW4g dW5uYW1lZCBtb2R1bGUgb2YgbG9hZGVyICdhcHAnKXVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNl RWxlbWVudDsCRio8PP0iOQIAAHhwAAAAGnNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRh CcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwA DmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAK bW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAACVXQAA2FwcHQAM2NvbS5j bG91ZC5uZXR3b3JrLmVsZW1lbnQuQ29uZmlnRHJpdmVOZXR3b3JrRWxlbWVudHQAHkNvbmZpZ0Ry aXZlTmV0d29ya0VsZW1lbnQuamF2YXQAFGNyZWF0ZUNvbmZpZ0RyaXZlSXNvcHBzcQB-AAwBAAAA 4XEAfgAOcQB-AA9xAH4AEHQAFmFkZFBhc3N3b3JkQW5kVXNlcmRhdGFwcHNxAH4ADAEAAAZKcQB- AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uTmV0d29ya09y Y2hlc3RyYXRvcnQAGE5ldHdvcmtPcmNoZXN0cmF0b3IuamF2YXQADnByZXBhcmVFbGVtZW50cHBz cQB-AAwBAAAHkHEAfgAOcQB-ABVxAH4AFnQACnByZXBhcmVOaWNwcHNxAH4ADAEAAAdOcQB-AA5x AH4AFXEAfgAWdAAHcHJlcGFyZXBwc3EAfgAMAQAABLFxAH4ADnQAJmNvbS5jbG91ZC52bS5WaXJ0 dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQ b3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAQAAFqpxAH4ADnEAfgAdcQB-AB5xAH4AH3Bwc3EAfgAM Av____5wdAAtamRrLmludGVybmFsLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAd TmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2UwdAAJamF2YS5iYXNldAAHMTEu MC4xM3NxAH4ADAIAAAA-cHEAfgAicQB-ACN0AAZpbnZva2VxAH4AJXEAfgAmc3EAfgAMAgAAACtw dAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURl bGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAocQB-ACVxAH4AJnNxAH4ADAIAAAI2 cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-AChxAH4AJXEAfgAm c3EAfgAMAQAAAGtxAH4ADnQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpW bVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgAMAQAAF09x AH4ADnEAfgAdcQB-AB5xAH4AMnBwc3EAfgAMAQAAAGZxAH4ADnQAIGNvbS5jbG91ZC52bS5WbVdv cmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB- AAwBAAACbHEAfgAOdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwu QXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5D b250ZXh0cHBzcQB-AAwBAAAAMHEAfgAOdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQu Y29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxl LmphdmF0AANydW5wcHNxAH4ADAEAAAA3cQB-AA50AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu YWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdl ZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4ADAEAAABmcQB-AA50AEBvcmcuYXBhY2hlLmNsb3Vk c3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-AEJ0AA9j YWxsV2l0aENvbnRleHRwcHNxAH4ADAEAAAA0cQB-AA5xAH4ARXEAfgBCdAAOcnVuV2l0aENvbnRl eHRwcHNxAH4ADAEAAAAtcQB-AA50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250 ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4APnEAfgA_cHBzcQB-AAwBAAACOHEAfgAOcQB- ADlxAH4AOnEAfgA_cHBzcQB-AAwCAAACA3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRv cnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AQ3EAfgAlcQB-ACZzcQB-AAwC AAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh cQB-AD9xAH4AJXEAfgAmc3EAfgAMAgAABGhwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk UG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AJXEA fgAmc3EAfgAMAgAAAnRwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y JFdvcmtlcnEAfgBUcQB-AD9xAH4AJXEAfgAmc3EAfgAMAgAAAz1wdAAQamF2YS5sYW5nLlRocmVh ZHQAC1RocmVhZC5qYXZhcQB-AD9xAH4AJXEAfgAmc3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRF bXB0eUxpc3R6uBe0PKee3gIAAHhweA 2021-11-17 14:47:27,771 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Publish async job-4915 complete on message bus 2021-11-17 14:47:27,771 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Wake up jobs related to job-4915 2021-11-17 14:47:27,771 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Update db status for job-4915 2021-11-17 14:47:27,772 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Wake up jobs joined with job-4915 and disjoin all subjobs created from job- 4915 2021-11-17 14:47:27,786 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Done executing com.cloud.vm.VmWorkStart for job-4915 2021-11-17 14:47:27,787 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-16dd5098 job-4912/job-4915) (logid:b59e4ee2) Remove job-4915 from job monitoring 2021-11-17 14:47:27,805 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-10:null) (logid:da9a8ffa) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.182 -p 623 -U piotr -P Seneka22 chassis power status]: [Chassis Power is on ]. 2021-11-17 14:47:27,805 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-10:null) (logid:da9a8ffa) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.182 -p 623 -U piotr -P Seneka22 chassis power status]: [Unable to Get Channel Cipher Suites Running Get PICMG Properties my_addr 0x20, transit 0, target 0x20 Error response 0xc1 from Get PICMG Properities Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20 Invalid completion code received: Invalid command Discovered IPMB address 0x0 ]. 2021-11-17 14:47:27,805 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-2-thread-10:null) (logid:da9a8ffa) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.182 -p 623 -U piotr -P Seneka22 chassis power status] was successful and got the result [Chassis Power is on]. 2021-11-17 14:47:27,806 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-11:null) (logid:da9a8ffa) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.181 -p 623 -U piotr -P Seneka22 chassis power status]: [Chassis Power is on ]. 2021-11-17 14:47:27,806 DEBUG [o.a.c.u.p.ProcessRunner] (pool-2-thread-11:null) (logid:da9a8ffa) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.181 -p 623 -U piotr -P Seneka22 chassis power status]: [Unable to Get Channel Cipher Suites Running Get PICMG Properties my_addr 0x20, transit 0, target 0x20 Error response 0xc1 from Get PICMG Properities Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20 Invalid completion code received: Invalid command Discovered IPMB address 0x0 ]. 2021-11-17 14:47:27,806 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-2-thread-11:null) (logid:da9a8ffa) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.0.181 -p 623 -U piotr -P Seneka22 chassis power status] was successful and got the result [Chassis Power is on]. 2021-11-17 14:47:27,811 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app') at com.cloud.network.element.ConfigDriveNetworkElement.createConfigDriveIso(Con figDriveNetworkElement.java:597) at com.cloud.network.element.ConfigDriveNetworkElement.addPasswordAndUserdata(C onfigDriveNetworkElement.java:225) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElemen t(NetworkOrchestrator.java:1610) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(Ne tworkOrchestrator.java:1936) at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo rkOrchestrator.java:1870) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:1201) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage rImpl.java:5802) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodA ccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Delegatin gMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.jav a:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManager Impl.java:5967) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext (AsyncJobManagerImpl.java:620) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo ntextRunnable.java:48) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa ultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon text(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont ext(DefaultManagedContext.java:52) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont extRunnable.java:45) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJob ManagerImpl.java:568) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java :515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut or.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu tor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2021-11-17 14:47:27,819 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Complete async job-4912, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"er rorcode":"530","errortext":"class com.cloud.agent.api.Answer cannot be cast to class com.cloud.agent.api.HandleConfigDriveIsoAnswer (com.cloud.agent.api.Answer and com.cloud.agent.api.HandleConfigDriveIsoAnswer are in unnamed module of loader 'app')"} 2021-11-17 14:47:27,820 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Publish async job-4912 complete on message bus 2021-11-17 14:47:27,820 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Wake up jobs related to job-4912 2021-11-17 14:47:27,820 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Update db status for job-4912 2021-11-17 14:47:27,821 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-a3c415d6 job-4912) (logid:b59e4ee2) Wake up jobs joined with job-4912 and disjoin all subjobs created