Pretty dead in the water here. Hopefully someone can give me a hint. Also, the docs mentioned java 17 being required, yet, 17 didn’t allow the manager to even start. This is on Rocky 9. I had to downgrade to java 11.
> On Monday, Feb 02, 2026 at 7:27 PM, Jeremy Hansen <[email protected] > (mailto:[email protected])> wrote: > 2026-02-03 03:25:42,761 WARN [utils.script.Script] > (AgentRequest-Handler-1:[]) (logid:e7581b89) Execution of process [1495568] > for command [qemu-img convert -O raw -U --image-opts > driver=qcow2,file.filename=/mnt/2c48f02e-c900-3f17-8de1-677e1e7d7af5/c47e8c51-83f4-3871-bc3d-4f87536d9c82.qcow2 > > rbd:csclx/4738f069-9bb0-4341-80d2-1e8baa96f8ae:mon_host=mon.ceph:auth_supported=cephx:id=csclx:key=******:rbd_default_format=2:client_mount_timeout=30 > ] failed. > 2026-02-03 03:25:42,761 WARN [utils.script.Script] > (AgentRequest-Handler-1:[]) (logid:e7581b89) Process [1495568] for command > [qemu-img convert -O raw -U --image-opts > driver=qcow2,file.filename=/mnt/2c48f02e-c900-3f17-8de1-677e1e7d7af5/c47e8c51-83f4-3871-bc3d-4f87536d9c82.qcow2 > > rbd:csclx/4738f069-9bb0-4341-80d2-1e8baa96f8ae:mon_host=mon.ceph:auth_supported=cephx:id=csclx:key=******:rbd_default_format=2:client_mount_timeout=30 > ] encountered the error: [qemu-img: > rbd:csclx/4738f069-9bb0-4341-80d2-1e8baa96f8ae:mon_host=mon.ceph:auth_supported=cephx:id=csclx:key=AQA5UKJlNsNmDhAA4R0kWpPCXmXMX11Rpjet7g==:rbd_default_format=2:client_mount_timeout=30: > error while converting raw: error rbd create: File exists]. > 2026-02-03 03:25:42,762 ERROR [kvm.storage.LibvirtStorageAdaptor] > (AgentRequest-Handler-1:[]) (logid:e7581b89) Failed to convert from > /mnt/2c48f02e-c900-3f17-8de1-677e1e7d7af5/c47e8c51-83f4-3871-bc3d-4f87536d9c82.qcow2 > to > rbd:csclx/4738f069-9bb0-4341-80d2-1e8baa96f8ae:mon_host=mon.ceph:auth_supported=cephx:id=csclx:key=AQA5UKJlNsNmDhAA4R0kWpPCXmXMX11Rpjet7g==:rbd_default_format=2:client_mount_timeout=30 > the error was: qemu-img: > rbd:csclx/4738f069-9bb0-4341-80d2-1e8baa96f8ae:mon_host=mon.ceph:auth_supported=cephx:id=csclx:key=AQA5UKJlNsNmDhAA4R0kWpPCXmXMX11Rpjet7g==:rbd_default_format=2:client_mount_timeout=30: > error while converting raw: error rbd create: File exists > 2026-02-03 03:25:42,762 INFO [kvm.storage.LibvirtStorageAdaptor] > (AgentRequest-Handler-1:[]) (logid:e7581b89) Attempting to remove storage > pool 2c48f02e-c900-3f17-8de1-677e1e7d7af5 from libvirt > > ==> agent.err <== > libvirt: Secrets Driver error : Secret not found: no secret with matching > uuid '2c48f02e-c900-3f17-8de1-677e1e7d7af5' > > > > > > On Monday, Feb 02, 2026 at 6:47 PM, Jeremy Hansen <[email protected] > > (mailto:[email protected])> wrote: > > I went through the upgrade process to get my cloudstack cluster up to 4.22. > > Now my system VMs are continually restarting. How do I debug this. I see > > these errorrs in the logs: > > > > 2026-02-03 02:39:13,149 WARN [c.c.v.ClusteredVirtualMachineManagerImpl] > > (Work-Job-Executor-8:[ctx-b6f69008, job-1332/job-1342, ctx-c2b0bf26]) > > (logid:a730585f) Unable to contact resource. > > com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] > > is unreachable: Unable to create volume > > [{"name":"ROOT-211","uuid":"359e21ea-713a-48c2-b031-5f804d9bbcfe"}] due to > > [com.cloud.utils.exception.CloudRuntimeException: Failed to copy > > /mnt/2c48f02e-c900-3f17-8de1-677e1e7d7af5/c47e8c51-83f4-3871-bc3d-4f87536d9c82.qcow2 > > to 4738f069-9bb0-4341-80d2-1e8baa96f8ae]. > > at > > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1873) > > at > > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:2024) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > at > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > > at > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > > at > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > > at com.sun.proxy.$Proxy264.prepare(Unknown Source) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1485) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5951) > > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102) > > at > > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtchineManagerImpl.java:6075) > > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646) > > at > > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:829) > > > > > > I left running VM up while upgrading. This seems storage related but I > > haven’t changed anything on the storage side. > > > > Any ideas? > > > > Thanks > > -jeremy > > > > > >
signature.asc
Description: PGP signature
