Hi all,
We've set up a new Apache CloudStack instance (4.19.1.2) and are
experimenting with it.
While we're able to migrate instances between the hosts of a single cluster
without issues, if we try to migrate an instance from one cluster to
another, we get the following error:
2024-12-03 08:02:29,431 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
instance
{"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
along with its volumes due to [java.lang.ClassCastException: class
com.cloud.agent.api.Answer cannot be cast to class
com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
'app')].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1451)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy241.migrateVolumes(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3284)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5534)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-12-03 08:02:29,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Complete async
job-862, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to migrate VM [VM instance
{"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
along with its volumes due to [java.lang.ClassCastException: class
com.cloud.agent.api.Answer cannot be cast to class
com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
'app')]."}
2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Publish async
job-862 complete on message bus
2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
related to job-862
2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Update db
status for job-862
2024-12-03 08:02:29,437 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
joined with job-862 and disjoin all subjobs created from job- 862
2024-12-03 08:02:29,443 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
for job-862
2024-12-03 08:02:29,443 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Remove job-862
from job monitoring
2024-12-03 08:02:29,666 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
(StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Starting to remove all
volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
2024-12-03 08:02:29,666 INFO [c.c.s.d.VolumeStatsDaoImpl]
(StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Removed a total of [0]
volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
It's a very simple Rocky Linux VM running on KVM hypervisors. As mentioned,
migrating between hypervisors works perfectly but migrating between
clusters (with storage migration enabled) results in the error above.
Has anyone experienced this before?
Kind regards,
Jeroen Kleijer
