Hi,
What storage do you use ?
-Wei
On Tue, Dec 3, 2024 at 3:52 PM Jeroen Kleijer <jeroen.klei...@gmail.com>
wrote:
> Hi all,
>
> We've set up a new Apache CloudStack instance (4.19.1.2) and are
> experimenting with it.
> While we're able to migrate instances between the hosts of a single cluster
> without issues, if we try to migrate an instance from one cluster to
> another, we get the following error:
> 2024-12-03 08:02:29,431 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Unexpected
> exception while executing
>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
> instance
>
> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
> along with its volumes due to [java.lang.ClassCastException: class
> com.cloud.agent.api.Answer cannot be cast to class
> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
> 'app')].
> at
>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1451)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> at
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> at com.sun.proxy.$Proxy241.migrateVolumes(Unknown Source)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3284)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5534)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
> at
>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
> at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2024-12-03 08:02:29,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Complete async
> job-862, jobStatus: FAILED, resultCode: 530, result:
>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> to migrate VM [VM instance
>
> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
> along with its volumes due to [java.lang.ClassCastException: class
> com.cloud.agent.api.Answer cannot be cast to class
> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
> 'app')]."}
> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Publish async
> job-862 complete on message bus
> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
> related to job-862
> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Update db
> status for job-862
> 2024-12-03 08:02:29,437 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
> joined with job-862 and disjoin all subjobs created from job- 862
> 2024-12-03 08:02:29,443 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Done executing
>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> for job-862
> 2024-12-03 08:02:29,443 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Remove job-862
> from job monitoring
> 2024-12-03 08:02:29,666 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Starting to remove all
> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
> 2024-12-03 08:02:29,666 INFO [c.c.s.d.VolumeStatsDaoImpl]
> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Removed a total of [0]
> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
>
> It's a very simple Rocky Linux VM running on KVM hypervisors. As mentioned,
> migrating between hypervisors works perfectly but migrating between
> clusters (with storage migration enabled) results in the error above.
> Has anyone experienced this before?
>
>
> Kind regards,
>
> Jeroen Kleijer
>
