Hi Jeroen,
Just to confirm, this is a fresh ACS 4.19.1.2 env and not an upgraded env? I
ask this to validate that the agent version on the hypervisor hosts is the same
as the management server.
Thanks,
Pearl
________________________________
From: Wei ZHOU <ustcweiz...@gmail.com>
Sent: December 3, 2024 11:09 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Unable to migrate between clusters
No, it is a very common setup.
Any errors on the kvm host ?
-Wei
On Tue, Dec 3, 2024 at 4:51 PM Jeroen Kleijer <jeroen.klei...@gmail.com>
wrote:
> We’re using NFS based storage that is cluster scoped. Does that make a
> difference?
>
> Kind regards,
>
> Jeroen Kleijer
>
> > On 3 Dec 2024, at 16:15, Wei ZHOU <ustcweiz...@gmail.com> wrote:
> >
> > Hi,
> >
> > What storage do you use ?
> >
> > -Wei
> >
> >> On Tue, Dec 3, 2024 at 3:52 PM Jeroen Kleijer <jeroen.klei...@gmail.com
> >
> >> wrote:
> >>
> >> Hi all,
> >>
> >> We've set up a new Apache CloudStack instance (4.19.1.2) and are
> >> experimenting with it.
> >> While we're able to migrate instances between the hosts of a single
> cluster
> >> without issues, if we try to migrate an instance from one cluster to
> >> another, we get the following error:
> >> 2024-12-03 08:02:29,431 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Unexpected
> >> exception while executing
> >>
> >>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM
> [VM
> >> instance
> >>
> >>
> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
> >> along with its volumes due to [java.lang.ClassCastException: class
> >> com.cloud.agent.api.Answer cannot be cast to class
> >> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
> >> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
> >> 'app')].
> >> at
> >>
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1451)
> >> at
> >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >> at
> >>
> >>
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >> at
> >>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >> at
> >>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >> at
> >>
> >>
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
> >> at
> >>
> >>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >> at
> >>
> >>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
> >> at com.sun.proxy.$Proxy241.migrateVolumes(Unknown Source)
> >> at
> >>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3284)
> >> at
> >>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5534)
> >> at
> >> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> Method)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> at
> >>
> >>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >> at
> >>
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
> >> at
> >>
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
> >> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >> at
> >>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
> >> at
> >>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >> at
> >>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >> at
> >>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >> at
> >>
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >> at
> >>
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >> at
> >>
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
> >> at
> >>
> >>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >> at
> >> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >> at
> >>
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >> at
> >>
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >> at java.base/java.lang.Thread.run(Thread.java:829)
> >> 2024-12-03 08:02:29,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Complete
> async
> >> job-862, jobStatus: FAILED, resultCode: 530, result:
> >>
> >>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
> >> to migrate VM [VM instance
> >>
> >>
> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
> >> along with its volumes due to [java.lang.ClassCastException: class
> >> com.cloud.agent.api.Answer cannot be cast to class
> >> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
> >> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
> >> 'app')]."}
> >> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Publish
> async
> >> job-862 complete on message bus
> >> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
> >> related to job-862
> >> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Update db
> >> status for job-862
> >> 2024-12-03 08:02:29,437 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
> >> joined with job-862 and disjoin all subjobs created from job- 862
> >> 2024-12-03 08:02:29,443 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Done
> executing
> >>
> >>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> >> for job-862
> >> 2024-12-03 08:02:29,443 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Remove
> job-862
> >> from job monitoring
> >> 2024-12-03 08:02:29,666 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
> >> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Starting to remove all
> >> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
> >> 2024-12-03 08:02:29,666 INFO [c.c.s.d.VolumeStatsDaoImpl]
> >> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Removed a total of [0]
> >> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
> >>
> >> It's a very simple Rocky Linux VM running on KVM hypervisors. As
> mentioned,
> >> migrating between hypervisors works perfectly but migrating between
> >> clusters (with storage migration enabled) results in the error above.
> >> Has anyone experienced this before?
> >>
> >>
> >> Kind regards,
> >>
> >> Jeroen Kleijer
> >>
>
