We’re using NFS based storage that is cluster scoped. Does that make a
difference?
Kind regards,
Jeroen Kleijer
> On 3 Dec 2024, at 16:15, Wei ZHOU <ustcweiz...@gmail.com> wrote:
>
> Hi,
>
> What storage do you use ?
>
> -Wei
>
>> On Tue, Dec 3, 2024 at 3:52 PM Jeroen Kleijer <jeroen.klei...@gmail.com>
>> wrote:
>>
>> Hi all,
>>
>> We've set up a new Apache CloudStack instance (4.19.1.2) and are
>> experimenting with it.
>> While we're able to migrate instances between the hosts of a single cluster
>> without issues, if we try to migrate an instance from one cluster to
>> another, we get the following error:
>> 2024-12-03 08:02:29,431 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Unexpected
>> exception while executing
>>
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
>> instance
>>
>> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
>> along with its volumes due to [java.lang.ClassCastException: class
>> com.cloud.agent.api.Answer cannot be cast to class
>> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
>> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
>> 'app')].
>> at
>>
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1451)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>>
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>>
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> at
>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>> at
>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>> at
>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>> at com.sun.proxy.$Proxy241.migrateVolumes(Unknown Source)
>> at
>>
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3284)
>> at
>>
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5534)
>> at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> at
>>
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>>
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>>
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
>> at
>>
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at
>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>> at
>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> at
>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> at
>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> at
>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> at
>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> at
>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>> at
>>
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2024-12-03 08:02:29,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Complete async
>> job-862, jobStatus: FAILED, resultCode: 530, result:
>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
>> to migrate VM [VM instance
>>
>> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
>> along with its volumes due to [java.lang.ClassCastException: class
>> com.cloud.agent.api.Answer cannot be cast to class
>> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
>> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
>> 'app')]."}
>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Publish async
>> job-862 complete on message bus
>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
>> related to job-862
>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Update db
>> status for job-862
>> 2024-12-03 08:02:29,437 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
>> joined with job-862 and disjoin all subjobs created from job- 862
>> 2024-12-03 08:02:29,443 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Done executing
>>
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>> for job-862
>> 2024-12-03 08:02:29,443 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Remove job-862
>> from job monitoring
>> 2024-12-03 08:02:29,666 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
>> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Starting to remove all
>> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
>> 2024-12-03 08:02:29,666 INFO [c.c.s.d.VolumeStatsDaoImpl]
>> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Removed a total of [0]
>> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
>>
>> It's a very simple Rocky Linux VM running on KVM hypervisors. As mentioned,
>> migrating between hypervisors works perfectly but migrating between
>> clusters (with storage migration enabled) results in the error above.
>> Has anyone experienced this before?
>>
>>
>> Kind regards,
>>
>> Jeroen Kleijer
>>
