Hi all,
This is indeed a completely fresh installation, not an upgrade. The strange
thing is, I don’t see anything happening at all on my agents.log files on the
source or destination hosts. The only place where anything is logged is the mgt
host which is what I sent. (Not much to go on I’m afraid)
Kind regards,
Jeroen Kleijer
> On 3 Dec 2024, at 17:28, Pearl d'Silva <pearl.dsi...@shapeblue.com> wrote:
>
> Hi Jeroen,
>
> Just to confirm, this is a fresh ACS 4.19.1.2 env and not an upgraded env? I
> ask this to validate that the agent version on the hypervisor hosts is the
> same as the management server.
>
> Thanks,
> Pearl
>
>
>
> ________________________________
> From: Wei ZHOU <ustcweiz...@gmail.com>
> Sent: December 3, 2024 11:09 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Unable to migrate between clusters
>
> No, it is a very common setup.
>
> Any errors on the kvm host ?
>
>
> -Wei
>
>> On Tue, Dec 3, 2024 at 4:51 PM Jeroen Kleijer <jeroen.klei...@gmail.com>
>> wrote:
>>
>> We’re using NFS based storage that is cluster scoped. Does that make a
>> difference?
>>
>> Kind regards,
>>
>> Jeroen Kleijer
>>
>>>> On 3 Dec 2024, at 16:15, Wei ZHOU <ustcweiz...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> What storage do you use ?
>>>
>>> -Wei
>>>
>>>> On Tue, Dec 3, 2024 at 3:52 PM Jeroen Kleijer <jeroen.klei...@gmail.com
>>>
>>>> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> We've set up a new Apache CloudStack instance (4.19.1.2) and are
>>>> experimenting with it.
>>>> While we're able to migrate instances between the hosts of a single
>> cluster
>>>> without issues, if we try to migrate an instance from one cluster to
>>>> another, we get the following error:
>>>> 2024-12-03 08:02:29,431 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Unexpected
>>>> exception while executing
>>>>
>>>>
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM
>> [VM
>>>> instance
>>>>
>>>>
>> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
>>>> along with its volumes due to [java.lang.ClassCastException: class
>>>> com.cloud.agent.api.Answer cannot be cast to class
>>>> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
>>>> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
>>>> 'app')].
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1451)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> at
>>>>
>>>>
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>>
>>>>
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> at
>>>>
>>>>
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>>>> at
>>>>
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>>>> at
>>>>
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>>>> at
>>>>
>>>>
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
>>>> at
>>>>
>>>>
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>>>> at
>>>>
>>>>
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
>>>> at com.sun.proxy.$Proxy241.migrateVolumes(Unknown Source)
>>>> at
>>>>
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3284)
>>>> at
>>>>
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5534)
>>>> at
>>>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> at
>>>>
>>>>
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>>
>>>>
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> at
>>>>
>>>>
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:105)
>>>> at
>>>>
>>>>
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5610)
>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>> at
>>>>
>>>>
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
>>>> at
>>>>
>>>>
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> at
>>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>> at
>>>>
>>>>
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> at
>>>>
>>>>
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>> 2024-12-03 08:02:29,434 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Complete
>> async
>>>> job-862, jobStatus: FAILED, resultCode: 530, result:
>>>>
>>>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
>>>> to migrate VM [VM instance
>>>>
>>>>
>> {"id":8,"instanceName":"i-2-8-VM","type":"User","uuid":"64c2ffbb-d7e9-4264-9a31-d9e79bce1ef7"}]
>>>> along with its volumes due to [java.lang.ClassCastException: class
>>>> com.cloud.agent.api.Answer cannot be cast to class
>>>> com.cloud.agent.api.ModifyTargetsAnswer (com.cloud.agent.api.Answer and
>>>> com.cloud.agent.api.ModifyTargetsAnswer are in unnamed module of loader
>>>> 'app')]."}
>>>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Publish
>> async
>>>> job-862 complete on message bus
>>>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
>>>> related to job-862
>>>> 2024-12-03 08:02:29,435 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Update db
>>>> status for job-862
>>>> 2024-12-03 08:02:29,437 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Wake up jobs
>>>> joined with job-862 and disjoin all subjobs created from job- 862
>>>> 2024-12-03 08:02:29,443 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Done
>> executing
>>>>
>>>>
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>>>> for job-862
>>>> 2024-12-03 08:02:29,443 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-84:ctx-87ebdcb5 job-862) (logid:485fa37d) Remove
>> job-862
>>>> from job monitoring
>>>> 2024-12-03 08:02:29,666 DEBUG [c.c.s.d.VolumeStatsDaoImpl]
>>>> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Starting to remove all
>>>> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
>>>> 2024-12-03 08:02:29,666 INFO [c.c.s.d.VolumeStatsDaoImpl]
>>>> (StatsCollector-5:ctx-1755907b) (logid:39acf2c5) Removed a total of [0]
>>>> volume_stats rows older than [Mon Dec 02 20:02:29 CET 2024].
>>>>
>>>> It's a very simple Rocky Linux VM running on KVM hypervisors. As
>> mentioned,
>>>> migrating between hypervisors works perfectly but migrating between
>>>> clusters (with storage migration enabled) results in the error above.
>>>> Has anyone experienced this before?
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Jeroen Kleijer
>>>>
>>
