Hi Wei,
See attached error log from when trying to migrate a vm that was migrated
before from the standalone host.
--
Jimmy
Van: Wei ZHOU <ustcweiz...@gmail.com>
Datum: zaterdag, 23 maart 2024 om 10:14
Aan: users@cloudstack.apache.org <users@cloudstack.apache.org>
Onderwerp: Re: Migrate migrated instance to another host
Hi,
/var/log/cloudstack/management/management-server.log
please note the log file is rotated everyday.
-Wei
On Sat, Mar 23, 2024 at 9:34 AM Jimmy Huybrechts <ji...@linservers.com> wrote:
>
> Hi Wei,
>
> I can, if you tell me where to look as I couldn’t find anything on them in
> the logs.
>
> --
> Jimmy
>
> Van: Wei ZHOU <ustcweiz...@gmail.com>
> Datum: vrijdag, 22 maart 2024 om 16:15
> Aan: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Onderwerp: Re: Migrate migrated instance to another host
> Can you share the logs of vm migration and vm import ?
>
>
> On Fri, Mar 22, 2024 at 3:55 PM Jimmy Huybrechts <ji...@linservers.com> wrote:
> >
> > I’ve migrated vm’s from a remote KVM host which worked fine and they run on
> > Cloudstack now.
> >
> > However, I just tried migrating one from one cloudstack host to another,
> > just like with instances made from a cloudstack host and it’s not moving
> > but ending up with an error.
> >
> > This is displayed in the webgui:
> > Failed to migrate VM [VM instance
> > {"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
> > along with its volumes due to [java.lang.NullPointerException].
> >
> > Now I’m trying to see in what log the complete error would be but I’m
> > coming up blank on which log I need.
> >
> > Instances created natively in Cloudstack are migrating just fine, it’s only
> > the ones that came from the remote KVM host before.
> >
> > --
> > Jimmy
024-03-24 00:28:17,779 DEBUG [o.a.c.s.m.StorageSystemDataMotionStrategy]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Copying template [212] of volume [122] from source storage
pool [5] to target storage pool [3].
2024-03-24 00:28:17,788 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) template 212 is not in store:2, type:Image
2024-03-24 00:28:17,799 DEBUG [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Failed to copy volume
java.lang.NullPointerException
at
org.apache.cloudstack.storage.image.store.TemplateObject.getId(TemplateObject.java:112)
at
org.apache.cloudstack.storage.datastore.ObjectInDataStoreManagerImpl.findObject(ObjectInDataStoreManagerImpl.java:354)
at
org.apache.cloudstack.storage.image.store.TemplateObject.getInstallPath(TemplateObject.java:366)
at
org.apache.cloudstack.storage.to.TemplateObjectTO.<init>(TemplateObjectTO.java:71)
at
org.apache.cloudstack.storage.motion.KvmNonManagedStorageDataMotionStrategy.copyTemplateToTargetFilesystemStorageIfNeeded(KvmNonManagedStorageDataMotionStrategy.java:224)
at
org.apache.cloudstack.storage.motion.StorageSystemDataMotionStrategy.copyAsync(StorageSystemDataMotionStrategy.java:1937)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:135)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:2307)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1441)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolumes(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3267)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5505)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-03-24 00:28:17,801 ERROR [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Failed to migrate VM [VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException].
2024-03-24 00:28:17,801 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Migration was unsuccessful. Cleaning up: VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}
2024-03-24 00:28:17,811 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Releasing [1] volumes for VM [VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
from host [Host
{"id":7,"name":"vmh1","type":"Routing","uuid":"f710d4ba-5ef7-4d46-b870-af8950044e5d"}].
2024-03-24 00:28:17,819 WARN [c.c.a.AlertManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) alertType=[17] dataCenterId=[2] podId=[2] clusterId=[null]
message=[Unable to migrate vm i-8-108-VM from host vmh3 in zone Amsterdam and
pod Amsterdam].
2024-03-24 00:28:17,821 DEBUG [c.c.a.AlertManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Have already sent: 1 emails for alert type '17' -- skipping
send email
2024-03-24 00:28:17,823 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}
is a [User], returning null for control Nic IP.
2024-03-24 00:28:17,838 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Seq 7-5826813492887328392: Routed from 104023939828503
2024-03-24 00:28:17,838 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Seq 7-5826813492887328392: Sending { Cmd , MgmtId:
104023939828503, via: 7(vmh1), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","forceStop":"false","vlanToPersistenceMap":{"3903":"true"},"volumesToDisconnect":[],"vmName":"i-8-108-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-03-24 00:28:17,851 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}
state transited from [Migrating] to [Running] with event [OperationFailed].
VM's original host: Host
{"id":16,"name":"vmh3","type":"Routing","uuid":"aa060b6f-0e98-41cb-b51b-d7b5584c5312"},
new host: Host
{"id":16,"name":"vmh3","type":"Routing","uuid":"aa060b6f-0e98-41cb-b51b-d7b5584c5312"},
host before state transition: Host
{"id":7,"name":"vmh1","type":"Routing","uuid":"f710d4ba-5ef7-4d46-b870-af8950044e5d"}
2024-03-24 00:28:17,862 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Hosts's actual total CPU: 83200 and CPU after applying
overprovisioning: 332800
2024-03-24 00:28:17,862 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Hosts's actual total RAM: (250.75 GB) 269244112896 and RAM
after applying overprovisioning: (250.75 GB) 269244104704
2024-03-24 00:28:17,862 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) release cpu from host: 7, old used: 8000,reserved: 0, actual
total: 83200, total with overprovisioning: 332800; new used: 4000,reserved:0;
movedfromreserved: false,moveToReserveredfalse
2024-03-24 00:28:17,862 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) release mem from host: 7, old used: (5.95 GB)
6392119296,reserved: (0 bytes) 0, total: (250.75 GB) 269244104704; new used:
(4.00 GB) 4294967296,reserved:(0 bytes) 0; movedfromreserved:
false,moveToReserveredfalse
2024-03-24 00:28:17,870 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException].
2024-03-24 00:28:17,870 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981 ctx-23d85860)
(logid:61b73463) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException].
2024-03-24 00:28:17,870 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Done with
run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for VM 109, job
origin: 980
2024-03-24 00:28:17,870 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Unable to
complete AsyncJobVO: {id:981, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAABtdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAAAHAAAAAAAAABBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAAdwgAAAAQAAAAAHg,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 104023939828503, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Sun Mar 24 00:28:16 CET 2024, removed: null}, job
origin:980
com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1447)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolumes(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3267)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5505)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-03-24 00:28:17,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Complete
async job-981, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AMVGYWlsZWQgdG8gbWlncmF0ZSBWTSBbVk0gaW5zdGFuY2UgeyJpZCI6MTA5LCJpbnN0YW5jZU5hbWUiOiJpLTgtMTA4LVZNIiwidHlwZSI6IlVzZXIiLCJ1dWlkIjoiMmZiYjBkODQtN2VhMi00MDkyLWFhNWQtMjVhNTNlYjNjZGEyIn1dIGFsb25nIHdpdGggaXRzIHZvbHVtZXMgZHVlIHRvIFtqYXZhLmxhbmcuTnVsbFBvaW50ZXJFeGNlcHRpb25dLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAIXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAFp3QAA2FwcHQAPW9yZy5hcGFjaGUuY2xvdWRzdGFjay5lbmdpbmUub3JjaGVzdHJhdGlvbi5Wb2x1bWVPcmNoZXN0cmF0b3J0ABdWb2x1bWVPcmNoZXN0cmF0b3IuamF2YXQADm1pZ3JhdGVWb2x1bWVzcHBzcQB-AAwC_____nB0AC1qZGsuaW50ZXJuYWwucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YXQAB2ludm9rZTB0AAlqYXZhLmJhc2V0AAcxMS4wLjIxc3EAfgAMAgAAAD5wcQB-ABNxAH4AFHQABmludm9rZXEAfgAWcQB-ABdzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABlxAH4AFnEAfgAXc3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AGXEAfgAWcQB-ABdzcQB-AAwBAAABWHEAfgAOdAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25wcHNxAH4ADAEAAADGcQB-AA50ADxvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb250AB9SZWZsZWN0aXZlTWV0aG9kSW52b2NhdGlvbi5qYXZhdAAPaW52b2tlSm9pbnBvaW50cHBzcQB-AAwBAAAAo3EAfgAOcQB-ACVxAH4AJnQAB3Byb2NlZWRwcHNxAH4ADAEAAABhcQB-AA50AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAZcHBzcQB-AAwBAAAAunEAfgAOcQB-ACVxAH4AJnEAfgApcHBzcQB-AAwBAAAA13EAfgAOdAA0b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLkpka0R5bmFtaWNBb3BQcm94eXQAF0pka0R5bmFtaWNBb3BQcm94eS5qYXZhcQB-ABlwcHNxAH4ADAD_____cHQAF2NvbS5zdW4ucHJveHkuJFByb3h5MjI3cHEAfgARcHBzcQB-AAwBAAAMw3EAfgAOdAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0AB1vcmNoZXN0cmF0ZU1pZ3JhdGVXaXRoU3RvcmFnZXBwc3EAfgAMAQAAFYFxAH4ADnEAfgA0cQB-ADVxAH4ANnBwc3EAfgAMAv____5wcQB-ABNxAH4AFHEAfgAVcQB-ABZxAH4AF3NxAH4ADAIAAAA-cHEAfgATcQB-ABRxAH4AGXEAfgAWcQB-ABdzcQB-AAwCAAAAK3BxAH4AG3EAfgAccQB-ABlxAH4AFnEAfgAXc3EAfgAMAgAAAjZwcQB-AB5xAH4AH3EAfgAZcQB-ABZxAH4AF3NxAH4ADAEAAABrcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABXNcQB-AA5xAH4ANHEAfgA1cQB-AD9wcHNxAH4ADAEAAABmcQB-AA50ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgAMAQAAAo5xAH4ADnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgAMAQAAADBxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-AAwBAAAAN3EAfgAOdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-AAwBAAAAZnEAfgAOdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgBPdAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-AAwBAAAANHEAfgAOcQB-AFJxAH4AT3QADnJ1bldpdGhDb250ZXh0cHBzcQB-AAwBAAAALXEAfgAOdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-AEtxAH4ATHBwc3EAfgAMAQAAAlpxAH4ADnEAfgBGcQB-AEdxAH4ATHBwc3EAfgAMAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AFBxAH4AFnEAfgAXc3EAfgAMAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgBMcQB-ABZxAH4AF3NxAH4ADAIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-ABZxAH4AF3NxAH4ADAIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4AYXEAfgBMcQB-ABZxAH4AF3NxAH4ADAIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgBMcQB-ABZxAH4AF3NyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHgAABCadwgAAAAAAAAAAHg
2024-03-24 00:28:17,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Publish
async job-981 complete on message bus
2024-03-24 00:28:17,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Wake up
jobs related to job-981
2024-03-24 00:28:17,898 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Update db
status for job-981
2024-03-24 00:28:17,900 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Wake up
jobs joined with job-981 and disjoin all subjobs created from job- 981
2024-03-24 00:28:17,910 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Done
executing com.cloud.vm.VmWorkMigrateWithStorage for job-981
2024-03-24 00:28:17,912 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-73:ctx-6a8adbf6 job-980/job-981) (logid:61b73463) Remove
job-981 from job monitoring
2024-03-24 00:28:17,928 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM
instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException].
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1447)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy227.migrateVolumes(Unknown Source)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:3267)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:5505)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5581)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-03-24 00:28:17,930 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Complete async
job-980, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to migrate VM [VM instance
{"id":109,"instanceName":"i-8-108-VM","type":"User","uuid":"2fbb0d84-7ea2-4092-aa5d-25a53eb3cda2"}]
along with its volumes due to [java.lang.NullPointerException]."}
2024-03-24 00:28:17,931 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Publish async
job-980 complete on message bus
2024-03-24 00:28:17,931 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Wake up jobs
related to job-980
2024-03-24 00:28:17,931 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Update db status
for job-980
2024-03-24 00:28:17,933 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Wake up jobs joined
with job-980 and disjoin all subjobs created from job- 980
2024-03-24 00:28:17,936 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
for job-980
2024-03-24 00:28:17,936 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-95:ctx-5b617a54 job-980) (logid:61b73463) Remove job-980 from
job monitoring
2024-03-24 00:28:18,082 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null)
(logid:) Seq 7-5826813492887328392: Processing: { Ans: , MgmtId:
104023939828503, via: 7, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-03-24 00:28:18,083 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-3:null) (logid:) Seq 7-5826813492887328392: Unable to
find listener.
2024-03-24 00:28:18,755 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) SeqA 17-256622: Processing Seq
17-256622: { Cmd , MgmtId: -1, via: 17, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"70","_loadInfo":"{
"connections": [],
"removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2024-03-24 00:28:18,759 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) SeqA 17-256622: Sending Seq 17-256622:
{ Ans: , MgmtId: 104023939828503, via: 17, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-03-24 00:28:19,095 DEBUG [c.c.a.ApiServlet]
(qtp1753127384-21473:ctx-fa7e9ca2) (logid:3347dc0e) ===START=== -- GET
jobId=61b73463-d774-4712-ba9c-36f4813fe19e&command=queryAsyncJobResult&response=json
2024-03-24 00:28:19,095 DEBUG [c.c.a.ApiServlet]
(qtp1753127384-21473:ctx-fa7e9ca2) (logid:3347dc0e) Two factor authentication
is already verified for the user 2, so skipping
2024-03-24 00:28:19,106 DEBUG [c.c.a.ApiServer]
(qtp1753127384-21473:ctx-fa7e9ca2 ctx-f488e135) (logid:3347dc0e) CIDRs from
which account 'Account
[{"accountName":"admin","id":2,"uuid":"035326c3-7278-11ee-8d27-5e9bf5fb4b17"}]'
is allowed to perform API calls: 0.0.0.0/0,::/0
2024-03-24 00:28:19,110 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker]
(qtp1753127384-21473:ctx-fa7e9ca2 ctx-f488e135) (logid:3347dc0e) Account
[Account
[{"accountName":"admin","id":2,"uuid":"035326c3-7278-11ee-8d27-5e9bf5fb4b17"}]]
is Root Admin or Domain Admin, all APIs are allowed.
2024-03-24 00:28:19,112 WARN [o.a.c.a.ProjectRoleBasedApiAccessChecker]
(qtp1753127384-21473:ctx-fa7e9ca2 ctx-f488e135) (logid:3347dc0e) Project is
null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"035455d7-7278-11ee-8d27-5e9bf5fb4b17"}.] as allowed.
2024-03-24 00:28:19,115 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
(qtp1753127384-21473:ctx-fa7e9ca2 ctx-f488e135) (logid:3347dc0e) RoleService is
enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
