I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
Thanks, - Dave On May 24, 2016, at 9:42 AM, Ivan Derbenev <ivan.derbe...@tech-corps.com<mailto:ivan.derbe...@tech-corps.com>> wrote: Is there a way I can fix this on my installation, without recompiling jar files? Regards, IT engineer Farheap, Russia Ivan Derbenev -----Original Message----- From: Koushik Das [mailto:koushik....@accelerite.com] Sent: Tuesday, May 24, 2016 2:06 PM To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> Subject: Re: Xenserver 6.5 live migration with local volumes This is a product bug and needs to be fixed. There is some serialization issue that needs fixing. In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately. final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler(); -Koushik On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbe...@tech-corps.com<mailto:ivan.derbe...@tech-corps.com>> wrote: Hello! After the migration onto XS 6.5 (and installing all the updates) and CS 4.8 I can't migrate VMs with Local Storage The issue is similar to this: https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas? Errors: 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process of VM state report. host: 136592 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network named cloud-main 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called cloud-main on host=10.7.130.103; Network=9632020b-4d98-1bed-9ae8-477e0993cab0; pif=3c7d1121-2bde-26cf-9478-18e51ffca75f 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on host 10.7.130.103 on device eth0 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for 1007. The uuid is ec053fee-0426-9665-150a-77399562196b 2016-05-24 12:10:17,170 WARN [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received: 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Processing: { Ans: , MgmtId: 128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack: java.lang.NullPointerException\n\tat com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm andWrapper.java:86)\n\tat com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm andWrapper.java:54)\n\tat com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr apper.execute(CitrixRequestWrapper.java:122)\n\tat com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque st(CitrixResourceBase.java:1677)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen tAttache.java:315)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana gedContextRunnable.java:49)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call (DefaultManagedContext.java:56)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi thContext(DefaultManagedContext.java:103)\n\tat org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit hContext(DefaultManagedContext.java:53)\n\tat org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage dContextRunnable.java:46)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru> n(ScheduledThreadPoolExecutor.java:292)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja va:1145)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] } 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: No more commands found 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: , MgmtId: 128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: 110, { MigrateWithStorageAnswer } } 2016-05-24 12:10:17,171 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM] failed. Details: Exception: java.lang.NullPointerException Message: null Stack: java.lang.NullPointerException at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,171 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) copy failed com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception: java.lang.NullPointerException Message: null Stack: java.lang.NullPointerException at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212) at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160) at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2), no need to delete from object in store ref table 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception: java.lang.NullPointerException Message: null Stack: java.lang.NullPointerException at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Migration was unsuccessful. Cleaning up: VM[User|i-3-271-VM] 2016-05-24 12:10:17,184 WARN [o.a.c.alerts] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod zone1.sar 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Have already sent: 1 emails for alert type '17' -- skipping send email 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Seq 136589-479633360314957888: Routed from 128437433941265 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd , MgmtId: 128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd , MgmtId: 128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq 136589-479633360314957888: Executing request 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) VM state transitted from :Migrating to Running with event: OperationFailedvm's original host id: 136592 new host id: 136592 host id before state transition: 136589 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying overprovisioning: 39900 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after applying overprovisioning: 11549697024 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) release cpu from host: 136589, old used: 3990,reserved: 0, actual total: 15960, total with overprovisioning: 39900; new used: 1995,reserved:0; movedfromreserved: false,moveToReserveredfalse 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) release mem from host: 136589, old used: 5368709120,reserved: 0, total: 11549697024; new used: 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received: 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Processing: { Ans: , MgmtId: 128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] } 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener. 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for VM 271, job origin: 4874 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 128437433941265, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job origin:4874 com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Complete async job-4875, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9 sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0 UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2 hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0 aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1 RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29 sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Publish async job-4875 complete on message bus 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Wake up jobs related to job-4875 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Update db status for job-4875 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Wake up jobs joined with job-4875 and disjoin all subjobs created from job- 4875 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Remove job-4875 from job monitoring 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol umeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete async job-4874, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[ ],"errorcode":530,"errortext":"Failed to migrated vm VM[User|i-3-271-VM] along with its volumes. "} 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish async job-4874 complete on message bus 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up jobs related to job-4874 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db status for job-4874 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up jobs joined with job-4874 and disjoin all subjobs created from job- 4874 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol umeCmd for job-4874 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove job-4874 from job monitoring ^C Regards, IT engineer Farheap, Russia Ivan Derbenev DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
