Is there anybody that wants to take this on? Thanks,
- Dave > On May 24, 2016, at 10:57 AM, Dave Garbus <davegar...@spiceworks.com> wrote: > > I updated the issue referenced in this email > (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack > 4.8 as an affected version. If we could at least get someone assigned to it, > that’d be a good start. > > Thanks, > > - Dave > > On May 24, 2016, at 9:42 AM, Ivan Derbenev > <ivan.derbe...@tech-corps.com<mailto:ivan.derbe...@tech-corps.com>> wrote: > > Is there a way I can fix this on my installation, without recompiling jar > files? > > Regards, > IT engineer > Farheap, Russia > Ivan Derbenev > > -----Original Message----- > From: Koushik Das [mailto:koushik....@accelerite.com] > Sent: Tuesday, May 24, 2016 2:06 PM > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> > Subject: Re: Xenserver 6.5 live migration with local volumes > > This is a product bug and needs to be fixed. There is some serialization > issue that needs fixing. > > In the below code volumeToFiler is null as Map is not a serializable type. It > needs to be replaced with getVolumeToFilerAsList() and then the data needs to > be read appropriately. > > final Map<VolumeTO, StorageFilerTO> volumeToFiler = > command.getVolumeToFiler(); > > -Koushik > > On 24/05/16, 3:23 PM, "Ivan Derbenev" > <ivan.derbe...@tech-corps.com<mailto:ivan.derbe...@tech-corps.com>> wrote: > > Hello! > After the migration onto XS 6.5 (and installing all the updates) and CS > 4.8 I can't migrate VMs with Local Storage > > The issue is similar to this: > https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have > any solutions Any ideas? > > Errors: > > 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] > (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process > of VM state report. host: 136592 > 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network > named cloud-main > 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called > cloud-main on host=10.7.130.103; > Network=9632020b-4d98-1bed-9ae8-477e0993cab0; > pif=3c7d1121-2bde-26cf-9478-18e51ffca75f > 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on > host 10.7.130.103 on device eth0 > 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for > 1007. The uuid is ec053fee-0426-9665-150a-77399562196b > 2016-05-24 12:10:17,170 WARN > [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception > java.lang.NullPointerException. Storage motion failed due to > java.lang.NullPointerException java.lang.NullPointerException > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq > 136589-479633360314957887: Response Received: > 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq > 136589-479633360314957887: Processing: { Ans: , MgmtId: > 128437433941265, via: > 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), > Ver: v1, Flags: > 110, > [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai > ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack: > java.lang.NullPointerException\n\tat > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr > ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm > andWrapper.java:86)\n\tat > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr > ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm > andWrapper.java:54)\n\tat > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr > apper.execute(CitrixRequestWrapper.java:122)\n\tat > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque > st(CitrixResourceBase.java:1677)\n\tat > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen > tAttache.java:315)\n\tat > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana > gedContextRunnable.java:49)\n\tat > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call > (DefaultManagedContext.java:56)\n\tat > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi > thContext(DefaultManagedContext.java:103)\n\tat > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit > hContext(DefaultManagedContext.java:53)\n\tat > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage > dContextRunnable.java:46)\n\tat > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac > cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru> > n(ScheduledThreadPoolExecutor.java:292)\n\tat > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja > va:1145)\n\tat > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] } > 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache] > (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq > 136589-479633360314957887: No more commands found > 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: , > MgmtId: 128437433941265, via: > 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), > Ver: v1, Flags: > 110, { MigrateWithStorageAnswer } } > 2016-05-24 12:10:17,171 ERROR > [o.a.c.s.m.XenServerStorageMotionStrategy] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM] > failed. Details: Exception: java.lang.NullPointerException > Message: null > Stack: java.lang.NullPointerException > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > 2016-05-24 12:10:17,171 ERROR > [o.a.c.s.m.XenServerStorageMotionStrategy] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) copy failed > com.cloud.utils.exception.CloudRuntimeException: Error while migrating > the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception: > java.lang.NullPointerException > Message: null > Stack: java.lang.NullPointerException > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > at > org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212) > at > org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116) > at > org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160) > at > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Unsupported data object (VOLUME, > org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2), > no need to delete from object in store ref table > 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with > its volumes. com.cloud.utils.exception.CloudRuntimeException: Error > while migrating the vm VM[User|i-3-271-VM] to host > Host[-136589-Routing]. Exception: java.lang.NullPointerException > Message: null > Stack: java.lang.NullPointerException > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) > at > com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) > at > com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Migration was unsuccessful. Cleaning up: > VM[User|i-3-271-VM] > 2016-05-24 12:10:17,184 WARN [o.a.c.alerts] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // > clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host > cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> > in zone zone1.sar and pod > zone1.sar > 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Have already sent: 1 emails for alert type '17' -- > skipping send email > 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Seq 136589-479633360314957888: Routed from > 128437433941265 > 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd , > MgmtId: 128437433941265, via: > 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), > Ver: v1, Flags: > 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence > ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } > 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd , > MgmtId: 128437433941265, via: > 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), > Ver: v1, Flags: > 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence > ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } > 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq > 136589-479633360314957888: Executing request > 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) VM state transitted from :Migrating to Running with > event: OperationFailedvm's original host id: 136592 new host id: 136592 > host id before state transition: 136589 > 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying > overprovisioning: 39900 > 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after > applying overprovisioning: 11549697024 > 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) release cpu from host: 136589, old used: > 3990,reserved: 0, actual total: 15960, total with overprovisioning: > 39900; new used: 1995,reserved:0; movedfromreserved: > false,moveToReserveredfalse > 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) release mem from host: 136589, old used: > 5368709120,reserved: 0, total: 11549697024; new used: > 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: > Response Received: > 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request] > (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq > 136589-479633360314957888: Processing: { Ans: , MgmtId: > 128437433941265, via: > 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), > Ver: v1, Flags: > 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM > does not exist","wait":0}}] } > 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] > (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: > Unable to find listener. > 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Invocation exception, caused by: > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-3-271-VM] along with its volumes. > 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) > (logid:8748bffe) Rethrow exception > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-3-271-VM] along with its volumes. > 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for > VM 271, job origin: 4874 > 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3, > instanceType: null, instanceId: null, cmd: > com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo: > rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM > CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX > RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ > EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw > AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA > AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW > RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0, > status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, > initMsid: 128437433941265, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job > origin:4874 > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-3-271-VM] along with its volumes. > at > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Complete async job-4875, jobStatus: FAILED, resultCode: 0, result: > rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl > vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH > Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa > mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro > cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t > UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG > NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z > WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe > W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF > 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG > VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c > AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s > dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9 > sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW > 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc > mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz > dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N > lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX > NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd > AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY > amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt > 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG > xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf > gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X > b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU > uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX > QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A > D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0 > UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA > AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX > VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb > GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p > bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA > MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2 > hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA > H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0 > aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp > hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA > APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb > nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X > b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN > 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1 > RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ > Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk > VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29 > sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH > AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA > 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Publish async job-4875 complete on message bus > 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Wake up jobs related to job-4875 > 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Update db status for job-4875 > 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Wake up jobs joined with job-4875 and disjoin all subjobs created from > job- 4875 > 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875 > 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) > Remove job-4875 from job monitoring > 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected > exception while executing > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol > umeCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|i-3-271-VM] along with its volumes. > at > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete > async job-4874, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[ > ],"errorcode":530,"errortext":"Failed to migrated vm > VM[User|i-3-271-VM] along with its volumes. "} > 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish > async job-4874 complete on message bus > 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up > jobs related to job-4874 > 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db > status for job-4874 > 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up > jobs joined with job-4874 and disjoin all subjobs created from job- > 4874 > 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done > executing > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol > umeCmd for job-4874 > 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor] > (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove > job-4874 from job monitoring ^C > > > Regards, > IT engineer > Farheap, Russia > Ivan Derbenev > > > > > > DISCLAIMER > ========== > This e-mail may contain privileged and confidential information which is the > property of Accelerite, a Persistent Systems business. It is intended only > for the use of the individual or entity to which it is addressed. If you are > not the intended recipient, you are not authorized to read, retain, copy, > print, distribute or use this message. If you have received this > communication in error, please notify the sender and delete all copies of > this message. Accelerite, a Persistent Systems business does not accept any > liability for virus infected mails. >