FYI testing this patch as it seems related to the issue I observe: https://github.com/apache/cloudstack/pull/925
Regards, Remi Sent from my iPhone > On 26 Jun 2016, at 17:41, Remi Bergsma <rberg...@schubergphilis.com> wrote: > > This is about xenserver 6.5. > > Regards, Remi > > Sent from my iPhone > >> On 26 Jun 2016, at 14:14, Will Stevens <williamstev...@gmail.com> wrote: >> >> Which hypervisor are you using? >>> On Jun 26, 2016 2:00 AM, "Remi Bergsma" <rberg...@schubergphilis.com> wrote: >>> >>> Hi Makrand, >>> >>> Thanks for the info. What I want is migrate a VM from one cluster (with >>> cluster scope shared storage) to another one. The UI / API allows that (it >>> says: storage migration required) but it doesn't work. The goal is to >>> consolidate two half empty clusters into one and then reprovison the empty >>> one. >>> >>> I was wondering whether others know this feature even works in which case >>> my issue might be a local config issue or so. >>> >>> I'll try to reproduce in a test setup so that I can hook in the debugger >>> and see what's going on. >>> >>> If there are tips please let me know :-) >>> >>> Thanks, Remi >>> >>> Sent from my iPhone >>> >>>> On 25 Jun 2016, at 17:41, Makrand <makrandsa...@gmail.com> wrote: >>>> >>>> Hi Remi, >>>> >>>> If you're talking about common storage for all clusters, then its Zone >>> wide >>>> primary storage. I think it works pretty well, only if the new created VM >>>> have disks already created on same new storage. >>>> >>>> With my previous employer, we rolled out zonewide primary (with cluster >>>> level primary setup already in place) and it worked pretty good once we >>>> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated >>> DB >>>> (storage & volumes tables). (We eventually ditched Cluster primary). >>>> >>>> -- >>>> Best, >>>> Makrand >>>> >>>> >>>> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma < >>> rberg...@schubergphilis.com> >>>> wrote: >>>> >>>>> Hi Anshul, >>>>> >>>>> Thanks! Did you ever try live migrating from CloudStack with storage >>>>> between clusters? I can't get that to work. I'll collect some details in >>>>> the next days but was wondering if it was supposed to work. >>>>> >>>>> Regards, Remi >>>>> >>>>> Sent from my iPhone >>>>> >>>>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar < >>> anshul.gang...@accelerite.com> >>>>>> wrote: >>>>>> >>>>>> Created the PR https://github.com/apache/cloudstack/pull/1596. >>>>>> >>>>>> Regards, >>>>>> Anshul >>>>>> >>>>>> On 24/06/16, 2:33 AM, "Dave Garbus" <davegar...@spiceworks.com> wrote: >>>>>> >>>>>> Is there anybody that wants to take this on? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> - Dave >>>>>> >>>>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <davegar...@spiceworks.com> >>>>> wrote: >>>>>>> >>>>>>> I updated the issue referenced in this email ( >>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include >>>>> CloudStack 4.8 as an affected version. If we could at least get someone >>>>> assigned to it, that’d be a good start. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> - Dave >>>>>>> >>>>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev < >>>>> ivan.derbe...@tech-corps.com<mailto:ivan.derbe...@tech-corps.com>> >>> wrote: >>>>>>> >>>>>>> Is there a way I can fix this on my installation, without recompiling >>>>> jar files? >>>>>>> >>>>>>> Regards, >>>>>>> IT engineer >>>>>>> Farheap, Russia >>>>>>> Ivan Derbenev >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Koushik Das [mailto:koushik....@accelerite.com] >>>>>>> Sent: Tuesday, May 24, 2016 2:06 PM >>>>>>> To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org> >>>>>>> Subject: Re: Xenserver 6.5 live migration with local volumes >>>>>>> >>>>>>> This is a product bug and needs to be fixed. There is some >>>>> serialization issue that needs fixing. >>>>>>> >>>>>>> In the below code volumeToFiler is null as Map is not a serializable >>>>> type. It needs to be replaced with getVolumeToFilerAsList() and then the >>>>> data needs to be read appropriately. >>>>>>> >>>>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler = >>>>> command.getVolumeToFiler(); >>>>>>> >>>>>>> -Koushik >>>>>>> >>>>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbe...@tech-corps.com >>>>> <mailto:ivan.derbe...@tech-corps.com>> wrote: >>>>>>> >>>>>>> Hello! >>>>>>> After the migration onto XS 6.5 (and installing all the updates) and >>> CS >>>>>>> 4.8 I can't migrate VMs with Local Storage >>>>>>> >>>>>>> The issue is similar to this: >>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't >>>>> have any solutions Any ideas? >>>>>>> >>>>>>> Errors: >>>>>>> >>>>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] >>>>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with >>> process >>>>>>> of VM state report. host: 136592 >>>>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network >>>>>>> named cloud-main >>>>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called >>>>>>> cloud-main on host=10.7.130.103; >>>>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0; >>>>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f >>>>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on >>>>>>> host 10.7.130.103 on device eth0 >>>>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for >>>>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b >>>>>>> 2016-05-24 12:10:17,170 WARN >>>>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] >>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception >>>>> java.lang.NullPointerException. Storage motion failed due to >>>>> java.lang.NullPointerException java.lang.NullPointerException >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) >>>>>>> at >>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq >>>>> 136589-479633360314957887: Response Received: >>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq >>>>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId: >>>>>>> 128437433941265, via: >>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com< >>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: >>>>>>> 110, >>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai >>>>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack: >>>>>>> java.lang.NullPointerException\n\tat >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr >>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm >>>>>>> andWrapper.java:86)\n\tat >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr >>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm >>>>>>> andWrapper.java:54)\n\tat >>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr >>>>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat >>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque >>>>>>> st(CitrixResourceBase.java:1677)\n\tat >>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen >>>>>>> tAttache.java:315)\n\tat >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana >>>>>>> gedContextRunnable.java:49)\n\tat >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call >>>>>>> (DefaultManagedContext.java:56)\n\tat >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi >>>>>>> thContext(DefaultManagedContext.java:103)\n\tat >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit >>>>>>> hContext(DefaultManagedContext.java:53)\n\tat >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage >>>>>>> dContextRunnable.java:46)\n\tat >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac >>>>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru< >>>>> http://ScheduledFutureTask.ru> >>>>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja >>>>>>> va:1145)\n\tat >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j >>>>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] } >>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache] >>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq >>>>>>> 136589-479633360314957887: No more commands found >>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: , >>>>>>> MgmtId: 128437433941265, via: >>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com< >>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: >>>>>>> 110, { MigrateWithStorageAnswer } } >>>>>>> 2016-05-24 12:10:17,171 ERROR >>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM] >>>>>>> failed. Details: Exception: java.lang.NullPointerException >>>>>>> Message: null >>>>>>> Stack: java.lang.NullPointerException >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) >>>>>>> at >>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> >>>>>>> 2016-05-24 12:10:17,171 ERROR >>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) copy failed >>>>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating >>>>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception: >>>>>>> java.lang.NullPointerException >>>>>>> Message: null >>>>>>> Stack: java.lang.NullPointerException >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) >>>>>>> at >>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> >>>>>>> at >>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212) >>>>>>> at >>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116) >>>>>>> at >>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90) >>>>>>> at >>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160) >>>>>>> at >>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at >>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) >>>>>>> at >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Unsupported data object (VOLUME, >>>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2 >>> ), >>>>>>> no need to delete from object in store ref table >>>>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with >>>>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error >>>>>>> while migrating the vm VM[User|i-3-271-VM] to host >>>>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException >>>>>>> Message: null >>>>>>> Stack: java.lang.NullPointerException >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) >>>>>>> at >>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) >>>>>>> at >>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) >>>>>>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> >>>>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up: >>>>>>> VM[User|i-3-271-VM] >>>>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // >>>>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from >>> host >>>>>>> cloudstack-computing01-sar01.tech-corps.com< >>>>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar >>> and >>>>> pod >>>>>>> zone1.sar >>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' -- >>>>>>> skipping send email >>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from >>>>>>> 128437433941265 >>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd , >>>>>>> MgmtId: 128437433941265, via: >>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com< >>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: >>>>>>> 100011, >>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence >>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } >>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd , >>>>>>> MgmtId: 128437433941265, via: >>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com< >>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: >>>>>>> 100011, >>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence >>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] } >>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache] >>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq >>>>>>> 136589-479633360314957888: Executing request >>>>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with >>>>>>> event: OperationFailedvm's original host id: 136592 new host id: >>> 136592 >>>>>>> host id before state transition: 136589 >>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after >>> applying >>>>>>> overprovisioning: 39900 >>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after >>>>>>> applying overprovisioning: 11549697024 >>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) release cpu from host: 136589, old used: >>>>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning: >>>>>>> 39900; new used: 1995,reserved:0; movedfromreserved: >>>>>>> false,moveToReserveredfalse >>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>>>> (logid:8748bffe) release mem from host: 136589, old used: >>>>>>> 5368709120,reserved: 0, total: 11549697024; new used: >>>>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse >>>>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] >>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq >>>>> 136589-479633360314957888: Response Received: >>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request] >>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq >>>>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId: >>>>>>> 128437433941265, via: >>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com< >>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags: >>>>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM >>>>>>> does not exist","wait":0}}] } >>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] >>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq >>>>> 136589-479633360314957888: Unable to find listener. >>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] >>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>> (logid:8748bffe) Invocation exception, caused by: >>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm >>>>> VM[User|i-3-271-VM] along with its volumes. >>>>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] >>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) >>>>> (logid:8748bffe) Rethrow exception >>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm >>>>> VM[User|i-3-271-VM] along with its volumes. >>>>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage >>> for >>>>>>> VM 271, job origin: 4874 >>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3, >>>>>>> instanceType: null, instanceId: null, cmd: >>>>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo: >>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM >>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX >>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ >>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw >>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA >>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW >>>>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0, >>>>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, >>>>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null, >>>>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job >>>>>>> origin:4874 >>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm >>>>> VM[User|i-3-271-VM] along with its volumes. >>>>>>> at >>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at >>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) >>>>>>> at >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result: >>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl >>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH >>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa >>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro >>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t >>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG >>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z >>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe >>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF >>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG >>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c >>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s >>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9 >>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW >>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc >>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz >>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N >>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX >>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd >>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY >>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt >>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG >>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf >>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X >>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU >>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX >>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A >>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0 >>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA >>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX >>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb >>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p >>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA >>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2 >>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA >>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0 >>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp >>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA >>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb >>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X >>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN >>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1 >>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ >>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk >>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29 >>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH >>>>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA >>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Publish async job-4875 complete on message bus >>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Wake up jobs related to job-4875 >>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Update db status for job-4875 >>>>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from >>>>>>> job- 4875 >>>>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875 >>>>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) >>>>>>> Remove job-4875 from job monitoring >>>>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected >>>>>>> exception while executing >>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol >>>>>>> umeCmd >>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm >>>>> VM[User|i-3-271-VM] along with its volumes. >>>>>>> at >>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at >>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) >>>>>>> at >>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741) >>>>>>> at >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) >>>>>>> at >>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) >>>>>>> at >>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) >>>>>>> at >>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) >>>>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete >>>>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result: >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[ >>>>>>> ],"errorcode":530,"errortext":"Failed to migrated vm >>>>>>> VM[User|i-3-271-VM] along with its volumes. "} >>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish >>>>>>> async job-4874 complete on message bus >>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up >>>>>>> jobs related to job-4874 >>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db >>>>>>> status for job-4874 >>>>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up >>>>>>> jobs joined with job-4874 and disjoin all subjobs created from job- >>>>>>> 4874 >>>>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done >>>>>>> executing >>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol >>>>>>> umeCmd for job-4874 >>>>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove >>>>>>> job-4874 from job monitoring ^C >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> IT engineer >>>>>>> Farheap, Russia >>>>>>> Ivan Derbenev >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> DISCLAIMER >>>>>>> ========== >>>>>>> This e-mail may contain privileged and confidential information which >>>>> is the property of Accelerite, a Persistent Systems business. It is >>>>> intended only for the use of the individual or entity to which it is >>>>> addressed. If you are not the intended recipient, you are not >>> authorized to >>>>> read, retain, copy, print, distribute or use this message. If you have >>>>> received this communication in error, please notify the sender and >>> delete >>>>> all copies of this message. Accelerite, a Persistent Systems business >>> does >>>>> not accept any liability for virus infected mails. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> DISCLAIMER >>>>>> ========== >>>>>> This e-mail may contain privileged and confidential information which >>> is >>>>> the property of Accelerite, a Persistent Systems business. It is >>> intended >>>>> only for the use of the individual or entity to which it is addressed. >>> If >>>>> you are not the intended recipient, you are not authorized to read, >>> retain, >>>>> copy, print, distribute or use this message. If you have received this >>>>> communication in error, please notify the sender and delete all copies >>> of >>>>> this message. Accelerite, a Persistent Systems business does not accept >>> any >>>>> liability for virus infected mails. >
