Hello,
Looks like migration fails in this scenario:
1.) New internal network, I will call it - NET1
2.) When perform migration to host without running instance with NET1,
live migration fails.
Also, when cloudstack create instance in other host (to which live
migration fails), after that I can live migrate successfully, without
problems.
And again, if I live migrate all instances to one host, I can't migrate
back if there in no running instance with NET1 network.
Sometimes again, VM's fail to launch, problem same, as described here:
http://comments.gmane.org/gmane.comp.apache.cloudstack.devel/41290
Migration error: http://pastebin.com/zhiqdy64
Grep job id from management log: http://pastebin.com/087Q3SRx
Shared networks in Advanced zone looks like not working as I expected,
because in network creation I should put VLAN ID.
On 2014.09.04. 20:56, Mārtiņš Jakubovičs wrote:
> Hello Geoff,
>
> Thanks for clarification. I have XenServer 6.2 pool, with all patches. I
> can send tomorrow both error logs when I use GRE and when I use VLAN
> guest network and live migration fails...
>
> I have 4 nic's, 2 nic's are bonded and are used for management and
> storage. 1 public nic (internet). 1 internal nic, for GRE tunnels or as
> secondary shared network for internal use.
>
> Can you, please, point me in a way how I can create Advanced networking
> with two shared networks? Should I use two Guest networks without Public
> network?
>
> Thanks.
>
> On 2014.09.04. 20:08, Geoff Higginbottom wrote:
>> Hi Martin,
>>
>> ACS 4.3 Advanced Zone Live Migration certainly does work, both within
>> the same Pool (using shared storage) , and across Pools / Shared
>> Storage (assume you are using XenServer as you mention Pool) Live
>> migration also works across any VMware Host or KVM Host within a Zone,
>> as long as you have configured Zone wide Primary Storage.
>>
>> Basic Zone only allows you to have the one Guest Network, but if your
>> end goal is to have each VM with a NIC on an Internal Network, and a
>> NIC on a Public Network this can easily be achieved with Advanced
>> Networking, using Shared Networks.
>>
>> Advanced Zones with Security Groups do work, what Hypervisor were you
>> using for this.
>>
>> Regards
>>
>> Geoff Higginbottom
>>
>> D: +44 20 3603 0542 | S: +44 20 3603 0540 | M: +447968161581
>>
>> geoff.higginbot...@shapeblue.com
>>
>> -----Original Message-----
>> From: Mārtiņš Jakubovičs [mailto:mart...@vertigs.lv]
>> Sent: 04 September 2014 15:32
>> To: users@cloudstack.apache.org
>> Subject: Live migration
>>
>> Hello all,
>>
>> In ACS 4.3 with advanced zone, live migration works? I tested guest
>> network with VLAN and GRE, but in both cases pool based live migration
>> fails.
>>
>> Is it possible to setup something like basic zone setup, but with two
>> networks? I want instances with two interfaces, one with public nic,
>> other with internal nic. I tryed to create advanced zone with security
>> groups, but it fails.
>>
>> Thanks.
>> Find out more about ShapeBlue and our range of CloudStack related
>> services
>>
>> IaaS Cloud Design &
>> Build<http://shapeblue.com/iaas-cloud-design-and-build//>
>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
>> CloudStack Infrastructure
>> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
>> CloudStack Bootcamp Training
>> Courses<http://shapeblue.com/cloudstack-training/>
>>
>> This email and any attachments to it may be confidential and are
>> intended solely for the use of the individual to whom it is addressed.
>> Any views or opinions expressed are solely those of the author and do
>> not necessarily represent those of Shape Blue Ltd or related
>> companies. If you are not the intended recipient of this email, you
>> must neither take any action based upon its contents, nor copy or show
>> it to anyone. Please contact the sender if you believe you have
>> received this email in error. Shape Blue Ltd is a company incorporated
>> in England & Wales. ShapeBlue Services India LLP is a company
>> incorporated in India and is operated under license from Shape Blue
>> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in
>> Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA
>> Pty Ltd is a company registered by The Republic of South Africa and is
>> traded under license from Shape Blue Ltd. ShapeBlue is a registered
>> trademark.
>
>
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-64:ctx-4b26da7f) Add job-363
into job monitoring
INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-64:ctx-4b26da7f
ctx-b5ed4a77) Migrating VM[User|Demo-Local-6-VM] to
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(7)-Pod(5)-Cluster(5)-Host(24)-Storage()]
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-e3c8d87c) Task failed!
Task record: uuid: 2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-e3c8d87c) Unable to
migrate VM(i-2-40-VM) from host(c299d66d-ee90-43ef-aee2-7dbcc0426f15) due to
Task failed! Task record: uuid:
2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
Task failed! Task record: uuid:
2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3764)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.migrateVM(CitrixResourceBase.java:3910)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:3468)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:509)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-e3c8d87c) Catch
Exception com.cloud.utils.exception.CloudRuntimeException: Migration failed due
to com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
VM(i-2-40-VM) from host(c299d66d-ee90-43ef-aee2-7dbcc0426f15) due to Task
failed! Task record: uuid: 2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
VM(i-2-40-VM) from host(c299d66d-ee90-43ef-aee2-7dbcc0426f15) due to Task
failed! Task record: uuid: 2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.migrateVM(CitrixResourceBase.java:3921)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:3468)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:509)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-64:ctx-4b26da7f
ctx-b5ed4a77) Migration was unsuccessful. Cleaning up: VM[User|Demo-Local-6-VM]
WARN [o.a.c.alerts] (Job-Executor-64:ctx-4b26da7f ctx-b5ed4a77) alertType::
17 // dataCenterId:: 7 // podId:: 5 // clusterId:: null // message:: Unable to
migrate vm i-2-40-VM from host cloudstack-2 in zone Cloud-1 and pod Pod-1
ERROR [c.c.a.ApiAsyncJobDispatcher] (Job-Executor-64:ctx-4b26da7f) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due to Catch
Exception com.cloud.utils.exception.CloudRuntimeException: Migration failed due
to com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
VM(i-2-40-VM) from host(c299d66d-ee90-43ef-aee2-7dbcc0426f15) due to Task
failed! Task record: uuid: 2894db93-aada-c709-a3b7-9a7538c491ad
nameLabel: Async.VM.pool_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Sep 05 14:57:38 EEST 2014
finished: Fri Sep 05 14:57:38 EEST 2014
status: failure
residentOn: com.xensource.xenapi.Host@2f447499
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:1840)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:1748)
at
com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:1712)
at
com.cloud.vm.UserVmManagerImpl.migrateVirtualMachine(UserVmManagerImpl.java:4219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy169.migrateVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd.execute(MigrateVMCmd.java:148)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:509)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-64:ctx-4b26da7f) Remove
job-363 from job monitoring2014-09-05 14:57:38,056 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-4:ctx-0910fd71 ctx-b5ed4a77) submit async job-363, details:
AsyncJobVO {id:363, userId: 3, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
{"response":"json","sessionkey":"QipU62upwDYl0/wmePTOWVTcias\u003d","virtualmachineid":"b70279da-254c-4389-b5dc-724b39e68aee","cmdEventType":"VM.MIGRATE","hostid":"f0dcbfb7-2cfe-4df5-9758-1fc61f13a961","ctxUserId":"3","httpmethod":"GET","_":"1409918258303","ctxAccountId":"2","ctxStartEventId":"1170"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 95136892860611, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-09-05 14:57:38,056 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-64:ctx-4b26da7f) Add job-363 into job monitoring
2014-09-05 14:57:39,562 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-64:ctx-4b26da7f) Complete async job-363, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to migrate due to Catch Exception
com.cloud.utils.exception.CloudRuntimeException: Migration failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
VM(i-2-40-VM) from host(c299d66d-ee90-43ef-aee2-7dbcc0426f15) due to Task
failed! Task record: uuid:
2894db93-aada-c709-a3b7-9a7538c491ad\n nameLabel:
Async.VM.pool_migrate\n nameDescription: \n allowedOperations: []\n
currentOperations: {}\n created: Fri Sep 05 14:57:38 EEST 2014\n
finished: Fri Sep 05 14:57:38 EEST 2014\n status:
failure\n residentOn: com.xensource.xenapi.Host@2f447499\n
progress: 1.0\n type: \u003cnone/\u003e\n result:
\n errorInfo: [VM_REQUIRES_NETWORK,
OpaqueRef:2b74dcc5-e91d-10a1-8d5a-11cc7a337ec8,
OpaqueRef:8b5b77ac-b16c-f6d2-e79c-d10b1ba8b713]\n otherConfig: {}\n
subtaskOf: com.xensource.xenapi.Task@aaf13f6f\n subtasks:
[]\n"}
2014-09-05 14:57:39,568 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-64:ctx-4b26da7f) Done executing
org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-363
2014-09-05 14:57:39,572 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-64:ctx-4b26da7f) Remove job-363 from job monitoring
