[jira] [Commented] (CLOUDSTACK-9547) ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic guest VM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15882372#comment-15882372
]
Mani Prashanth Varma Manthena commented on CLOUDSTACK-9547:
---
Here are the possible fixes for this issue:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=2015340
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=2081503
> ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic guest VM
> --
>
> Key: CLOUDSTACK-9547
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9547
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server, VMware
>Affects Versions: 4.9.0
>Reporter: Mani Prashanth Varma Manthena
>Priority: Critical
> Labels: vmware
>
> Steps to reproduce:
> 1) Deploy a multi-nic guest VM (or) add a nic to a single-nic guest VM
> 2) Remove the non-default nic from the multi-nic guest VM, which fails with
> the following error/exception in the management server log:
> {noformat}
> 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98) ===END===
> 10.31.52.95 -- GET
> command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86=json&_=1475673245452
> 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
> UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
> java.lang.RuntimeException: The guest operating system did not respond to a
> hot-remove request for device ethernet1 in a timely manner.
> at
> com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:354)
> at
> com.cloud.hypervisor.vmware.mo.VirtualMachineMO.configureVm(VirtualMachineMO.java:949)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1103)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:469)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508:
> Response Received:
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508:
> MgmtId 275619427423488: Resp: Routing to peer
> 2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508: No
> more commands found
> 2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory]
> (API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f) (logid:9ad66ce9)
> Returning cached instance of singleton bean 'messageBus'
> 2016-10-05 06:13:31,127 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin
> com.cloud.utils.exception.CloudRuntimeException: Unable to remove
> Ntwk[205|Guest|16] from VM[User|i-2-3-VM]
> at
> com.cloud.vm.UserVmManagerImpl.removeNicFromVirtualMachine(UserVmManagerImpl.java:1291)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Commented] (CLOUDSTACK-9547) ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic guest VM
[
https://issues.apache.org/jira/browse/CLOUDSTACK-9547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584950#comment-15584950
]
Mani Prashanth Varma Manthena commented on CLOUDSTACK-9547:
---
I am only hitting this issue for guest VMs (i.e. not with VPC VRs) created in
ACS 4.9 (i.e. not in ACS 4.7) with VMware setups.
Moreover, I get the same error when I am trying to remove the NIC (i.e. network
adapter) directly from VMware's Vcenter.
There is a possible workaround for this issue from VMware on Internet, which
doesn't work in this scenario both from CloudStack and VMware:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=2081503
Most likely, this issue has something to do with how we deploy (multi-nic)
guest VMs in ACS 4.9 with VMware setups.
However, installing vmware tools (not easy in our built-in templates) made the
problem go away. With the tools installed the removal was pretty much instant.
Doesn't explain the difference 4.7 and 4.9 that we are seeing though on the
same VMware setup.
> ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic guest VM
> --
>
> Key: CLOUDSTACK-9547
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9547
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server, VMware
>Affects Versions: 4.9.0
>Reporter: Mani Prashanth Varma Manthena
>Priority: Critical
> Labels: vmware
>
> Steps to reproduce:
> 1) Deploy a multi-nic guest VM (or) add a nic to a single-nic guest VM
> 2) Remove the non-default nic from the multi-nic guest VM, which fails with
> the following error/exception in the management server log:
> {noformat}
> 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98) ===END===
> 10.31.52.95 -- GET
> command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86=json&_=1475673245452
> 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
> UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
> java.lang.RuntimeException: The guest operating system did not respond to a
> hot-remove request for device ethernet1 in a timely manner.
> at
> com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:354)
> at
> com.cloud.hypervisor.vmware.mo.VirtualMachineMO.configureVm(VirtualMachineMO.java:949)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1103)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:469)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508:
> Response Received:
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508:
> MgmtId 275619427423488: Resp: Routing to peer
> 2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508: No
> more commands found
> 2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory]
> (API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f)
