Re: ACS master: Failing to dedicate a vlan range to an account

[Prashanth Manthena]

Hi all,

The above issue is due to the created physical network not having a
VLAN/VNI range configured.

Not a bug.


On Mon, Mar 6, 2017 at 3:31 PM, Prashanth Manthena <
prashanth.manth...@nuagenetworks.net> wrote:

> Hi all,
>
> I am seeing the following issue while trying to dedicate a vlan range to
> an account:
>
> (local)  > dedicate guestvlanrange account=admin
> domainid=efd478ff-0145-11e7-ba73-faaca6101800 vlanrange=100-300
> physicalnetworkid=7c658a8b-6e24-4a2d-ad58-75f318815dfa
> Error 530: Index: 0, Size: 0
> cserrorcode = 
> errorcode = 530
> errortext = Index: 0, Size: 0
> uuidList:
>
>
> Here is the exception in the logs:
>
> 2017-03-06 06:23:46,476 ERROR [c.c.a.ApiServer]
> (catalina-exec-8:ctx-4703c806 ctx-672b4276 ctx-ee68b9c9) (logid:2af07a55)
> unhandled exception executing api command: [Ljava.lang.String;@752957ea
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at com.cloud.network.NetworkServiceImpl.dedicateGuestVlanRange(
> NetworkServiceImpl.java:3222)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsin
> gReflection(AopUtils.java:333)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .invokeJoinpoint(ReflectiveMethodInvocation.java:190)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:157)
> at org.apache.cloudstack.network.contrail.management.EventUtils
> $EventInterceptor.invoke(EventUtils.java:107)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:168)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInt
> erceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:168)
> at org.springframework.aop.interceptor.ExposeInvocationIntercep
> tor.invoke(ExposeInvocationInterceptor.java:92)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:179)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(
> JdkDynamicAopProxy.java:213)
> at com.sun.proxy.$Proxy171.dedicateGuestVlanRange(Unknown Source)
> at org.apache.cloudstack.api.command.admin.network.DedicateGues
> tVlanRangeCmd.execute(DedicateGuestVlanRangeCmd.java:113)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
> at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.
> java:299)
> at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text.runWithContext(DefaultManagedContext.java:53)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
> lter(ApplicationFilterChain.java:303)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
> licationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilte
> r.java:52)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFi
> lter(ApplicationFilterChain.java:241)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(App
> licationFilterChain.java:208)
> at org.apache.catalina.core.StandardWrapperValve.invoke(Standar
> dWrapperValve.java:220)
> at org.apache.catalina.core.StandardContextValve.invoke(Standar
> dContextValve.java:122)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(A
> uthenticatorBase.java:501)
> 

ACS master: Failing to dedicate a vlan range to an account

[Prashanth Manthena]

Hi all,

I am seeing the following issue while trying to dedicate a vlan range to an
account:

(local)  > dedicate guestvlanrange account=admin
domainid=efd478ff-0145-11e7-ba73-faaca6101800 vlanrange=100-300
physicalnetworkid=7c658a8b-6e24-4a2d-ad58-75f318815dfa
Error 530: Index: 0, Size: 0
cserrorcode = 
errorcode = 530
errortext = Index: 0, Size: 0
uuidList:


Here is the exception in the logs:

2017-03-06 06:23:46,476 ERROR [c.c.a.ApiServer]
(catalina-exec-8:ctx-4703c806 ctx-672b4276 ctx-ee68b9c9) (logid:2af07a55)
unhandled exception executing api command: [Ljava.lang.String;@752957ea
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at com.cloud.network.NetworkServiceImpl.dedicateGuestVlanRange(
NetworkServiceImpl.java:3222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.springframework.aop.support.AopUtils.
invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.
invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.
proceed(ReflectiveMethodInvocation.java:157)
at org.apache.cloudstack.network.contrail.management.
EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.
proceed(ReflectiveMethodInvocation.java:168)
at com.cloud.event.ActionEventInterceptor.invoke(
ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.
proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.
invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.
proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.
invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy171.dedicateGuestVlanRange(Unknown Source)
at org.apache.cloudstack.api.command.admin.network.
DedicateGuestVlanRangeCmd.execute(DedicateGuestVlanRangeCmd.java:113)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:709)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:533)
at com.cloud.api.ApiServlet.processRequestInContext(
ApiServlet.java:299)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:129)
at org.apache.cloudstack.managed.context.impl.
DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:126)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(
WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(
AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(
AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.
process(AbstractProtocol.java:607)
at 

ACS 4.9.x + VMware/ESXi 6.0: Sometimes VRs are failing to fully come up into running state

[Prashanth Manthena]

Hi All,

Is anyone aware of the following issue in ACS 4.9.x + VMware/ESXi 6.0
environment:
https://issues.apache.org/jira/browse/CLOUDSTACK-9575

Note: cloudstack management server and ESXi hotd logs are attached in the
above Jira issue.

I am hitting this issue randomly, mostly with restart networks with cleanup
(destroying and re-deploying VR). Moreover, VR comes up after one or more
retries of the same operation (or) manually restarting the VR (or) powering
on the VR from the vcenter.

Any help in debugging this issue will be greatly appreciated.

With regards,
Prashanth

ACS 4.9 + VMware/ESXi 6.0: Sometimes VRs are failing to fully come up into running state (CLOUDSTACK-9575)

[Prashanth Manthena]

Hi All,

We are hitting this issue quite often (not always, but one out of three
times) in particular use cases (e.g. restart networks with cleanup, create
VPCs, etc.).

I have raised the following bug, which contains all the information
regarding this issue: CLOUDSTACK-9575


Let me know, if you are also hitting this issue with ACS 4.9 + VMware/ESXi
6.0 setups.

With regards,
Prashanth

Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM

[Prashanth Manthena]

Hi All,

Raised the following issue on Jira to track this bug:
https://issues.apache.org/jira/browse/CLOUDSTACK-9547

With regards,
Prashanth

On Mon, Oct 17, 2016 at 12:38 PM, Prashanth Manthena <
prashanth.manth...@nuagenetworks.net> wrote:

> Hi Paul,
>
> First of all, thank you for your time and investigation.
>
> I have tried with timeouts up to 180 seconds, and that didn't help either
> :)
>
> Yes, it doesn't explain the difference between 4.7 and 4.9, because I am
> using the same vcenter/vmware setup (i.e. no vmware tools installed).
>
> Let me know, if you want me to raise an issue on Jira for tracking this
> bug.
>
> On Sat, Oct 15, 2016 at 8:55 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
>> I tried extending the timeout to 30 seconds, but that didn't help.
>> However, installing vmware tools (not easy in our built-in templates)
>> made the problem go away.  With the tools installed the removal was pretty
>> much instant.
>> Doesn't explain the difference 4.7 and 4.9 that you are seeing though.
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.an...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net]
>> Sent: 14 October 2016 13:24
>> To: dev@cloudstack.apache.org
>> Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
>> multi-nic VM
>>
>> Hi Paul,
>>
>> Thank you for trying it out.
>>
>> I am only hitting this issue for guest VMs (i.e. not with VPC VRs)
>> created in ACS 4.9 (i.e. not in ACS 4.7) with VMware setups.
>>
>> Moreover, I get the same error when I am trying to remove the NIC (i.e.
>> network adapter) directly from VMware's Vcenter.
>>
>> There is a possible workaround for this issue from VMware on Internet,
>> which doesn't work in this scenario both from CloudStack and VMware:
>> https://kb.vmware.com/selfservice/microsites/search.do?
>> language=en_US=displayKC=2081503
>>
>> Most likely, this issue has something to do with how we deploy
>> (multi-nic) guest VMs in ACS 4.9 with VMware setups.
>>
>> On Fri, Oct 14, 2016 at 1:00 PM, Paul Angus <paul.an...@shapeblue.com>
>> wrote:
>>
>> > Hi Prashanth,
>> >
>> > I've just tried that. I get the same error - The guest operating
>> > system did not respond to a hot-remove request for device ethernet1 in
>> > a timely manner.
>> >
>> > Kind regards,
>> >
>> > Paul Angus
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net]
>> > Sent: 14 October 2016 09:21
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
>> > multi-nic VM
>> >
>> > HI All,
>> >
>> > Does this issue ring a bell, and is anyone else hitting this issue ?
>> >
>> > Let me know, if it is a known issue.
>> >
>> > Thanking you in advance !!
>> >
>> > On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena <
>> prashanth.manthena@
>> > nuagenetworks.net> wrote:
>> >
>> > > Hi,
>> > >
>> > > I am hitting the following issue on an ACS 4.9 + VMware setup (steps
>> > > to
>> > > reproduce):
>> > >
>> > > 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM
>> > >
>> > > 2) Remove the non-default nic from the multi-nic VM, which fails with
>> > > the following error/exception in the management server log:
>> > >
>> > > 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
>> > > (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98)
>> > > ===END===  10.31.52.95 -- GET
>> > > command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-76
>> 3f4586dd86
>> > > =json&_=1475673245452
>> > > 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
>> > (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
>> > UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
>> > > java.lang.RuntimeException: Th

Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM

[Prashanth Manthena]

Hi Paul,

First of all, thank you for your time and investigation.

I have tried with timeouts up to 180 seconds, and that didn't help either
:)

Yes, it doesn't explain the difference between 4.7 and 4.9, because I am
using the same vcenter/vmware setup (i.e. no vmware tools installed).

Let me know, if you want me to raise an issue on Jira for tracking this bug.

On Sat, Oct 15, 2016 at 8:55 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> I tried extending the timeout to 30 seconds, but that didn't help.
> However, installing vmware tools (not easy in our built-in templates) made
> the problem go away.  With the tools installed the removal was pretty much
> instant.
> Doesn't explain the difference 4.7 and 4.9 that you are seeing though.
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net]
> Sent: 14 October 2016 13:24
> To: dev@cloudstack.apache.org
> Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
> multi-nic VM
>
> Hi Paul,
>
> Thank you for trying it out.
>
> I am only hitting this issue for guest VMs (i.e. not with VPC VRs) created
> in ACS 4.9 (i.e. not in ACS 4.7) with VMware setups.
>
> Moreover, I get the same error when I am trying to remove the NIC (i.e.
> network adapter) directly from VMware's Vcenter.
>
> There is a possible workaround for this issue from VMware on Internet,
> which doesn't work in this scenario both from CloudStack and VMware:
> https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=
> displayKC=2081503
>
> Most likely, this issue has something to do with how we deploy (multi-nic)
> guest VMs in ACS 4.9 with VMware setups.
>
> On Fri, Oct 14, 2016 at 1:00 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > Hi Prashanth,
> >
> > I've just tried that. I get the same error - The guest operating
> > system did not respond to a hot-remove request for device ethernet1 in
> > a timely manner.
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net]
> > Sent: 14 October 2016 09:21
> > To: dev@cloudstack.apache.org
> > Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
> > multi-nic VM
> >
> > HI All,
> >
> > Does this issue ring a bell, and is anyone else hitting this issue ?
> >
> > Let me know, if it is a known issue.
> >
> > Thanking you in advance !!
> >
> > On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena < prashanth.manthena@
> > nuagenetworks.net> wrote:
> >
> > > Hi,
> > >
> > > I am hitting the following issue on an ACS 4.9 + VMware setup (steps
> > > to
> > > reproduce):
> > >
> > > 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM
> > >
> > > 2) Remove the non-default nic from the multi-nic VM, which fails with
> > > the following error/exception in the management server log:
> > >
> > > 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
> > > (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98)
> > > ===END===  10.31.52.95 -- GET
> > > command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86
> > > =json&_=1475673245452
> > > 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
> > (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
> > UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
> > > java.lang.RuntimeException: The guest operating system did not respond
> > to a hot-remove request for device ethernet1 in a timely manner.
> > > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(
> > VmwareClient.java:354)
> > > at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.
> > configureVm(VirtualMachineMO.java:949)
> > > at com.cloud.hypervisor.vmware.resource.VmwareResource.
> > execute(VmwareResource.java:1103)
> > > at com.cloud.hypervisor.vmware.resource.VmwareResource.
> > executeRequest(VmwareResource.java:469)
> > > at com.cloud.agent.manager.DirectAgentAttache$Task.
> runInContext(
> > DirectAgentAttache.java:315)

Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM

[Prashanth Manthena]

Hi Paul,

Thank you for trying it out.

I am only hitting this issue for guest VMs (i.e. not with VPC VRs) created
in ACS 4.9 (i.e. not in ACS 4.7) with VMware setups.

Moreover, I get the same error when I am trying to remove the NIC (i.e.
network adapter) directly from VMware's Vcenter.

There is a possible workaround for this issue from VMware on Internet,
which doesn't work in this scenario both from CloudStack and VMware:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US=displayKC=2081503

Most likely, this issue has something to do with how we deploy (multi-nic)
guest VMs in ACS 4.9 with VMware setups.

On Fri, Oct 14, 2016 at 1:00 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> Hi Prashanth,
>
> I've just tried that. I get the same error -
> The guest operating system did not respond to a hot-remove request for
> device ethernet1 in a timely manner.
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Prashanth Manthena [mailto:prashanth.manth...@nuagenetworks.net]
> Sent: 14 October 2016 09:21
> To: dev@cloudstack.apache.org
> Subject: Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a
> multi-nic VM
>
> HI All,
>
> Does this issue ring a bell, and is anyone else hitting this issue ?
>
> Let me know, if it is a known issue.
>
> Thanking you in advance !!
>
> On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena < prashanth.manthena@
> nuagenetworks.net> wrote:
>
> > Hi,
> >
> > I am hitting the following issue on an ACS 4.9 + VMware setup (steps
> > to
> > reproduce):
> >
> > 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM
> >
> > 2) Remove the non-default nic from the multi-nic VM, which fails with
> > the following error/exception in the management server log:
> >
> > 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98)
> > ===END===  10.31.52.95 -- GET
> > command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86
> > =json&_=1475673245452
> > 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
> UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
> > java.lang.RuntimeException: The guest operating system did not respond
> to a hot-remove request for device ethernet1 in a timely manner.
> > at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(
> VmwareClient.java:354)
> > at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.
> configureVm(VirtualMachineMO.java:949)
> > at com.cloud.hypervisor.vmware.resource.VmwareResource.
> execute(VmwareResource.java:1103)
> > at com.cloud.hypervisor.vmware.resource.VmwareResource.
> executeRequest(VmwareResource.java:469)
> > at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(
> DirectAgentAttache.java:315)
> > at org.apache.cloudstack.managed.context.
> ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > at org.apache.cloudstack.managed.context.
> ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508:
> Response Received:
> > 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
> > (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
> > 4-1440

Re: ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM

[Prashanth Manthena]

HI All,

Does this issue ring a bell, and is anyone else hitting this issue ?

Let me know, if it is a known issue.

Thanking you in advance !!

On Thu, Oct 13, 2016 at 6:25 PM, Prashanth Manthena <
prashanth.manth...@nuagenetworks.net> wrote:

> Hi,
>
> I am hitting the following issue on an ACS 4.9 + VMware setup (steps to
> reproduce):
>
> 1) Deploy a multi-nic VM (or) add a nic to a single-nic VM
>
> 2) Remove the non-default nic from the multi-nic VM, which fails with the
> following error/exception in the management server log:
>
> 2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet] 
> (catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98) ===END===  
> 10.31.52.95 -- GET  
> command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86=json&_=1475673245452
> 2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource] 
> (DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd: 
> UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
> java.lang.RuntimeException: The guest operating system did not respond to a 
> hot-remove request for device ethernet1 in a timely manner.
> at 
> com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:354)
> at 
> com.cloud.hypervisor.vmware.mo.VirtualMachineMO.configureVm(VirtualMachineMO.java:949)
> at 
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1103)
> at 
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:469)
> at 
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at 
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at 
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache] 
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508: 
> Response Received:
> 2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508: 
> MgmtId 275619427423488: Resp: Routing to peer
> 2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache] 
> (DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq 4-1440588930805137508: No 
> more commands found
> 2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory] 
> (API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f) (logid:9ad66ce9) 
> Returning cached instance of singleton bean 'messageBus'
> 2016-10-05 06:13:31,127 ERROR [c.c.a.ApiAsyncJobDispatcher] 
> (API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Unexpected 
> exception while executing 
> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin
> com.cloud.utils.exception.CloudRuntimeException: Unable to remove 
> Ntwk[205|Guest|16] from VM[User|i-2-3-VM]
> at 
> com.cloud.vm.UserVmManagerImpl.removeNicFromVirtualMachine(UserVmManagerImpl.java:1291)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at 
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvo

ACS 4.9 + VMware: Unable to remove one of the NICs of a multi-nic VM

[Prashanth Manthena]

Hi,

I am hitting the following issue on an ACS 4.9 + VMware setup (steps to
reproduce):

1) Deploy a multi-nic VM (or) add a nic to a single-nic VM

2) Remove the non-default nic from the multi-nic VM, which fails with the
following error/exception in the management server log:

2016-10-05 06:13:28,251 DEBUG [c.c.a.ApiServlet]
(catalina-exec-14:ctx-f8dc6bd0 ctx-ee610e01) (logid:58e9cf98)
===END===  10.31.52.95 -- GET
command=queryAsyncJobResult=9ad66ce9-6e1b-4c25-bd2e-763f4586dd86=json&_=1475673245452
2016-10-05 06:13:29,787 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-302:ctx-78a58d67 10.31.56.178, job-171/job-172, cmd:
UnPlugNicCommand) (logid:9ad66ce9) Unexpected exception:
java.lang.RuntimeException: The guest operating system did not respond
to a hot-remove request for device ethernet1 in a timely manner.
at 
com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:354)
at 
com.cloud.hypervisor.vmware.mo.VirtualMachineMO.configureVm(VirtualMachineMO.java:949)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1103)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:469)
at 
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-10-05 06:13:29,788 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
4-1440588930805137508: Response Received:
2016-10-05 06:13:29,788 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
(DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
4-1440588930805137508: MgmtId 275619427423488: Resp: Routing to peer
2016-10-05 06:13:29,789 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-302:ctx-78a58d67) (logid:9ad66ce9) Seq
4-1440588930805137508: No more commands found
2016-10-05 06:13:31,120 DEBUG [o.s.b.f.s.DefaultListableBeanFactory]
(API-Job-Executor-8:ctx-a6e36538 job-171 ctx-446c510f)
(logid:9ad66ce9) Returning cached instance of singleton bean
'messageBus'
2016-10-05 06:13:31,127 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-8:ctx-a6e36538 job-171) (logid:9ad66ce9) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Unable to remove
Ntwk[205|Guest|16] from VM[User|i-2-3-VM]
at 
com.cloud.vm.UserVmManagerImpl.removeNicFromVirtualMachine(UserVmManagerImpl.java:1291)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at 

Re: ACS 4.9 + VMware: Failed to deploy VPC VR due to IPAssoc failure

[Prashanth Manthena]

Hi Murali,

Thank you for the quick response.

We will try out your PR, and will keep you updated.

I have couple of questions for you:

Do you know of any other issues in ACS 4.9/master with VMware ?

Is ACS 4.9/master with VMware being supported and qualified by the
community ?

Thanking you in advance !!

With regards,
Prashanth

On Mon, Sep 26, 2016 at 2:06 PM, Murali Reddy <muralimmre...@gmail.com>
wrote:

>
> findRouterEthDeviceIndex() in VmwareResource returned -1.
>
>
> Looks like CLOUDSTACK-9491: incorrect parsing of device list to find
> ethernet index of plugged NIC
>
> There is a PR https://github.com/apache/cloudstack/pull/1681
>
>
>
> On 26/09/16, 5:13 PM, "Prashanth Manthena" <prashanth.manthena@
> nuagenetworks.net> wrote:
>
> >Hi All,
> >
> >We are hitting an issue where VPC VR is failing to come up due to IpAssoc
> >failure in ACS+VMware setup.
> >
> >We have tried with the following system VM templates:
> >
> >http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-
> vmware.ova
> >
> >http://cloudstack.apt-get.eu/templates/4.7/systemvm64template-2016-05-18-
> 4.7.0-vmware.ova
> >
> >
> >Is there a more latest system VM template that fixes this issue ?
> >
> >
> >Here is the error log:
> >
> >[1:00]
> >IpAssocVpcCommand) (logid:f727c105) Prepare Ip Assoc failure on applying
> >one ip due to exception:
> >com.cloud.exception.InternalErrorException: Failed to find DomR VIF to
> >associate/disassociate IP with.
> >   at
> >com.cloud.hypervisor.vmware.resource.VmwareResource.
> prepareNetworkElementCommand(VmwareResource.java:917)
> >   at
> >com.cloud.hypervisor.vmware.resource.VmwareResource.
> prepareCommand(VmwareResource.java:803)
> >   at
> >com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.
> executeRequest(VirtualRoutingResource.java:97)
> >   at
> >com.cloud.hypervisor.vmware.resource.VmwareResource.
> executeRequest(VmwareResource.java:387)
> >   at
> >com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(
> DirectAgentAttache.java:315)
> >   at
> >org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(
> ManagedContextRunnable.java:49)
> >   at
> >org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(
> DefaultManagedContext.java:56)
> >   at
> >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> callWithContext(DefaultManagedContext.java:103)
> >   at
> >org.apache.cloudstack.managed.context.impl.DefaultManagedContext.
> runWithContext(DefaultManagedContext.java:53)
> >   at
> >org.apache.cloudstack.managed.context.ManagedContextRunnable.run(
> ManagedContextRunnable.java:46)
> >   at
> >java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >   at
> >java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >   at
> >java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(
> ScheduledThreadPoolExecutor.java:292)
> >   at
> >java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> >   at
> >java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> >   at java.lang.Thread.run(Thread.java:745)
> >2016-09-26 02:30:22,053 ERROR [c.c.a.r.v.VirtualRoutingResource]
> >(DirectAgent-80:ctx-d0fb047f 10.31.56.182, job-65/job-66, cmd:
> >IpAssocVpcCommand) (logid:f727c105) Failed to prepare VR command due to
> >com.cloud.exception.InternalErrorException: Failed to find DomR VIF to
> >associate/disassociate IP with.
> >
> >With regards,
> >Prashanth
>
>

ACS 4.9 + VMware: Failed to deploy VPC VR due to IPAssoc failure

[Prashanth Manthena]

Hi All,

We are hitting an issue where VPC VR is failing to come up due to IpAssoc
failure in ACS+VMware setup.

We have tried with the following system VM templates:

http://cloudstack.apt-get.eu/systemvm/4.6/systemvm64template-4.6.0-vmware.ova

http://cloudstack.apt-get.eu/templates/4.7/systemvm64template-2016-05-18-4.7.0-vmware.ova


Is there a more latest system VM template that fixes this issue ?


Here is the error log:

[1:00]
IpAssocVpcCommand) (logid:f727c105) Prepare Ip Assoc failure on applying
one ip due to exception:
com.cloud.exception.InternalErrorException: Failed to find DomR VIF to
associate/disassociate IP with.
   at
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkElementCommand(VmwareResource.java:917)
   at
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareCommand(VmwareResource.java:803)
   at
com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:97)
   at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:387)
   at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
   at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
   at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
   at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
   at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
   at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   at java.lang.Thread.run(Thread.java:745)
2016-09-26 02:30:22,053 ERROR [c.c.a.r.v.VirtualRoutingResource]
(DirectAgent-80:ctx-d0fb047f 10.31.56.182, job-65/job-66, cmd:
IpAssocVpcCommand) (logid:f727c105) Failed to prepare VR command due to
com.cloud.exception.InternalErrorException: Failed to find DomR VIF to
associate/disassociate IP with.

With regards,
Prashanth

Re: Virtual router failed to start on KVM + Linux Bridges

[Prashanth Manthena]

Thank you Wei & Remi !!

I will start using the latest system VM templates.

With regards,
Prashanth

On Wed, May 11, 2016 at 2:20 PM, Remi Bergsma <rberg...@schubergphilis.com>
wrote:

> Hi Prashanth,
>
> As Wei suggested, when you use CloudStack 4.6/4.7/4.8/4.9 you also need
> the systemvm template with version 4.6.0. This does include the needed
> Python modules. The 4.4 template you used won't fly. You will run into
> other missing packages and changes as well.
>
> Regards, Remi
>
> Sent from my iPhone
>
> > On 11 May 2016, at 13:50, Prashanth Manthena <
> prashanth.manth...@nuagenetworks.net> wrote:
> >
> > Hi Will,
> >
> > I was able to put a break-point and debug the actual issue in the VR
> > (system VM template), it's a Python import error of the package
> "netaddr",
> > which can be fixed by updating the system template with the "netaddr"
> > package installed in it. Hopefully, it fixes this issue.
> >
> > Here is the error in VR's /var/log/cloud.log:
> >
> > Wed May 11 10:29:37 UTC 2016 : VR config: configuation format version 1.0
> > Wed May 11 10:29:37 UTC 2016 : VR config: creating file:
> > /var/cache/cloud/ip_asss
> > ociations.json
> > Wed May 11 10:29:37 UTC 2016 : VR config: create file success
> > Wed May 11 10:29:37 UTC 2016 : VR config: executing:
> > /opt/cloud/bin/update_confii
> > g.py ip_associations.json
> > Traceback (most recent call last):
> >  File "/opt/cloud/bin/update_config.py", line 20, in 
> >from merge import QueueFile
> >  File "/opt/cloud/bin/merge.py", line 23, in 
> >import cs_ip
> >  File "/opt/cloud/bin/cs_ip.py", line 19, in 
> >from netaddr import *
> > ImportError: No module named netaddr
> > Wed May 11 10:29:37 UTC 2016 : VR config: executing failed:
> > /opt/cloud/bin/updatt
> > e_config.py ip_associations.json
> >
> > With regards,
> > Prashanth
> >
> > On Wed, May 11, 2016 at 12:06 PM, Prashanth Manthena <
> > prashanth.manth...@nuagenetworks.net> wrote:
> >
> >> Hi Will,
> >>
> >> I am hitting an issue similar to CLOUDSTACK-9013 on master (4.9.0). Here
> >> are the management server error logs:
> >>
> >> 2016-05-10 08:24:03,109 ERROR
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> >> (logid:2735b868) VR config: execution failed:
> >> "/opt/cloud/bin/update_config.py ip_associations.json", check
> >> /var/log/cloud.log in VR for details
> >>
> >> 2016-05-10 08:24:03,109 WARN
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> >> (logid:2735b868) Command: com.cloud.agent.api.Command failed while
> starting
> >> virtual router
> >>
> >> 2016-05-10 08:24:03,109 INFO  [c.c.v.VirtualMachineManagerImpl]
> >> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> >> (logid:2735b868) The guru did not like the answers so stopping
> >> VM[DomainRouter|r-30-VM]
> >>
> >> Response:
> >>
> >> 2016-05-10 08:24:03,067 DEBUG [c.c.a.t.Request]
> >> (AgentManager-Handler-10:null) (logid:) Seq 1-4347381015296081926:
> >> Processing:  { Ans: , MgmtId: 275619426932736, via: 1, Ver: v1, Flags:
> 10,
> >>
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":30,"name":"r-30-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> >> GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
> >> 5","bootArgs":" template=domP name=r-30-VM eth2ip=10.100.100.117
> >> eth2mask=255.255.255.0 gateway=10.100.100.1 eth0ip=10.10.1.1
> >> eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24
> >> dhcprange=10.10.1.1 eth1ip=169.254.2.130 eth1mask=255.255.0.0
> type=router
> >> disable_rp_filter=true dns1=128.251.10.29 dns2=4.4.4.4
> >>
> baremetalnotificationsecuritykey=ODFTmygvs7oVAHVFZMmv7RpvsNNsawY9tf6sfeMvkUJlmEhWXGli5sO_oWAr52vrF18Z5xMgzjclu9P7G_w9Fg
> >>
> baremetalnotificationapikey=pguqxBGrlU2ItCuTHa7xb6pmjrpl9KbVMEE7o6TKrJ-mjDoaRayL_Nj21VkqrKvbtc10L8BxPqaYIhnGJkeHfw
> >> host=10.30.36.11
> >>
> port=8080","enableHA":true,"limitCpuUse"

Re: Virtual router failed to start on KVM + Linux Bridges

[Prashanth Manthena]

Hi Will,

I was able to put a break-point and debug the actual issue in the VR
(system VM template), it's a Python import error of the package "netaddr",
which can be fixed by updating the system template with the "netaddr"
package installed in it. Hopefully, it fixes this issue.

Here is the error in VR's /var/log/cloud.log:

Wed May 11 10:29:37 UTC 2016 : VR config: configuation format version 1.0
Wed May 11 10:29:37 UTC 2016 : VR config: creating file:
/var/cache/cloud/ip_asss
ociations.json
Wed May 11 10:29:37 UTC 2016 : VR config: create file success
Wed May 11 10:29:37 UTC 2016 : VR config: executing:
/opt/cloud/bin/update_confii
g.py ip_associations.json
Traceback (most recent call last):
  File "/opt/cloud/bin/update_config.py", line 20, in 
from merge import QueueFile
  File "/opt/cloud/bin/merge.py", line 23, in 
import cs_ip
  File "/opt/cloud/bin/cs_ip.py", line 19, in 
from netaddr import *
ImportError: No module named netaddr
Wed May 11 10:29:37 UTC 2016 : VR config: executing failed:
/opt/cloud/bin/updatt
e_config.py ip_associations.json

With regards,
Prashanth

On Wed, May 11, 2016 at 12:06 PM, Prashanth Manthena <
prashanth.manth...@nuagenetworks.net> wrote:

> Hi Will,
>
> I am hitting an issue similar to CLOUDSTACK-9013 on master (4.9.0). Here
> are the management server error logs:
>
> 2016-05-10 08:24:03,109 ERROR [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> (logid:2735b868) VR config: execution failed:
> "/opt/cloud/bin/update_config.py ip_associations.json", check
> /var/log/cloud.log in VR for details
>
> 2016-05-10 08:24:03,109 WARN  [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> (logid:2735b868) Command: com.cloud.agent.api.Command failed while starting
> virtual router
>
> 2016-05-10 08:24:03,109 INFO  [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
> (logid:2735b868) The guru did not like the answers so stopping
> VM[DomainRouter|r-30-VM]
>
> Response:
>
> 2016-05-10 08:24:03,067 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-10:null) (logid:) Seq 1-4347381015296081926:
> Processing:  { Ans: , MgmtId: 275619426932736, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":30,"name":"r-30-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
> GNU/Linux 5.0 (64-bit)","platformEmulator":"Debian GNU/Linux
> 5","bootArgs":" template=domP name=r-30-VM eth2ip=10.100.100.117
> eth2mask=255.255.255.0 gateway=10.100.100.1 eth0ip=10.10.1.1
> eth0mask=255.255.255.0 domain=cs2cloud.internal cidrsize=24
> dhcprange=10.10.1.1 eth1ip=169.254.2.130 eth1mask=255.255.0.0 type=router
> disable_rp_filter=true dns1=128.251.10.29 dns2=4.4.4.4
> baremetalnotificationsecuritykey=ODFTmygvs7oVAHVFZMmv7RpvsNNsawY9tf6sfeMvkUJlmEhWXGli5sO_oWAr52vrF18Z5xMgzjclu9P7G_w9Fg
> baremetalnotificationapikey=pguqxBGrlU2ItCuTHa7xb6pmjrpl9KbVMEE7o6TKrJ-mjDoaRayL_Nj21VkqrKvbtc10L8BxPqaYIhnGJkeHfw
> host=10.30.36.11
> port=8080","enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"Pj8GDPW2vqLfkdOMoCOogg","vncAddr":"10.100.100.11","params":{},"uuid":"f406776f-3b4c-4d23-917e-715ceaea0a7a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"f44bbb4e-a788-4034-9389-dc1fa47dce70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"63a122cd-ac74-3cc1-85b0-e3ad105234dc","id":1,"poolType":"NetworkFilesystem","host":"
> andcvtb05.be.alcatel-lucent.com
> ","path":"/andcvtb05/storage","port":2049,"url":"NetworkFilesystem://
> andcvtb05.be.alcatel-lucent.com/andcvtb05/storage/?ROLE=Primary=63a122cd-ac74-3cc1-85b0-e3ad105234dc
> "}},"name":"ROOT-30","size":304774144,"path":"f44bbb4e-a788-4034-9389-dc1fa47dce70","volumeId":30,"vmName":"r-30-VM","accountId":2,"format":"QCOW2","provisioningType":"THIN","id":30,"deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path"

Re: Virtual router failed to start on KVM + Linux Bridges

[Prashanth Manthena]

0:6b:7b:00:02","dns1":"128.251.10.29","dns2":"4.4.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://300","isolationUri":"vlan://300","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"fe94aabc-705f-47e3-9981-0139acf1af3d","uuid":"6f1abde1-d0cb-4411-9e9b-9db1fdd5f7ab","ip":"169.254.2.130","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:82","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":true,"wait":0}},{"com.cloud.agent.api.GetDomRVersionAnswer":{"templateVersion":"Cloudstack
Release 4.4.0 Wed Mar 26 23:26:41 UTC
2014","scriptsVersion":"e54d6a03fe7d28ff66432d33c8897934\n","result":true,"details":"Cloudstack
Release 4.4.0 Wed Mar 26 23:26:41 UTC
2014\n","wait":0}},{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-30-VM","bytesSent":0,"bytesReceived":0,"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"details":"Command
aggregation
started","wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":true,"wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"VR
config: execution failed: \"/opt/cloud/bin/update_config.py
ip_associations.json\", check /var/log/cloud.log in VR for details
","wait":0}}] }
2016-05-10 08:24:03,068 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
(logid:2735b868) Seq 1-4347381015296081926: Received:  { Ans: , MgmtId:
275619426932736, via: 1(ovs-1.andcvtb05.be.alcatel-lucent.com), Ver: v1,
Flags: 10, { StartAnswer, CheckSshAnswer, GetDomRVersionAnswer,
NetworkUsageAnswer, Answer, Answer, Answer, Answer, Answer } }
2016-05-10 08:24:03,091 WARN  [o.a.c.alerts]
(Work-Job-Executor-3:ctx-5069ea04 job-247/job-250 ctx-48a763a9)
(logid:2735b868)  alertType:: 9 // dataCenterId:: 2 // podId:: 2 //
clusterId:: null // message:: Command: com.cloud.agent.api.Command failed
while starting virtual router


With regards,
Prashanth




On Tue, May 10, 2016 at 6:18 PM, Will Stevens <wstev...@cloudops.com> wrote:

> I am doing CI on KVM and I believe I am using bridged networking in this
> environment.  Can you give more details of what you are seeing?
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, May 10, 2016 at 11:51 AM, Prashanth Manthena <
> prashanth.manth...@nuagenetworks.net> wrote:
>
> > Hi All,
> >
> >  I am seeing CLOUDSTACK-9013 on master (4.9.0) with KVM+Linux Bridges.
> >
> > Any pointers on this issue ?
> >
> > CLOUDSTACK-9013: https://issues.apache.org/jira/browse/CLOUDSTACK-9013
> >
> > With reagrds,
> > Prashanth
> >
>

Virtual router failed to start on KVM + Linux Bridges

[Prashanth Manthena]

Hi All,

 I am seeing CLOUDSTACK-9013 on master (4.9.0) with KVM+Linux Bridges.

Any pointers on this issue ?

CLOUDSTACK-9013: https://issues.apache.org/jira/browse/CLOUDSTACK-9013

With reagrds,
Prashanth

Re: Python Question (with regards to Marvin)

[Prashanth Manthena]

Hi Mike,

classmethod's first argument is always its class as it is always bound to
the class, which is not the case with normal methods. Thus, both of the
above methods can in essence still co-exist in base.py as they serve
different purposes.

In the latter classmethod, you need to pass your host object ID as one of
the argument while calling it with its class name as its first argument.

e.g. Host.enableMaintenance(self.apiclient, host_1.id)

Note: "Host" is the class name of the above methods in base.py. "host_1" is
your host object.

In the former method, you just need to call it with your host object as its
first argument, it takes the host ID from the passed host object.

e.g. host_1.enableMaintenance(self.apiclient)


On Sat, Apr 30, 2016 at 7:08 AM, Tutkowski, Mike 
wrote:

> Hi everyone,
>
>
> I received an error when trying to invoke the instance version of
> enableMaintenance (below).
>
>
> 'TypeError: enableMaintenance() takes exactly 3 arguments (2 given)\n']
>
>
> I looked at base.py and it has the following with regards to maintenance
> mode for hosts:
>
>
> def enableMaintenance(self, apiclient):
>
> """enables maintenance mode Host"""
>
>
> cmd = prepareHostForMaintenance.prepareHostForMaintenanceCmd()
>
> cmd.id = self.id
>
> return apiclient.prepareHostForMaintenance(cmd)
>
>
> @classmethod
>
> def enableMaintenance(cls, apiclient, id):
>
> """enables maintenance mode Host"""
>
>
> cmd = prepareHostForMaintenance.prepareHostForMaintenanceCmd()
>
> cmd.id = id
>
> return apiclient.prepareHostForMaintenance(cmd)
>
>
> Now, I definitely have a lot more Java experience than Python, but - as
> far as I know - having two methods with the same name such as this (even if
> one is an instance method and the other is a class method) is not really
> "permitted" in Python.
>
>
> I mean, technically it's permitted, but the second one will override the
> first one.
>
>
> Can any of our Python people comment on this?
>
>
> I was thinking I'd remove the class method (assuming my knowledge here
> regarding this topic is correct).
>
>
> Thanks!
>
> Mike
>
>
>
>

Re: Another question about Marvin

[Prashanth Manthena]

Hi Will,

1) will run a, b, c
2) will run only b
3) will run only c

Nosetests will run all the testcases that match attribute (tags)
combination you specify in the command.

Default (no tag(s) specified): It will run all the testcases.

With regards,
Prashanth



On Wed, Apr 27, 2016 at 4:30 PM, Will Stevens 
wrote:

> So I see a few different ways in which tests are defined.
>
> For example, here are a few different attribute configurations:
>
> a) @attr(tags=["advanced", "intervlan"])
> b) @attr(tags=["advanced", "intervlan"], required_hardware="true")
> c) @attr(tags=["advanced", "intervlan"], required_hardware="false")
>
> Then there are different ways to run tests.  Some examples are:
>
> 1) nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced
> 2) nosetests --with-marvin --marvin-config=${marvinCfg} -s -a
> tags=advanced,required_hardware=true
> 3) nosetests --with-marvin --marvin-config=${marvinCfg} -s -a
> tags=advanced,required_hardware=false
>
> I am trying to figure out which tests are run in each case.
>
> I think 2) is the easiest case because it should only run b).
>
> 3) is a little less clear to me because I know it will run c), but will it
> run a)?
>
> 1) is the least clear to me because I don't know if it will run any of the
> 'required_hardware' tests.
>
> Can someone drop a little knowledge on me?
>
> Thanks,
>
> Will
>

Failed to create VPC: Unable to start VPC VR (VM DomainRouter) due to error in finalizeStart, not retrying

[Prashanth Manthena]

Hi All,

I am seeing the following issue on master:
https://issues.apache.org/jira/browse/CLOUDSTACK-9370.

Is anyone else seeing this issue on master ?

Let me know, if you need more information from my side.

With regards,
Prashanth

Request for CloudStack Dev Slack channel access

[Prashanth Manthena]

Hi,

Can you please grant me access and point me to the CloudStack Dev Slack
channel.

With regards,
Prashanth

Re: Request for CloudStack Dev Slack channel access

[Prashanth Manthena]

Thank you, Daan :)

On Fri, Nov 20, 2015 at 11:47 AM, Daan Hoogland <daan.hoogl...@gmail.com>
wrote:

> yab
>
> On Fri, Nov 20, 2015 at 11:32 AM, Prashanth Manthena <
> prashanth.manth...@nuagenetworks.net> wrote:
>
> > Hi,
> >
> > Can you please grant me access and point me to the CloudStack Dev Slack
> > channel.
> >
> > With regards,
> > Prashanth
> >
>
>
>
> --
> Daan
>

[Marvin] Failed to wget from VM in "/test/integration/component/test_vpc_network_staticnatrule.py"

[Prashanth Manthena]

Hi All,

I get an error "Failed to wget from VM..." while running the Marvin
test "/test/integration/component/test_vpc_network_staticnatrule.py" with
CentOS 5.5 template (KVM hypervisor).

I understand that a http server service needs to be initialized/installed
in the VM template for this test to run successfully. Refer:
https://issues.apache.org/jira/browse/CLOUDSTACK-5623

Anyhow, I just want to confirm the following:

1) Does the default CentOS 5.3 (XenServer) template meet all the
requirements for this Marvin test to run succesfully ?

2) Are there any proposals to make this Marvin test generic (e.g. changing
the actual test code to start a http server service on the guest VMs during
the actual test run) ?


Looking forward for your responses !!



With regards,
Prashanth