Re: 4.9/master Testing Coordination
Hi Will, I've created a PR that aims to make Nio related connections slightly aggressive which should solve any timeout/reconnection issues. If you're still seeing the issue with *latest* master, please test and merge this PR https://github.com/apache/cloudstack/pull/1601 I could not reproduce the issue at all, in all cases it worked without an issue in my environment. Please, do let me know if hit the issue at all in your environment. Regards. From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: 29 June 2016 19:39:00 To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination We are keeping an eye on the NIO SSH stuff because we have had reports of the connection being lost for more than 10 minutes (before manual intervention), but it seems intermittent. There is also a potential issue with the VR coming up with only two nics instead of three, but there are conflicting reports on that one, so keep an eye out for that too. Most of the details of the things people have come up against are in this thread. I will be camping till monday, so I will have very limited connectivity. I will hopefully cut the RC when I get back, so please let me know if you find anything this week. Thanks, Will *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Jun 29, 2016 at 9:15 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Hi Will, > > What's the current situation with the 4.9 release? > I'm helping John with setting up the environments which he's listed for > testing. Do we have a list of any blocker bugs that need clearing or a > timetable? > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 28 June 2016 18:16 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Rohit, > Simon and his team is in the final push getting their next internal > production release ready to ship, so their testing of 4.9 has paused while > they get that ready. I suspect he and his team will have limited time to > dedicate to this in this week. > > Just figured I would pass that along if he is not able to get back to you > quickly so you understand why there may be a delay. :) > > Cheers, > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Simon, > > > > > > Thanks for sharing. I'm continuing the discussion from a PR ( > > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > > here wrt NIO ssl agent connection. Are you using latest master, and > > are you able to consistently reproduce the issue? Make sure that both > > management server and kvm-agent (or ssvm agents etc.) are using the > > packages built from latest master as well (i.e. updating both agent > > and mgmt server pkg jars is necessary). > > > > > > > ""I did also run into the NIO SSL agent not connecting problem again. > > When I telnetted to 8250, the agent immediately came up without me > > having to restart it. So keep an eye out for that as well."" > > > > > > When you ran into the issue, what was the environment like. Did you > > have to wait long enough before you decided to telnet on agent's port > > 8250? The current master's NioConnection code makes sure to check for > > new channels/connections every one second, this change ensures that > > reconnections are fast and non-blocking. > > > > > > Regards. > > > > > > From: Simon Weller <swel...@ena.com> > > Sent: 18 June 2016 02:17:08 > > To: dev@cloudstack.apache.org > > Subject: Re: 4.9/master Testing Coordination > > > > Here's a quick run down on the configuration(s) we're testing: > > > > > > Centos 7.2 > > > > Advanced Zone with VXLAN on KVM > > > > VPC functionality including Private GW, VPN, Static Routes, ACL Lists > > et al > > > > Redundant VPC VRs > > > > Ceph Primary Storage > > > > NFS and S3 secondary storage > > > > As Will mentioned, we've
Re: 4.9/master Testing Coordination
Great, thanks for the support Paul. :) *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Jun 29, 2016 at 10:12 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Thanks for the update Will. > I'll look for the VR issue, I think that I've it in 4.5 intermittently on > vmware. - in fact I've seen it come up with 6 interfaces before... > > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 29 June 2016 15:09 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > We are keeping an eye on the NIO SSH stuff because we have had reports of > the connection being lost for more than 10 minutes (before manual > intervention), but it seems intermittent. > > There is also a potential issue with the VR coming up with only two nics > instead of three, but there are conflicting reports on that one, so keep an > eye out for that too. > > Most of the details of the things people have come up against are in this > thread. > > I will be camping till monday, so I will have very limited connectivity. > I will hopefully cut the RC when I get back, so please let me know if you > find anything this week. > > Thanks, > > Will > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Wed, Jun 29, 2016 at 9:15 AM, Paul Angus <paul.an...@shapeblue.com> > wrote: > > > Hi Will, > > > > What's the current situation with the 4.9 release? > > I'm helping John with setting up the environments which he's listed > > for testing. Do we have a list of any blocker bugs that need clearing > > or a timetable? > > > > Kind regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > -Original Message- > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > > Behalf Of Will Stevens > > Sent: 28 June 2016 18:16 > > To: dev@cloudstack.apache.org > > Subject: Re: 4.9/master Testing Coordination > > > > Rohit, > > Simon and his team is in the final push getting their next internal > > production release ready to ship, so their testing of 4.9 has paused > while > > they get that ready. I suspect he and his team will have limited time to > > dedicate to this in this week. > > > > Just figured I would pass that along if he is not able to get back to you > > quickly so you understand why there may be a delay. :) > > > > Cheers, > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > > @CloudOps_ > > > > On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > > wrote: > > > > > Hi Simon, > > > > > > > > > Thanks for sharing. I'm continuing the discussion from a PR ( > > > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > > > here wrt NIO ssl agent connection. Are you using latest master, and > > > are you able to consistently reproduce the issue? Make sure that both > > > management server and kvm-agent (or ssvm agents etc.) are using the > > > packages built from latest master as well (i.e. updating both agent > > > and mgmt server pkg jars is necessary). > > > > > > > > > > ""I did also run into the NIO SSL agent not connecting problem again. > > > When I telnetted to 8250, the agent immediately came up without me > > > having to restart it. So keep an eye out for that as well."" > > > > > > > > > When you ran into the issue, what was the environment like. Did you > > > have to wait long enough before you decided to telnet on agent's port > > > 8250? The current master's NioConnection code makes sure to check for > > > new channels/connections every one second, this change ensures that > > > reconnections are fast and non-blocking. > > > > > > > > > Regards. > > > > > > __
RE: 4.9/master Testing Coordination
Np. Happy 'tenting' :) Kind regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On Behalf Of Will Stevens Sent: 29 June 2016 15:16 To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Great, thanks for the support Paul. :) *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Jun 29, 2016 at 10:12 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Thanks for the update Will. > I'll look for the VR issue, I think that I've it in 4.5 intermittently > on vmware. - in fact I've seen it come up with 6 interfaces before... > > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > -Original Message- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 29 June 2016 15:09 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > We are keeping an eye on the NIO SSH stuff because we have had reports of > the connection being lost for more than 10 minutes (before manual > intervention), but it seems intermittent. > > There is also a potential issue with the VR coming up with only two nics > instead of three, but there are conflicting reports on that one, so keep an > eye out for that too. > > Most of the details of the things people have come up against are in this > thread. > > I will be camping till monday, so I will have very limited connectivity. > I will hopefully cut the RC when I get back, so please let me know if you > find anything this week. > > Thanks, > > Will > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Wed, Jun 29, 2016 at 9:15 AM, Paul Angus <paul.an...@shapeblue.com> > wrote: > > > Hi Will, > > > > What's the current situation with the 4.9 release? > > I'm helping John with setting up the environments which he's listed > > for testing. Do we have a list of any blocker bugs that need clearing > > or a timetable? > > > > Kind regards, > > > > Paul Angus > > > > paul.an...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > > > > > > -Original Message- > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > > Behalf Of Will Stevens > > Sent: 28 June 2016 18:16 > > To: dev@cloudstack.apache.org > > Subject: Re: 4.9/master Testing Coordination > > > > Rohit, > > Simon and his team is in the final push getting their next internal > > production release ready to ship, so their testing of 4.9 has paused > while > > they get that ready. I suspect he and his team will have limited time to > > dedicate to this in this week. > > > > Just figured I would pass that along if he is not able to get back to you > > quickly so you understand why there may be a delay. :) > > > > Cheers, > > > > *Will STEVENS* > > Lead Developer > > > > *CloudOps* *| *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > > @CloudOps_ > > > > On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > > wrote: > > > > > Hi Simon, > > > > > > > > > Thanks for sharing. I'm continuing the discussion from a PR ( > > > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > > > here wrt NIO ssl agent connection. Are you using latest master, and > > > are you able to consistently reproduce the issue? Make sure that both > > > management server and kvm-agent (or ssvm agents etc.) are using the > > > packages built from latest master as well (i.e. updating both agent > > > and mgmt server pkg jars is necessary). > > > > > > > > > > ""I did also run into the NIO SSL agent not connecting problem again. > > > When I telnetted to 8250, the agent immediately came up without me > > > having to restart it. So keep an eye out for that as well."" > > > > > > > > > When you ran into the issue, what was the environment like. Did you > > > hav
RE: 4.9/master Testing Coordination
Thanks for the update Will. I'll look for the VR issue, I think that I've it in 4.5 intermittently on vmware. - in fact I've seen it come up with 6 interfaces before... Kind regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On Behalf Of Will Stevens Sent: 29 June 2016 15:09 To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination We are keeping an eye on the NIO SSH stuff because we have had reports of the connection being lost for more than 10 minutes (before manual intervention), but it seems intermittent. There is also a potential issue with the VR coming up with only two nics instead of three, but there are conflicting reports on that one, so keep an eye out for that too. Most of the details of the things people have come up against are in this thread. I will be camping till monday, so I will have very limited connectivity. I will hopefully cut the RC when I get back, so please let me know if you find anything this week. Thanks, Will *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Jun 29, 2016 at 9:15 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Hi Will, > > What's the current situation with the 4.9 release? > I'm helping John with setting up the environments which he's listed > for testing. Do we have a list of any blocker bugs that need clearing > or a timetable? > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > > -Original Message- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 28 June 2016 18:16 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Rohit, > Simon and his team is in the final push getting their next internal > production release ready to ship, so their testing of 4.9 has paused while > they get that ready. I suspect he and his team will have limited time to > dedicate to this in this week. > > Just figured I would pass that along if he is not able to get back to you > quickly so you understand why there may be a delay. :) > > Cheers, > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Simon, > > > > > > Thanks for sharing. I'm continuing the discussion from a PR ( > > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > > here wrt NIO ssl agent connection. Are you using latest master, and > > are you able to consistently reproduce the issue? Make sure that both > > management server and kvm-agent (or ssvm agents etc.) are using the > > packages built from latest master as well (i.e. updating both agent > > and mgmt server pkg jars is necessary). > > > > > > > ""I did also run into the NIO SSL agent not connecting problem again. > > When I telnetted to 8250, the agent immediately came up without me > > having to restart it. So keep an eye out for that as well."" > > > > > > When you ran into the issue, what was the environment like. Did you > > have to wait long enough before you decided to telnet on agent's port > > 8250? The current master's NioConnection code makes sure to check for > > new channels/connections every one second, this change ensures that > > reconnections are fast and non-blocking. > > > > > > Regards. > > > > > > From: Simon Weller <swel...@ena.com> > > Sent: 18 June 2016 02:17:08 > > To: dev@cloudstack.apache.org > > Subject: Re: 4.9/master Testing Coordination > > > > Here's a quick run down on the configuration(s) we're testing: > > > > > > Centos 7.2 > > > > Advanced Zone with VXLAN on KVM > > > > VPC functionality including Private GW, VPN, Static Routes, ACL Lists > > et al > > > > Redundant VPC VRs > > > > Ceph Primary Storage > > > > NFS and S3 secondary storage > > > > As Will mentioned, we've found an odd issue with VPCs that we're still > > debugging. > > > > Here's a summary of what we've found thus far: > > When a tier is added, the network interface for the tier network is >
Re: 4.9/master Testing Coordination
We are keeping an eye on the NIO SSH stuff because we have had reports of the connection being lost for more than 10 minutes (before manual intervention), but it seems intermittent. There is also a potential issue with the VR coming up with only two nics instead of three, but there are conflicting reports on that one, so keep an eye out for that too. Most of the details of the things people have come up against are in this thread. I will be camping till monday, so I will have very limited connectivity. I will hopefully cut the RC when I get back, so please let me know if you find anything this week. Thanks, Will *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Wed, Jun 29, 2016 at 9:15 AM, Paul Angus <paul.an...@shapeblue.com> wrote: > Hi Will, > > What's the current situation with the 4.9 release? > I'm helping John with setting up the environments which he's listed for > testing. Do we have a list of any blocker bugs that need clearing or a > timetable? > > Kind regards, > > Paul Angus > > paul.an...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > -Original Message- > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On > Behalf Of Will Stevens > Sent: 28 June 2016 18:16 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Rohit, > Simon and his team is in the final push getting their next internal > production release ready to ship, so their testing of 4.9 has paused while > they get that ready. I suspect he and his team will have limited time to > dedicate to this in this week. > > Just figured I would pass that along if he is not able to get back to you > quickly so you understand why there may be a delay. :) > > Cheers, > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Simon, > > > > > > Thanks for sharing. I'm continuing the discussion from a PR ( > > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > > here wrt NIO ssl agent connection. Are you using latest master, and > > are you able to consistently reproduce the issue? Make sure that both > > management server and kvm-agent (or ssvm agents etc.) are using the > > packages built from latest master as well (i.e. updating both agent > > and mgmt server pkg jars is necessary). > > > > > > > ""I did also run into the NIO SSL agent not connecting problem again. > > When I telnetted to 8250, the agent immediately came up without me > > having to restart it. So keep an eye out for that as well."" > > > > > > When you ran into the issue, what was the environment like. Did you > > have to wait long enough before you decided to telnet on agent's port > > 8250? The current master's NioConnection code makes sure to check for > > new channels/connections every one second, this change ensures that > > reconnections are fast and non-blocking. > > > > > > Regards. > > > > > > From: Simon Weller <swel...@ena.com> > > Sent: 18 June 2016 02:17:08 > > To: dev@cloudstack.apache.org > > Subject: Re: 4.9/master Testing Coordination > > > > Here's a quick run down on the configuration(s) we're testing: > > > > > > Centos 7.2 > > > > Advanced Zone with VXLAN on KVM > > > > VPC functionality including Private GW, VPN, Static Routes, ACL Lists > > et al > > > > Redundant VPC VRs > > > > Ceph Primary Storage > > > > NFS and S3 secondary storage > > > > As Will mentioned, we've found an odd issue with VPCs that we're still > > debugging. > > > > Here's a summary of what we've found thus far: > > When a tier is added, the network interface for the tier network is > > never plugged by libvirt. You only get 2 interfaces (eth0 and eth1). > > eth2 is never plugged when you attempt to provision the first VM > > within the VPC and the VM creation fails. If you have existing VMs and > > you restart the router, you lose the eth2 interface in the libvirt > > configuration (confirmed with a virsh dumpxml). > > If you leave the VRs alone after the upgrade, VMs can be provisioned > > correctly (until you reboot the VRs). > > > > I did also run into the
RE: 4.9/master Testing Coordination
Hi Will, What's the current situation with the 4.9 release? I'm helping John with setting up the environments which he's listed for testing. Do we have a list of any blocker bugs that need clearing or a timetable? Kind regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -Original Message- From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On Behalf Of Will Stevens Sent: 28 June 2016 18:16 To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Rohit, Simon and his team is in the final push getting their next internal production release ready to ship, so their testing of 4.9 has paused while they get that ready. I suspect he and his team will have limited time to dedicate to this in this week. Just figured I would pass that along if he is not able to get back to you quickly so you understand why there may be a delay. :) Cheers, *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi Simon, > > > Thanks for sharing. I'm continuing the discussion from a PR ( > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > here wrt NIO ssl agent connection. Are you using latest master, and > are you able to consistently reproduce the issue? Make sure that both > management server and kvm-agent (or ssvm agents etc.) are using the > packages built from latest master as well (i.e. updating both agent > and mgmt server pkg jars is necessary). > > > > ""I did also run into the NIO SSL agent not connecting problem again. > When I telnetted to 8250, the agent immediately came up without me > having to restart it. So keep an eye out for that as well."" > > > When you ran into the issue, what was the environment like. Did you > have to wait long enough before you decided to telnet on agent's port > 8250? The current master's NioConnection code makes sure to check for > new channels/connections every one second, this change ensures that > reconnections are fast and non-blocking. > > > Regards. > > > From: Simon Weller <swel...@ena.com> > Sent: 18 June 2016 02:17:08 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Here's a quick run down on the configuration(s) we're testing: > > > Centos 7.2 > > Advanced Zone with VXLAN on KVM > > VPC functionality including Private GW, VPN, Static Routes, ACL Lists > et al > > Redundant VPC VRs > > Ceph Primary Storage > > NFS and S3 secondary storage > > As Will mentioned, we've found an odd issue with VPCs that we're still > debugging. > > Here's a summary of what we've found thus far: > When a tier is added, the network interface for the tier network is > never plugged by libvirt. You only get 2 interfaces (eth0 and eth1). > eth2 is never plugged when you attempt to provision the first VM > within the VPC and the VM creation fails. If you have existing VMs and > you restart the router, you lose the eth2 interface in the libvirt > configuration (confirmed with a virsh dumpxml). > If you leave the VRs alone after the upgrade, VMs can be provisioned > correctly (until you reboot the VRs). > > I did also run into the NIO SSL agent not connecting problem again. > When I telnetted to 8250, the agent immediately came up without me > having to restart it. So keep an eye out for that as well. > > > - Si > > ____ > From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of > Will Stevens <wstev...@cloudops.com> > Sent: Friday, June 17, 2016 3:02 PM > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Syed is looking into a potential issue with Swift as secondary storage > on master. > > ENA is looking into a potential problem when upgrading a VR which was > working in 4.8.0, but after an upgrade to 4.9.0 and restarting the > network there are only 2 nics instead of 3. If they spin a new VR > from scratch it seems to work. I need to follow up with them to see > if they have an updated status of their testing. > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > > rohit.ya...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue > > > > On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com&
Re: 4.9/master Testing Coordination
Rohit, Simon and his team is in the final push getting their next internal production release ready to ship, so their testing of 4.9 has paused while they get that ready. I suspect he and his team will have limited time to dedicate to this in this week. Just figured I would pass that along if he is not able to get back to you quickly so you understand why there may be a delay. :) Cheers, *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Tue, Jun 28, 2016 at 1:08 PM, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi Simon, > > > Thanks for sharing. I'm continuing the discussion from a PR ( > https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) > here wrt NIO ssl agent connection. Are you using latest master, and are you > able to consistently reproduce the issue? Make sure that both management > server and kvm-agent (or ssvm agents etc.) are using the packages built > from latest master as well (i.e. updating both agent and mgmt server pkg > jars is necessary). > > > > ""I did also run into the NIO SSL agent not connecting problem again. > When I telnetted to 8250, the agent immediately came up without me having > to restart it. So keep an eye out for that as well."" > > > When you ran into the issue, what was the environment like. Did you have > to wait long enough before you decided to telnet on agent's port 8250? The > current master's NioConnection code makes sure to check for new > channels/connections every one second, this change ensures that > reconnections are fast and non-blocking. > > > Regards. > > > From: Simon Weller <swel...@ena.com> > Sent: 18 June 2016 02:17:08 > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Here's a quick run down on the configuration(s) we're testing: > > > Centos 7.2 > > Advanced Zone with VXLAN on KVM > > VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al > > Redundant VPC VRs > > Ceph Primary Storage > > NFS and S3 secondary storage > > As Will mentioned, we've found an odd issue with VPCs that we're still > debugging. > > Here's a summary of what we've found thus far: > When a tier is added, the network interface for the tier network is never > plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is > never plugged when you attempt to provision the first VM within the VPC and > the VM creation fails. If you have existing VMs and you restart the router, > you lose the eth2 interface in the libvirt configuration (confirmed with a > virsh dumpxml). > If you leave the VRs alone after the upgrade, VMs can be provisioned > correctly (until you reboot the VRs). > > I did also run into the NIO SSL agent not connecting problem again. When I > telnetted to 8250, the agent immediately came up without me having to > restart it. So keep an eye out for that as well. > > > - Si > > ________ > From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of > Will Stevens <wstev...@cloudops.com> > Sent: Friday, June 17, 2016 3:02 PM > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Syed is looking into a potential issue with Swift as secondary storage on > master. > > ENA is looking into a potential problem when upgrading a VR which was > working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network > there are only 2 nics instead of 3. If they spin a new VR from scratch it > seems to work. I need to follow up with them to see if they have an > updated status of their testing. > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > w cloudops.com *|* tw @CloudOps_ > > > rohit.ya...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> > wrote: > > > The following tests are running cleanly on KVM with Advanced networking > > and 2 hosts. > > > > echo "Running tests with required_hardware=true" > > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > > tags=advanced,required_hardware=true \ > > smoke/test_password_server.py \ > > smoke/test_vpc_redundant.py \ > > smoke/test_routers_iptables_default_policy.py \ > > smoke/test_routers_network_ops.py \ > > smoke/test_vpc_router_nics.py \ > > smoke/test_router_dhcphosts.py \ > > sm
Re: 4.9/master Testing Coordination
Hi Simon, Thanks for sharing. I'm continuing the discussion from a PR (https://github.com/apache/cloudstack/pull/1549#issuecomment-229070969) here wrt NIO ssl agent connection. Are you using latest master, and are you able to consistently reproduce the issue? Make sure that both management server and kvm-agent (or ssvm agents etc.) are using the packages built from latest master as well (i.e. updating both agent and mgmt server pkg jars is necessary). > ""I did also run into the NIO SSL agent not connecting problem again. When I > telnetted to 8250, the agent immediately came up without me having to restart > it. So keep an eye out for that as well."" When you ran into the issue, what was the environment like. Did you have to wait long enough before you decided to telnet on agent's port 8250? The current master's NioConnection code makes sure to check for new channels/connections every one second, this change ensures that reconnections are fast and non-blocking. Regards. From: Simon Weller <swel...@ena.com> Sent: 18 June 2016 02:17:08 To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Here's a quick run down on the configuration(s) we're testing: Centos 7.2 Advanced Zone with VXLAN on KVM VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al Redundant VPC VRs Ceph Primary Storage NFS and S3 secondary storage As Will mentioned, we've found an odd issue with VPCs that we're still debugging. Here's a summary of what we've found thus far: When a tier is added, the network interface for the tier network is never plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is never plugged when you attempt to provision the first VM within the VPC and the VM creation fails. If you have existing VMs and you restart the router, you lose the eth2 interface in the libvirt configuration (confirmed with a virsh dumpxml). If you leave the VRs alone after the upgrade, VMs can be provisioned correctly (until you reboot the VRs). I did also run into the NIO SSL agent not connecting problem again. When I telnetted to 8250, the agent immediately came up without me having to restart it. So keep an eye out for that as well. - Si From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: Friday, June 17, 2016 3:02 PM To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Syed is looking into a potential issue with Swift as secondary storage on master. ENA is looking into a potential problem when upgrading a VR which was working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network there are only 2 nics instead of 3. If they spin a new VR from scratch it seems to work. I need to follow up with them to see if they have an updated status of their testing. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: > The following tests are running cleanly on KVM with Advanced networking > and 2 hosts. > > echo "Running tests with required_hardware=true" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=true \ > smoke/test_password_server.py \ > smoke/test_vpc_redundant.py \ > smoke/test_routers_iptables_default_policy.py \ > smoke/test_routers_network_ops.py \ > smoke/test_vpc_router_nics.py \ > smoke/test_router_dhcphosts.py \ > smoke/test_loadbalance.py \ > smoke/test_internal_lb.py \ > smoke/test_ssvm.py \ > smoke/test_vpc_vpn.py \ > smoke/test_privategw_acl.py \ > smoke/test_network.py > > echo "Running tests with required_hardware=false" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=false \ > smoke/test_routers.py \ > smoke/test_network_acl.py \ > smoke/test_reset_vm_on_reboot.py \ > smoke/test_vm_life_cycle.py \ > smoke/test_service_offerings.py \ > smoke/test_network.py \ > component/test_vpc_offerings.py \ > component/test_vpc_routers.py > > I need to do some more manual testing... > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > w cloudops.com *|* tw @CloudOps_ > > On Fri, Jun 17, 2016 at 3:45 PM, Tutkowski, Mike < > mike.tutkow...@netapp.com> wrote: > >> My testing has been performed using XenServer 6.5 and ESXi 5.5. >> >> I executed all of the tests in test/integration/plugins/solidfire. &
Re: 4.9/master Testing Coordination
I am testing the following configuration: CentOS 6.7 Mgmt Server + XenServer 6.5 + SolidFire (using Managed pluign) + Swift (Secondary Storage). I've hit the following issue when uploading templates: https://github.com/apache/cloudstack/pull/1598/ Would someone be able to review this please? Thanks, -Syed On Thu, Jun 23, 2016 at 4:01 PM, John Burwell <john.burw...@shapeblue.com> wrote: > All, > > I had hoped to get testing off the ground earlier, but $dayjob duties have > gotten in the way. Over the weekend, I am planning to kick off tests of > the following configurations: > > • CentOS 6.8 management server + CentOS 6.8 KVM Hosts using NFS > primary and secondary storage (would allow us to verify/fix the documented > libvirt/qemu versions) > • CentOS 6.8 management server + vCenter 5.5u3d + ESXi 5.5u3b > using NFS primary and secondary storage > • CentOS 6.8 management server + vCenter 6.0u2 + ESXi Express > Patch 6 using NFS primary and secondary storage > • CentOS 6.8 management server + XenServer 6.2 SP1 using NFS > primary and secondary storage > • CentOS 6.8 management server + XenServer 6.5 SP1 using NFS > primary and secondary storage > > In each of these environments, I plan to run the following tests: > > • All smoke tests > • Component Tests > • test_accounts.py > • test_acl_*.py > • test_sharednetwork*.py > • test_add_remove_network.py > • test_advancedsg_networks.py > • test_affinity_groups*.py > • test_cpu_domain_limits.py > • test_cpu_limits.py > • test_cpu_max_limits.py > • test_host_maintenance.py > • test_memory_limits.py > • test_network_offering.py > • test_overcommit.py > • test_persistent_networks.py > • test_ps_domain_limits.py > • test_ps_limits.py > • test_ps_max_limits.py > • test_ps_resize_volume.py > • test_ps_resource_limits_volume.py > • test_resource_limits.py > • test_routers.py > • test_security_groups.py > • test_shared_networks.py > • test_snapshots.py > • test_ss_domain_limits.py > • test_ss_limits.py > • test_ss_max_limits.py > • test_templates.py > • test_update_vm.py > • test_volumes.py > • test_vpc.py > > I will start this set, and may adjust based on success rates and runtime. > > I hope to post results by COB, Monday (27 June 2016). > > Thanks, > -John > > > > > john.burw...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK > @shapeblue > > > > On Jun 20, 2016, at 4:31 PM, John Burwell <john.burw...@shapeblue.com> > wrote: > > > > All, > > > > I am working to coordinate some testing at ShapeBlue. I send an update > as soon as I have the list of environment and tests we plan to run. > > > > Thanks, > > -John > > > >> > > john.burw...@shapeblue.com > > www.shapeblue.com > > 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK > > @shapeblue > > > > > > > > On Jun 20, 2016, at 8:27 AM, Simon Weller <swel...@ena.com> wrote: > >> > >> > >> Remi, > >> > >> > >> The KVM VXLAN feature uses the standard BridgeVifDriver, as OVS doesn't > support multicast. > >> > >> > >> To reproduce this: > >> > >> > >> Deploy 4.9 RPMS to Centos 7.2 installation, upgrading a 4.8 install. > >> > >> Restart existing VPC router pair for a established VPC. > >> > >> When VPC routers come back up, the guest tier (isolated) network is > missing. > >> > >> > >> If you deploy a new VPC, the same problem occurs. > >> > >> > >> We have confirmed that the same problem exists on a non-redundant VPC > router as well. > >> > >> > >> David is working on reproducing this in a bubble. We have reproduced > this on 2 hardware labs thus far. > >> > >> > >> - Si > >> > >> > >> From: Remi Bergsma <rberg...@schubergphilis.com> > >> Sent: Saturday, June 18, 2016 2:58 AM > >> To: dev@cloudstack.apache.org > >> Subject: Re: 4.9/m
Re: 4.9/master Testing Coordination
All, I had hoped to get testing off the ground earlier, but $dayjob duties have gotten in the way. Over the weekend, I am planning to kick off tests of the following configurations: • CentOS 6.8 management server + CentOS 6.8 KVM Hosts using NFS primary and secondary storage (would allow us to verify/fix the documented libvirt/qemu versions) • CentOS 6.8 management server + vCenter 5.5u3d + ESXi 5.5u3b using NFS primary and secondary storage • CentOS 6.8 management server + vCenter 6.0u2 + ESXi Express Patch 6 using NFS primary and secondary storage • CentOS 6.8 management server + XenServer 6.2 SP1 using NFS primary and secondary storage • CentOS 6.8 management server + XenServer 6.5 SP1 using NFS primary and secondary storage In each of these environments, I plan to run the following tests: • All smoke tests • Component Tests • test_accounts.py • test_acl_*.py • test_sharednetwork*.py • test_add_remove_network.py • test_advancedsg_networks.py • test_affinity_groups*.py • test_cpu_domain_limits.py • test_cpu_limits.py • test_cpu_max_limits.py • test_host_maintenance.py • test_memory_limits.py • test_network_offering.py • test_overcommit.py • test_persistent_networks.py • test_ps_domain_limits.py • test_ps_limits.py • test_ps_max_limits.py • test_ps_resize_volume.py • test_ps_resource_limits_volume.py • test_resource_limits.py • test_routers.py • test_security_groups.py • test_shared_networks.py • test_snapshots.py • test_ss_domain_limits.py • test_ss_limits.py • test_ss_max_limits.py • test_templates.py • test_update_vm.py • test_volumes.py • test_vpc.py I will start this set, and may adjust based on success rates and runtime. I hope to post results by COB, Monday (27 June 2016). Thanks, -John > john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue On Jun 20, 2016, at 4:31 PM, John Burwell <john.burw...@shapeblue.com> wrote: > > All, > > I am working to coordinate some testing at ShapeBlue. I send an update as > soon as I have the list of environment and tests we plan to run. > > Thanks, > -John > >> > john.burw...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK > @shapeblue > > > > On Jun 20, 2016, at 8:27 AM, Simon Weller <swel...@ena.com> wrote: >> >> >> Remi, >> >> >> The KVM VXLAN feature uses the standard BridgeVifDriver, as OVS doesn't >> support multicast. >> >> >> To reproduce this: >> >> >> Deploy 4.9 RPMS to Centos 7.2 installation, upgrading a 4.8 install. >> >> Restart existing VPC router pair for a established VPC. >> >> When VPC routers come back up, the guest tier (isolated) network is missing. >> >> >> If you deploy a new VPC, the same problem occurs. >> >> >> We have confirmed that the same problem exists on a non-redundant VPC router >> as well. >> >> >> David is working on reproducing this in a bubble. We have reproduced this on >> 2 hardware labs thus far. >> >> >> - Si >> >> >> From: Remi Bergsma <rberg...@schubergphilis.com> >> Sent: Saturday, June 18, 2016 2:58 AM >> To: dev@cloudstack.apache.org >> Subject: Re: 4.9/master Testing Coordination >> >> Hi Simon, >> >> Do you have the exact stept to reproduce? >> To me it sounds like the issue is in either the ovsVifDriver or the VXLAN >> stuff. Can you reproduce the scenario in the bubble (with its default >> vlan/bridgeVifDriver)? >> >> If there is a clear scenario, I think we should write an integration test >> (even if that shows it’s broken). >> >> Regards, Remi >> >> >> On 17/06/16 22:47, "Simon Weller" <swel...@ena.com> wrote: >> >>> Here's a quick run down on the configuration(s) we're testing: >>> >>> >>> Centos 7.2 >>> >>> Advanced Zone with VXLAN on KVM >>> >>> VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al >>> >>> Redundant VPC VRs >>> >>
Re: 4.9/master Testing Coordination
All, I am working to coordinate some testing at ShapeBlue. I send an update as soon as I have the list of environment and tests we plan to run. Thanks, -John > john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue On Jun 20, 2016, at 8:27 AM, Simon Weller <swel...@ena.com> wrote: > > > Remi, > > > The KVM VXLAN feature uses the standard BridgeVifDriver, as OVS doesn't > support multicast. > > > To reproduce this: > > > Deploy 4.9 RPMS to Centos 7.2 installation, upgrading a 4.8 install. > > Restart existing VPC router pair for a established VPC. > > When VPC routers come back up, the guest tier (isolated) network is missing. > > > If you deploy a new VPC, the same problem occurs. > > > We have confirmed that the same problem exists on a non-redundant VPC router > as well. > > > David is working on reproducing this in a bubble. We have reproduced this on > 2 hardware labs thus far. > > > - Si > > > From: Remi Bergsma <rberg...@schubergphilis.com> > Sent: Saturday, June 18, 2016 2:58 AM > To: dev@cloudstack.apache.org > Subject: Re: 4.9/master Testing Coordination > > Hi Simon, > > Do you have the exact stept to reproduce? > To me it sounds like the issue is in either the ovsVifDriver or the VXLAN > stuff. Can you reproduce the scenario in the bubble (with its default > vlan/bridgeVifDriver)? > > If there is a clear scenario, I think we should write an integration test > (even if that shows it’s broken). > > Regards, Remi > > > On 17/06/16 22:47, "Simon Weller" <swel...@ena.com> wrote: > >> Here's a quick run down on the configuration(s) we're testing: >> >> >> Centos 7.2 >> >> Advanced Zone with VXLAN on KVM >> >> VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al >> >> Redundant VPC VRs >> >> Ceph Primary Storage >> >> NFS and S3 secondary storage >> >> As Will mentioned, we've found an odd issue with VPCs that we're still >> debugging. >> >> Here's a summary of what we've found thus far: >> When a tier is added, the network interface for the tier network is never >> plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is never >> plugged when you attempt to provision the first VM within the VPC and the VM >> creation fails. If you have existing VMs and you restart the router, you >> lose the eth2 interface in the libvirt configuration (confirmed with a virsh >> dumpxml). >> If you leave the VRs alone after the upgrade, VMs can be provisioned >> correctly (until you reboot the VRs). >> >> I did also run into the NIO SSL agent not connecting problem again. When I >> telnetted to 8250, the agent immediately came up without me having to >> restart it. So keep an eye out for that as well. >> >> >> - Si >> >> >> From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will >> Stevens <wstev...@cloudops.com> >> Sent: Friday, June 17, 2016 3:02 PM >> To: dev@cloudstack.apache.org >> Subject: Re: 4.9/master Testing Coordination >> >> Syed is looking into a potential issue with Swift as secondary storage on >> master. >> >> ENA is looking into a potential problem when upgrading a VR which was >> working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network >> there are only 2 nics instead of 3. If they spin a new VR from scratch it >> seems to work. I need to follow up with them to see if they have an >> updated status of their testing. >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >> w cloudops.com *|* tw @CloudOps_ >> >> On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: >> >>> The following tests are running cleanly on KVM with Advanced networking >>> and 2 hosts. >>> >>> echo "Running tests with required_hardware=true" >>> nosetests --with-marvin --marvin-config=${marvinCfg} -s -a >>> tags=advanced,required_hardware=true \ >>> smoke/test_password_server.py \ >>> smoke/test_vpc_redundant.py \ >>> smoke/test_routers_iptables_default_policy.py \ >>> smoke/test_routers_network_ops.py \ >>> smoke/test_vpc_router_nics.py \ >>> smoke/test_router_dhcphosts.py \ >>&g
Re: 4.9/master Testing Coordination
Remi, The KVM VXLAN feature uses the standard BridgeVifDriver, as OVS doesn't support multicast. To reproduce this: Deploy 4.9 RPMS to Centos 7.2 installation, upgrading a 4.8 install. Restart existing VPC router pair for a established VPC. When VPC routers come back up, the guest tier (isolated) network is missing. If you deploy a new VPC, the same problem occurs. We have confirmed that the same problem exists on a non-redundant VPC router as well. David is working on reproducing this in a bubble. We have reproduced this on 2 hardware labs thus far. - Si From: Remi Bergsma <rberg...@schubergphilis.com> Sent: Saturday, June 18, 2016 2:58 AM To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Hi Simon, Do you have the exact stept to reproduce? To me it sounds like the issue is in either the ovsVifDriver or the VXLAN stuff. Can you reproduce the scenario in the bubble (with its default vlan/bridgeVifDriver)? If there is a clear scenario, I think we should write an integration test (even if that shows it’s broken). Regards, Remi On 17/06/16 22:47, "Simon Weller" <swel...@ena.com> wrote: >Here's a quick run down on the configuration(s) we're testing: > > >Centos 7.2 > >Advanced Zone with VXLAN on KVM > >VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al > >Redundant VPC VRs > >Ceph Primary Storage > >NFS and S3 secondary storage > >As Will mentioned, we've found an odd issue with VPCs that we're still >debugging. > >Here's a summary of what we've found thus far: >When a tier is added, the network interface for the tier network is never >plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is never >plugged when you attempt to provision the first VM within the VPC and the VM >creation fails. If you have existing VMs and you restart the router, you lose >the eth2 interface in the libvirt configuration (confirmed with a virsh >dumpxml). >If you leave the VRs alone after the upgrade, VMs can be provisioned correctly >(until you reboot the VRs). > >I did also run into the NIO SSL agent not connecting problem again. When I >telnetted to 8250, the agent immediately came up without me having to restart >it. So keep an eye out for that as well. > > >- Si > > >From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will >Stevens <wstev...@cloudops.com> >Sent: Friday, June 17, 2016 3:02 PM >To: dev@cloudstack.apache.org >Subject: Re: 4.9/master Testing Coordination > >Syed is looking into a potential issue with Swift as secondary storage on >master. > >ENA is looking into a potential problem when upgrading a VR which was >working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network >there are only 2 nics instead of 3. If they spin a new VR from scratch it >seems to work. I need to follow up with them to see if they have an >updated status of their testing. > >*Will STEVENS* >Lead Developer > >*CloudOps* *| *Cloud Solutions Experts >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >w cloudops.com *|* tw @CloudOps_ > >On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: > >> The following tests are running cleanly on KVM with Advanced networking >> and 2 hosts. >> >> echo "Running tests with required_hardware=true" >> nosetests --with-marvin --marvin-config=${marvinCfg} -s -a >> tags=advanced,required_hardware=true \ >> smoke/test_password_server.py \ >> smoke/test_vpc_redundant.py \ >> smoke/test_routers_iptables_default_policy.py \ >> smoke/test_routers_network_ops.py \ >> smoke/test_vpc_router_nics.py \ >> smoke/test_router_dhcphosts.py \ >> smoke/test_loadbalance.py \ >> smoke/test_internal_lb.py \ >> smoke/test_ssvm.py \ >> smoke/test_vpc_vpn.py \ >> smoke/test_privategw_acl.py \ >> smoke/test_network.py >> >> echo "Running tests with required_hardware=false" >> nosetests --with-marvin --marvin-config=${marvinCfg} -s -a >> tags=advanced,required_hardware=false \ >> smoke/test_routers.py \ >> smoke/test_network_acl.py \ >> smoke/test_reset_vm_on_reboot.py \ >> smoke/test_vm_life_cycle.py \ >> smoke/test_service_offerings.py \ >> smoke/test_network.py \ >> component/test_vpc_offerings.py \ >> component/test_vpc_routers.py >> >> I need to do some more manual testing... >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >> w cloudops.com *|* tw @Clo
Re: 4.9/master Testing Coordination
Would it make sense to use the wiki for tracking this? Ron On 17/06/2016 2:56 PM, John Burwell wrote: All, It is a bit lo-fi, but if you are testing master in preparation for the 4.9 RC, could you please share information about the configurations you testing (e.g. hypervisors, storage backends, network configurations, etc)? Any test results could also be helpful. The hope is to reduce duplication of effort and understand how much of the system has been covered. Thanks, -John john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue -- Ron Wheeler President Artifact Software Inc email: rwhee...@artifact-software.com skype: ronaldmwheeler phone: 866-970-2435, ext 102
Re: 4.9/master Testing Coordination
Hi Simon, Do you have the exact stept to reproduce? To me it sounds like the issue is in either the ovsVifDriver or the VXLAN stuff. Can you reproduce the scenario in the bubble (with its default vlan/bridgeVifDriver)? If there is a clear scenario, I think we should write an integration test (even if that shows it’s broken). Regards, Remi On 17/06/16 22:47, "Simon Weller" <swel...@ena.com> wrote: >Here's a quick run down on the configuration(s) we're testing: > > >Centos 7.2 > >Advanced Zone with VXLAN on KVM > >VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al > >Redundant VPC VRs > >Ceph Primary Storage > >NFS and S3 secondary storage > >As Will mentioned, we've found an odd issue with VPCs that we're still >debugging. > >Here's a summary of what we've found thus far: >When a tier is added, the network interface for the tier network is never >plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is never >plugged when you attempt to provision the first VM within the VPC and the VM >creation fails. If you have existing VMs and you restart the router, you lose >the eth2 interface in the libvirt configuration (confirmed with a virsh >dumpxml). >If you leave the VRs alone after the upgrade, VMs can be provisioned correctly >(until you reboot the VRs). > >I did also run into the NIO SSL agent not connecting problem again. When I >telnetted to 8250, the agent immediately came up without me having to restart >it. So keep an eye out for that as well. > > >- Si > > >From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will >Stevens <wstev...@cloudops.com> >Sent: Friday, June 17, 2016 3:02 PM >To: dev@cloudstack.apache.org >Subject: Re: 4.9/master Testing Coordination > >Syed is looking into a potential issue with Swift as secondary storage on >master. > >ENA is looking into a potential problem when upgrading a VR which was >working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network >there are only 2 nics instead of 3. If they spin a new VR from scratch it >seems to work. I need to follow up with them to see if they have an >updated status of their testing. > >*Will STEVENS* >Lead Developer > >*CloudOps* *| *Cloud Solutions Experts >420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >w cloudops.com *|* tw @CloudOps_ > >On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: > >> The following tests are running cleanly on KVM with Advanced networking >> and 2 hosts. >> >> echo "Running tests with required_hardware=true" >> nosetests --with-marvin --marvin-config=${marvinCfg} -s -a >> tags=advanced,required_hardware=true \ >> smoke/test_password_server.py \ >> smoke/test_vpc_redundant.py \ >> smoke/test_routers_iptables_default_policy.py \ >> smoke/test_routers_network_ops.py \ >> smoke/test_vpc_router_nics.py \ >> smoke/test_router_dhcphosts.py \ >> smoke/test_loadbalance.py \ >> smoke/test_internal_lb.py \ >> smoke/test_ssvm.py \ >> smoke/test_vpc_vpn.py \ >> smoke/test_privategw_acl.py \ >> smoke/test_network.py >> >> echo "Running tests with required_hardware=false" >> nosetests --with-marvin --marvin-config=${marvinCfg} -s -a >> tags=advanced,required_hardware=false \ >> smoke/test_routers.py \ >> smoke/test_network_acl.py \ >> smoke/test_reset_vm_on_reboot.py \ >> smoke/test_vm_life_cycle.py \ >> smoke/test_service_offerings.py \ >> smoke/test_network.py \ >> component/test_vpc_offerings.py \ >> component/test_vpc_routers.py >> >> I need to do some more manual testing... >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >> w cloudops.com *|* tw @CloudOps_ >> >> On Fri, Jun 17, 2016 at 3:45 PM, Tutkowski, Mike < >> mike.tutkow...@netapp.com> wrote: >> >>> My testing has been performed using XenServer 6.5 and ESXi 5.5. >>> >>> I executed all of the tests in test/integration/plugins/solidfire. >>> >>> They all came back successful. >>> >>> From: John Burwell <john.burw...@shapeblue.com> >>> Sent: Friday, June 17, 2016 12:56 PM >>> To: dev@cloudstack.apache.org >>> Subject: 4.9/master Testing Coordination >>> >>> All, >>> >>> It is a bit lo-fi, but if you are testing master in preparation for the >>> 4.9 RC, could you p
Re: 4.9/master Testing Coordination
Here's a quick run down on the configuration(s) we're testing: Centos 7.2 Advanced Zone with VXLAN on KVM VPC functionality including Private GW, VPN, Static Routes, ACL Lists et al Redundant VPC VRs Ceph Primary Storage NFS and S3 secondary storage As Will mentioned, we've found an odd issue with VPCs that we're still debugging. Here's a summary of what we've found thus far: When a tier is added, the network interface for the tier network is never plugged by libvirt. You only get 2 interfaces (eth0 and eth1). eth2 is never plugged when you attempt to provision the first VM within the VPC and the VM creation fails. If you have existing VMs and you restart the router, you lose the eth2 interface in the libvirt configuration (confirmed with a virsh dumpxml). If you leave the VRs alone after the upgrade, VMs can be provisioned correctly (until you reboot the VRs). I did also run into the NIO SSL agent not connecting problem again. When I telnetted to 8250, the agent immediately came up without me having to restart it. So keep an eye out for that as well. - Si From: williamstev...@gmail.com <williamstev...@gmail.com> on behalf of Will Stevens <wstev...@cloudops.com> Sent: Friday, June 17, 2016 3:02 PM To: dev@cloudstack.apache.org Subject: Re: 4.9/master Testing Coordination Syed is looking into a potential issue with Swift as secondary storage on master. ENA is looking into a potential problem when upgrading a VR which was working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network there are only 2 nics instead of 3. If they spin a new VR from scratch it seems to work. I need to follow up with them to see if they have an updated status of their testing. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: > The following tests are running cleanly on KVM with Advanced networking > and 2 hosts. > > echo "Running tests with required_hardware=true" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=true \ > smoke/test_password_server.py \ > smoke/test_vpc_redundant.py \ > smoke/test_routers_iptables_default_policy.py \ > smoke/test_routers_network_ops.py \ > smoke/test_vpc_router_nics.py \ > smoke/test_router_dhcphosts.py \ > smoke/test_loadbalance.py \ > smoke/test_internal_lb.py \ > smoke/test_ssvm.py \ > smoke/test_vpc_vpn.py \ > smoke/test_privategw_acl.py \ > smoke/test_network.py > > echo "Running tests with required_hardware=false" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=false \ > smoke/test_routers.py \ > smoke/test_network_acl.py \ > smoke/test_reset_vm_on_reboot.py \ > smoke/test_vm_life_cycle.py \ > smoke/test_service_offerings.py \ > smoke/test_network.py \ > component/test_vpc_offerings.py \ > component/test_vpc_routers.py > > I need to do some more manual testing... > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > w cloudops.com *|* tw @CloudOps_ > > On Fri, Jun 17, 2016 at 3:45 PM, Tutkowski, Mike < > mike.tutkow...@netapp.com> wrote: > >> My testing has been performed using XenServer 6.5 and ESXi 5.5. >> >> I executed all of the tests in test/integration/plugins/solidfire. >> >> They all came back successful. >> >> From: John Burwell <john.burw...@shapeblue.com> >> Sent: Friday, June 17, 2016 12:56 PM >> To: dev@cloudstack.apache.org >> Subject: 4.9/master Testing Coordination >> >> All, >> >> It is a bit lo-fi, but if you are testing master in preparation for the >> 4.9 RC, could you please share information about the configurations you >> testing (e.g. hypervisors, storage backends, network configurations, etc)? >> Any test results could also be helpful. The hope is to reduce duplication >> of effort and understand how much of the system has been covered. >> >> Thanks, >> -John >> john.burw...@shapeblue.com >> www.shapeblue.com<http://www.shapeblue.com> ShapeBlue - The CloudStack Company<http://www.shapeblue.com/> www.shapeblue.com Overview Apache CloudStack contains an authentication module providing "single sign-on" functionality via the SAML data format. Under certain conditions, a >> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK >> @shapeblue >> >> >> >
Re: 4.9/master Testing Coordination
Syed is looking into a potential issue with Swift as secondary storage on master. ENA is looking into a potential problem when upgrading a VR which was working in 4.8.0, but after an upgrade to 4.9.0 and restarting the network there are only 2 nics instead of 3. If they spin a new VR from scratch it seems to work. I need to follow up with them to see if they have an updated status of their testing. *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Fri, Jun 17, 2016 at 4:00 PM, Will Stevens <wstev...@cloudops.com> wrote: > The following tests are running cleanly on KVM with Advanced networking > and 2 hosts. > > echo "Running tests with required_hardware=true" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=true \ > smoke/test_password_server.py \ > smoke/test_vpc_redundant.py \ > smoke/test_routers_iptables_default_policy.py \ > smoke/test_routers_network_ops.py \ > smoke/test_vpc_router_nics.py \ > smoke/test_router_dhcphosts.py \ > smoke/test_loadbalance.py \ > smoke/test_internal_lb.py \ > smoke/test_ssvm.py \ > smoke/test_vpc_vpn.py \ > smoke/test_privategw_acl.py \ > smoke/test_network.py > > echo "Running tests with required_hardware=false" > nosetests --with-marvin --marvin-config=${marvinCfg} -s -a > tags=advanced,required_hardware=false \ > smoke/test_routers.py \ > smoke/test_network_acl.py \ > smoke/test_reset_vm_on_reboot.py \ > smoke/test_vm_life_cycle.py \ > smoke/test_service_offerings.py \ > smoke/test_network.py \ > component/test_vpc_offerings.py \ > component/test_vpc_routers.py > > I need to do some more manual testing... > > *Will STEVENS* > Lead Developer > > *CloudOps* *| *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 > w cloudops.com *|* tw @CloudOps_ > > On Fri, Jun 17, 2016 at 3:45 PM, Tutkowski, Mike < > mike.tutkow...@netapp.com> wrote: > >> My testing has been performed using XenServer 6.5 and ESXi 5.5. >> >> I executed all of the tests in test/integration/plugins/solidfire. >> >> They all came back successful. >> ____ >> From: John Burwell <john.burw...@shapeblue.com> >> Sent: Friday, June 17, 2016 12:56 PM >> To: dev@cloudstack.apache.org >> Subject: 4.9/master Testing Coordination >> >> All, >> >> It is a bit lo-fi, but if you are testing master in preparation for the >> 4.9 RC, could you please share information about the configurations you >> testing (e.g. hypervisors, storage backends, network configurations, etc)? >> Any test results could also be helpful. The hope is to reduce duplication >> of effort and understand how much of the system has been covered. >> >> Thanks, >> -John >> john.burw...@shapeblue.com >> www.shapeblue.com >> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK >> @shapeblue >> >> >> >
Re: 4.9/master Testing Coordination
The following tests are running cleanly on KVM with Advanced networking and 2 hosts. echo "Running tests with required_hardware=true" nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \ smoke/test_password_server.py \ smoke/test_vpc_redundant.py \ smoke/test_routers_iptables_default_policy.py \ smoke/test_routers_network_ops.py \ smoke/test_vpc_router_nics.py \ smoke/test_router_dhcphosts.py \ smoke/test_loadbalance.py \ smoke/test_internal_lb.py \ smoke/test_ssvm.py \ smoke/test_vpc_vpn.py \ smoke/test_privategw_acl.py \ smoke/test_network.py echo "Running tests with required_hardware=false" nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \ smoke/test_routers.py \ smoke/test_network_acl.py \ smoke/test_reset_vm_on_reboot.py \ smoke/test_vm_life_cycle.py \ smoke/test_service_offerings.py \ smoke/test_network.py \ component/test_vpc_offerings.py \ component/test_vpc_routers.py I need to do some more manual testing... *Will STEVENS* Lead Developer *CloudOps* *| *Cloud Solutions Experts 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw @CloudOps_ On Fri, Jun 17, 2016 at 3:45 PM, Tutkowski, Mike <mike.tutkow...@netapp.com> wrote: > My testing has been performed using XenServer 6.5 and ESXi 5.5. > > I executed all of the tests in test/integration/plugins/solidfire. > > They all came back successful. > > From: John Burwell <john.burw...@shapeblue.com> > Sent: Friday, June 17, 2016 12:56 PM > To: dev@cloudstack.apache.org > Subject: 4.9/master Testing Coordination > > All, > > It is a bit lo-fi, but if you are testing master in preparation for the > 4.9 RC, could you please share information about the configurations you > testing (e.g. hypervisors, storage backends, network configurations, etc)? > Any test results could also be helpful. The hope is to reduce duplication > of effort and understand how much of the system has been covered. > > Thanks, > -John > john.burw...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK > @shapeblue > > >
Re: 4.9/master Testing Coordination
My testing has been performed using XenServer 6.5 and ESXi 5.5. I executed all of the tests in test/integration/plugins/solidfire. They all came back successful. From: John Burwell <john.burw...@shapeblue.com> Sent: Friday, June 17, 2016 12:56 PM To: dev@cloudstack.apache.org Subject: 4.9/master Testing Coordination All, It is a bit lo-fi, but if you are testing master in preparation for the 4.9 RC, could you please share information about the configurations you testing (e.g. hypervisors, storage backends, network configurations, etc)? Any test results could also be helpful. The hope is to reduce duplication of effort and understand how much of the system has been covered. Thanks, -John john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue
4.9/master Testing Coordination
All, It is a bit lo-fi, but if you are testing master in preparation for the 4.9 RC, could you please share information about the configurations you testing (e.g. hypervisors, storage backends, network configurations, etc)? Any test results could also be helpful. The hope is to reduce duplication of effort and understand how much of the system has been covered. Thanks, -John john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue