Re: 4.8, 4.9, and master Testing Status

[Syed Ahmed]

@Paul

I'll test this out on my XS7 setup. I need a little help in getting started
with the details of how you are running them.

-Syed

On Wed, Oct 19, 2016 at 9:03 AM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> To give a specific example,
>
> Running XenServer7 smoke tests result in a large number of failures due to
> problems with the tests.  If we're going to make any real forward progress,
> we need to be able to get meaningful test results.
>
> Plae review/test these PRs
>
>
>
> pretty please (with sugar on top).
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com]
> Sent: 19 October 2016 12:39
> To: dev@cloudstack.apache.org
> Subject: Re: 4.8, 4.9, and master Testing Status
>
> All,
>
>
> At least one more test/review LGTM is required on following PRs, please
> help with your review/tests as they will help us work towards cutting the
> RCs:
>
>
> https://github.com/apache/cloudstack/pull/1692
>
> https://github.com/apache/cloudstack/pull/1703
>
> https://github.com/apache/cloudstack/pull/1708
>
>
> As John has shared in the previous email, we still have outstanding
> failures and we'll be working towards fixing all three of them. If we can
> get them merged before end of the week, we can trigger tests on them again
> to know test status on each of the 4.8, 4.9 and master branches that can
> help us determine the quality on each of the branches and we can work
> towards RCs.
>
>
> Regards.
>
> ____________
> From: John Burwell <john.burw...@shapeblue.com>
> Sent: 14 October 2016 12:11:04
> To: dev@cloudstack.apache.org
> Subject: Re: 4.8, 4.9, and master Testing Status
>
> All,
>
> We have made great strides stabilizing the 4.8 [1] and 4.9 [2] smoke
> tests.  While we are not super green, the following remaining
> failures/issues are isolated to the VPC VR and secondary storage.
>
> * CLOUDSTACK-9541: redundant VPC VR: issues when master and backup
> switch happens on failover [3]
> * CLOUDSTACK-9540: createPrivateGateway create private network
> does not create proper VLAN network on XenServer
> * CLOUDSTACK-9528: SSVM Downloads (built-in) template multiple
> times
>
> Therefore, I would like to merge these two PRs so that we can begin the
> process of rebasing and retesting the PRs slotted for 4.8 and 4.9 that are
> not affected by these issues (i.e. PRs unrelated to secondary storage or
> the VR).  Our hope is that we can correct these issues quickly, and by the
> time we have worked through the backlog of pending PRs, these issues will
> be addressed and we can move those impacted forward.
>
> Unfortunately, the master PR [5] has 6 failures and 4 errors on XenServer
> [6] that we are currently analyzing.  We hope to have these resolved
> shortly in order to begin progressing PRs targeting master.
>
> I would like to get 1692 [1] and 1703 [2] merged in the next 24 hours.  We
> need to complete the following actions in order to accomplish this goal:
>
> * Obtain at least one code review LGTM on PR #1692 [1]
> * Obtain at least one code review LGTM on PR #1703 [2]
> * Obtain at least one test review LGTM on PR #1703 [2]
>
> Once these PRs, I will be updating PRs slotted for 4.8 and 4.9 to ping
> authors for a rebase.  Following each rebase, we will trigger blueorangutan
> to retest each one.
>
> Thank again for your patience and assistance, -John
>
> [1]: https://github.com/apache/cloudstack/pull/1692
> [2]: https://github.com/apache/cloudstack/pull/1703
> [3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9541
> [4]: https://issues.apache.org/jira/browse/CLOUDSTACK-9540
> [5]: https://github.com/apache/cloudstack/pull/1708
> [6]: https://github.com/apache/cloudstack/pull/1708#issuecomment-253698099
>
> > On Oct 7, 2016, at 10:12 AM, Will Stevens <wstev...@cloudops.com> wrote:
> >
> > Great work everyone.  Don't worry about the sporadic updates, that is
> > just the nature of the beast when working through stuff like this.
> > Well done so far...
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw
> > @CloudOps_
> >
> > On Fri, Oct 7, 2016 at 9:53 AM, John Burwell
> > <john.burw...@shapeblue.com>
> > wrote:
> >
> >> All,
> >>
> >> Than

RE: 4.8, 4.9, and master Testing Status

[Paul Angus]

To give a specific example,

Running XenServer7 smoke tests result in a large number of failures due to 
problems with the tests.  If we're going to make any real forward progress, we 
need to be able to get meaningful test results.

Plae review/test these PRs



pretty please (with sugar on top).


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rohit Yadav [mailto:rohit.ya...@shapeblue.com] 
Sent: 19 October 2016 12:39
To: dev@cloudstack.apache.org
Subject: Re: 4.8, 4.9, and master Testing Status

All,


At least one more test/review LGTM is required on following PRs, please help 
with your review/tests as they will help us work towards cutting the RCs:


https://github.com/apache/cloudstack/pull/1692

https://github.com/apache/cloudstack/pull/1703

https://github.com/apache/cloudstack/pull/1708


As John has shared in the previous email, we still have outstanding failures 
and we'll be working towards fixing all three of them. If we can get them 
merged before end of the week, we can trigger tests on them again to know test 
status on each of the 4.8, 4.9 and master branches that can help us determine 
the quality on each of the branches and we can work towards RCs.


Regards.


From: John Burwell <john.burw...@shapeblue.com>
Sent: 14 October 2016 12:11:04
To: dev@cloudstack.apache.org
Subject: Re: 4.8, 4.9, and master Testing Status

All,

We have made great strides stabilizing the 4.8 [1] and 4.9 [2] smoke tests.  
While we are not super green, the following remaining failures/issues are 
isolated to the VPC VR and secondary storage.

* CLOUDSTACK-9541: redundant VPC VR: issues when master and backup 
switch happens on failover [3]
* CLOUDSTACK-9540: createPrivateGateway create private network does not 
create proper VLAN network on XenServer
* CLOUDSTACK-9528: SSVM Downloads (built-in) template multiple times

Therefore, I would like to merge these two PRs so that we can begin the process 
of rebasing and retesting the PRs slotted for 4.8 and 4.9 that are not affected 
by these issues (i.e. PRs unrelated to secondary storage or the VR).  Our hope 
is that we can correct these issues quickly, and by the time we have worked 
through the backlog of pending PRs, these issues will be addressed and we can 
move those impacted forward.

Unfortunately, the master PR [5] has 6 failures and 4 errors on XenServer [6] 
that we are currently analyzing.  We hope to have these resolved shortly in 
order to begin progressing PRs targeting master.

I would like to get 1692 [1] and 1703 [2] merged in the next 24 hours.  We need 
to complete the following actions in order to accomplish this goal:

* Obtain at least one code review LGTM on PR #1692 [1]
* Obtain at least one code review LGTM on PR #1703 [2]
* Obtain at least one test review LGTM on PR #1703 [2]

Once these PRs, I will be updating PRs slotted for 4.8 and 4.9 to ping authors 
for a rebase.  Following each rebase, we will trigger blueorangutan to retest 
each one.

Thank again for your patience and assistance, -John

[1]: https://github.com/apache/cloudstack/pull/1692
[2]: https://github.com/apache/cloudstack/pull/1703
[3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9541
[4]: https://issues.apache.org/jira/browse/CLOUDSTACK-9540
[5]: https://github.com/apache/cloudstack/pull/1708
[6]: https://github.com/apache/cloudstack/pull/1708#issuecomment-253698099

> On Oct 7, 2016, at 10:12 AM, Will Stevens <wstev...@cloudops.com> wrote:
>
> Great work everyone.  Don't worry about the sporadic updates, that is 
> just the nature of the beast when working through stuff like this.  
> Well done so far...
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw 
> @CloudOps_
>
> On Fri, Oct 7, 2016 at 9:53 AM, John Burwell 
> <john.burw...@shapeblue.com>
> wrote:
>
>> All,
>>
>> Thank you Ilya and Haijao for your words of encouragement.  In 
>> addition to the efforts of Paul, Rohit, Murali, Abhi, and Bobby, 
>> Sergey Levitskiy has been providing great help testing VMware.
>>
>> I apologize for my sporadic status updates.  We have made significant 
>> progress in getting smoke tests to pass on KVM, XenServer, and VMware.
>> Currently, we have the following number of failures and errors:
>>
>>* KVM: 0
>>* VMware: 4
>>* XenServer: 8
>>
>> The outstanding failures and errors seem to be the caused by the 
>> following
>> issues:
>>
>>1. On VMware and XenServer, guest VMs in VPCs start but don’t 
>> acquire IP addresses causing tests relyin

Re: 4.8, 4.9, and master Testing Status

[Rohit Yadav]

All,


At least one more test/review LGTM is required on following PRs, please help 
with your review/tests as they will help us work towards cutting the RCs:


https://github.com/apache/cloudstack/pull/1692

https://github.com/apache/cloudstack/pull/1703

https://github.com/apache/cloudstack/pull/1708


As John has shared in the previous email, we still have outstanding failures 
and we'll be working towards fixing all three of them. If we can get them 
merged before end of the week, we can trigger tests on them again to know test 
status on each of the 4.8, 4.9 and master branches that can help us determine 
the quality on each of the branches and we can work towards RCs.


Regards.


From: John Burwell <john.burw...@shapeblue.com>
Sent: 14 October 2016 12:11:04
To: dev@cloudstack.apache.org
Subject: Re: 4.8, 4.9, and master Testing Status

All,

We have made great strides stabilizing the 4.8 [1] and 4.9 [2] smoke tests.  
While we are not super green, the following remaining failures/issues are 
isolated to the VPC VR and secondary storage.

* CLOUDSTACK-9541: redundant VPC VR: issues when master and backup 
switch happens on failover [3]
* CLOUDSTACK-9540: createPrivateGateway create private network does not 
create proper VLAN network on XenServer
* CLOUDSTACK-9528: SSVM Downloads (built-in) template multiple times

Therefore, I would like to merge these two PRs so that we can begin the process 
of rebasing and retesting the PRs slotted for 4.8 and 4.9 that are not affected 
by these issues (i.e. PRs unrelated to secondary storage or the VR).  Our hope 
is that we can correct these issues quickly, and by the time we have worked 
through the backlog of pending PRs, these issues will be addressed and we can 
move those impacted forward.

Unfortunately, the master PR [5] has 6 failures and 4 errors on XenServer [6] 
that we are currently analyzing.  We hope to have these resolved shortly in 
order to begin progressing PRs targeting master.

I would like to get 1692 [1] and 1703 [2] merged in the next 24 hours.  We need 
to complete the following actions in order to accomplish this goal:

* Obtain at least one code review LGTM on PR #1692 [1]
* Obtain at least one code review LGTM on PR #1703 [2]
* Obtain at least one test review LGTM on PR #1703 [2]

Once these PRs, I will be updating PRs slotted for 4.8 and 4.9 to ping authors 
for a rebase.  Following each rebase, we will trigger blueorangutan to retest 
each one.

Thank again for your patience and assistance,
-John

[1]: https://github.com/apache/cloudstack/pull/1692
[2]: https://github.com/apache/cloudstack/pull/1703
[3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9541
[4]: https://issues.apache.org/jira/browse/CLOUDSTACK-9540
[5]: https://github.com/apache/cloudstack/pull/1708
[6]: https://github.com/apache/cloudstack/pull/1708#issuecomment-253698099

> On Oct 7, 2016, at 10:12 AM, Will Stevens <wstev...@cloudops.com> wrote:
>
> Great work everyone.  Don't worry about the sporadic updates, that is just
> the nature of the beast when working through stuff like this.  Well done so
> far...
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Fri, Oct 7, 2016 at 9:53 AM, John Burwell <john.burw...@shapeblue.com>
> wrote:
>
>> All,
>>
>> Thank you Ilya and Haijao for your words of encouragement.  In addition to
>> the efforts of Paul, Rohit, Murali, Abhi, and Bobby, Sergey Levitskiy has
>> been providing great help testing VMware.
>>
>> I apologize for my sporadic status updates.  We have made significant
>> progress in getting smoke tests to pass on KVM, XenServer, and VMware.
>> Currently, we have the following number of failures and errors:
>>
>>* KVM: 0
>>* VMware: 4
>>* XenServer: 8
>>
>> The outstanding failures and errors seem to be the caused by the following
>> issues:
>>
>>1. On VMware and XenServer, guest VMs in VPCs start but don’t
>> acquire IP addresses causing tests relying on SSH connectivity tests to
>> fail.  The issue occurs does not occur on KVM, intermittently on VMware,
>> and consistently on XenServer.  This issue affects the test_vpc_redundant,
>> test_privategw_acl, and test_vpc_vpn test suites.   We believe that this
>> issue may be caused by either the guest VMs startup/DHCP wait period
>> winning the race with the VPC VR configuration or there is a problem on the
>> VPC VR assigning IP addresses.  We are currently investigating and expect
>> to identify the root cause shortly.
>>2. SSVM downloads str being restarted due to ping timeouts on
>&g

Re: 4.8, 4.9, and master Testing Status

[John Burwell]

All,

We have made great strides stabilizing the 4.8 [1] and 4.9 [2] smoke tests.  
While we are not super green, the following remaining failures/issues are 
isolated to the VPC VR and secondary storage.  

* CLOUDSTACK-9541: redundant VPC VR: issues when master and backup 
switch happens on failover [3]
* CLOUDSTACK-9540: createPrivateGateway create private network does not 
create proper VLAN network on XenServer
* CLOUDSTACK-9528: SSVM Downloads (built-in) template multiple times

Therefore, I would like to merge these two PRs so that we can begin the process 
of rebasing and retesting the PRs slotted for 4.8 and 4.9 that are not affected 
by these issues (i.e. PRs unrelated to secondary storage or the VR).  Our hope 
is that we can correct these issues quickly, and by the time we have worked 
through the backlog of pending PRs, these issues will be addressed and we can 
move those impacted forward.

Unfortunately, the master PR [5] has 6 failures and 4 errors on XenServer [6] 
that we are currently analyzing.  We hope to have these resolved shortly in 
order to begin progressing PRs targeting master.

I would like to get 1692 [1] and 1703 [2] merged in the next 24 hours.  We need 
to complete the following actions in order to accomplish this goal:

* Obtain at least one code review LGTM on PR #1692 [1]
* Obtain at least one code review LGTM on PR #1703 [2]
* Obtain at least one test review LGTM on PR #1703 [2]

Once these PRs, I will be updating PRs slotted for 4.8 and 4.9 to ping authors 
for a rebase.  Following each rebase, we will trigger blueorangutan to retest 
each one.

Thank again for your patience and assistance,
-John

[1]: https://github.com/apache/cloudstack/pull/1692
[2]: https://github.com/apache/cloudstack/pull/1703
[3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9541
[4]: https://issues.apache.org/jira/browse/CLOUDSTACK-9540
[5]: https://github.com/apache/cloudstack/pull/1708
[6]: https://github.com/apache/cloudstack/pull/1708#issuecomment-253698099

> On Oct 7, 2016, at 10:12 AM, Will Stevens  wrote:
> 
> Great work everyone.  Don't worry about the sporadic updates, that is just
> the nature of the beast when working through stuff like this.  Well done so
> far...
> 
> *Will STEVENS*
> Lead Developer
> 
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
> 
> On Fri, Oct 7, 2016 at 9:53 AM, John Burwell 
> wrote:
> 
>> All,
>> 
>> Thank you Ilya and Haijao for your words of encouragement.  In addition to
>> the efforts of Paul, Rohit, Murali, Abhi, and Bobby, Sergey Levitskiy has
>> been providing great help testing VMware.
>> 
>> I apologize for my sporadic status updates.  We have made significant
>> progress in getting smoke tests to pass on KVM, XenServer, and VMware.
>> Currently, we have the following number of failures and errors:
>> 
>>* KVM: 0
>>* VMware: 4
>>* XenServer: 8
>> 
>> The outstanding failures and errors seem to be the caused by the following
>> issues:
>> 
>>1. On VMware and XenServer, guest VMs in VPCs start but don’t
>> acquire IP addresses causing tests relying on SSH connectivity tests to
>> fail.  The issue occurs does not occur on KVM, intermittently on VMware,
>> and consistently on XenServer.  This issue affects the test_vpc_redundant,
>> test_privategw_acl, and test_vpc_vpn test suites.   We believe that this
>> issue may be caused by either the guest VMs startup/DHCP wait period
>> winning the race with the VPC VR configuration or there is a problem on the
>> VPC VR assigning IP addresses.  We are currently investigating and expect
>> to identify the root cause shortly.
>>2. SSVM downloads str being restarted due to ping timeouts on
>> XenServer and VMware.  We are seeing the following messages such as the
>> following in the Management Server logs:
>> 
>>com.cloud.utils.exception.CloudRuntimeException: Failed
>> to send command, due to 
>> Agent:5,com.cloud.exception.OperationTimedoutException:
>> Commands
>>9042102151853113352 to Host 5 timed out after 2400
>> 
>>  Our initial investigation discovered different timezones being
>> used by the system VM templates and Management Server.  This discrepancy We
>> have modified Trillian to ensure consistent configuration of time zones
>> across a cluster, and are preparing another run for XenServer and VMware.
>> KVM is not affected by this time zone issue because KVM hosts use the same
>> CentOS template as CentOS based Management Servers -- creating time zone
>> consistency by side effect.
>> 
>> Reports of each test run are available on PR #1692 [1].  We have kicked a
>> new round of tests on KVM, VMware, and XenServer with the time zone fix and
>> additional instrumentation to run down the VPC VR race condition.
>> 
>> Instead of directly forward 

Re: 4.8, 4.9, and master Testing Status

[Will Stevens]

Great work everyone.  Don't worry about the sporadic updates, that is just
the nature of the beast when working through stuff like this.  Well done so
far...

*Will STEVENS*
Lead Developer

*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_

On Fri, Oct 7, 2016 at 9:53 AM, John Burwell 
wrote:

> All,
>
> Thank you Ilya and Haijao for your words of encouragement.  In addition to
> the efforts of Paul, Rohit, Murali, Abhi, and Bobby, Sergey Levitskiy has
> been providing great help testing VMware.
>
> I apologize for my sporadic status updates.  We have made significant
> progress in getting smoke tests to pass on KVM, XenServer, and VMware.
> Currently, we have the following number of failures and errors:
>
> * KVM: 0
> * VMware: 4
> * XenServer: 8
>
> The outstanding failures and errors seem to be the caused by the following
> issues:
>
> 1. On VMware and XenServer, guest VMs in VPCs start but don’t
> acquire IP addresses causing tests relying on SSH connectivity tests to
> fail.  The issue occurs does not occur on KVM, intermittently on VMware,
> and consistently on XenServer.  This issue affects the test_vpc_redundant,
> test_privategw_acl, and test_vpc_vpn test suites.   We believe that this
> issue may be caused by either the guest VMs startup/DHCP wait period
> winning the race with the VPC VR configuration or there is a problem on the
> VPC VR assigning IP addresses.  We are currently investigating and expect
> to identify the root cause shortly.
> 2. SSVM downloads str being restarted due to ping timeouts on
> XenServer and VMware.  We are seeing the following messages such as the
> following in the Management Server logs:
>
> com.cloud.utils.exception.CloudRuntimeException: Failed
> to send command, due to 
> Agent:5,com.cloud.exception.OperationTimedoutException:
> Commands
> 9042102151853113352 to Host 5 timed out after 2400
>
>   Our initial investigation discovered different timezones being
> used by the system VM templates and Management Server.  This discrepancy We
> have modified Trillian to ensure consistent configuration of time zones
> across a cluster, and are preparing another run for XenServer and VMware.
> KVM is not affected by this time zone issue because KVM hosts use the same
> CentOS template as CentOS based Management Servers -- creating time zone
> consistency by side effect.
>
> Reports of each test run are available on PR #1692 [1].  We have kicked a
> new round of tests on KVM, VMware, and XenServer with the time zone fix and
> additional instrumentation to run down the VPC VR race condition.
>
> Instead of directly forward merging these changes, we plan to open a PR
> for each forward merge.  Since we are very close to having 4.8 resolved,
> Rohit has open PR 1703 [2] for the 4.9 forward merge and kicked off a test
> run.  While we cannot close this PR until 1692 is complete, we are hoping
> to get a head start on any issues in the 4.9 branch.
>
> Thank you again for your patience,
> -John
>
> [1]: https://github.com/apache/cloudstack/pull/1692
> [2]: https://github.com/apache/cloudstack/pull/1703
>
> > On Oct 5, 2016, at 4:32 AM, Haijiao <18602198...@163.com> wrote:
> >
> > Though I am one of the silent majority, I would thank John the dev team
> for your continuous effort, you keep ACS alive and better !
> >
> >
> > Just heard one of biggest finance company in China running 10,000+ VMs
> on ACS 4.4 for production/dev/QAS,  you guys should be proud of that.
> > Salute to you!
> >
> >
> >
> >
> >
> >
> >
> > 在2016年10月05 03时09分, "ilya"写道:
> >
> > John and Team
> >
> > Thanks for amazing work and contributing back.
> >
> > Regards,
> > ilya
> >
> > On 10/3/16 9:48 PM, John Burwell wrote:
> >> All,
> >>
> >> A quick update on our progress to pass all smoke tests aka super
> green.  We have reduced the failures and errors for XenServer from 93 to 9
> and for VMware from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently
> executing.  Based on manual tests/fixes, we are expecting to be the first
> super green configuration.  We have also found the following additional
> defects:
> >>
> >>  * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple
> Times
> >>  * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly
> >>
> >> 9528 is causing XenServer environments to fail to install and startup
> cleanly.  A lack of cleanup described in 9529 is causing XenServer to
> exhaust available resources before a test run completes.  We believe that
> resolution of these issues will address most, if not all, of the XenServer
> issues.
> >>
> >> Thanks,
> >> -John
> >>
> >> [1]: https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=65873020
> >> [2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
> >> [3]: 

Re: 4.8, 4.9, and master Testing Status

[John Burwell]

All,

Thank you Ilya and Haijao for your words of encouragement.  In addition to the 
efforts of Paul, Rohit, Murali, Abhi, and Bobby, Sergey Levitskiy has been 
providing great help testing VMware.  

I apologize for my sporadic status updates.  We have made significant progress 
in getting smoke tests to pass on KVM, XenServer, and VMware.  Currently, we 
have the following number of failures and errors:

* KVM: 0
* VMware: 4
* XenServer: 8

The outstanding failures and errors seem to be the caused by the following 
issues:

1. On VMware and XenServer, guest VMs in VPCs start but don’t acquire 
IP addresses causing tests relying on SSH connectivity tests to fail.  The 
issue occurs does not occur on KVM, intermittently on VMware, and consistently 
on XenServer.  This issue affects the test_vpc_redundant, test_privategw_acl, 
and test_vpc_vpn test suites.   We believe that this issue may be caused by 
either the guest VMs startup/DHCP wait period winning the race with the VPC VR 
configuration or there is a problem on the VPC VR assigning IP addresses.  We 
are currently investigating and expect to identify the root cause shortly.
2. SSVM downloads str being restarted due to ping timeouts on XenServer 
and VMware.  We are seeing the following messages such as the following in the 
Management Server logs:

com.cloud.utils.exception.CloudRuntimeException: Failed to send 
command, due to Agent:5,com.cloud.exception.OperationTimedoutException: 
Commands 
9042102151853113352 to Host 5 timed out after 2400

  Our initial investigation discovered different timezones being used 
by the system VM templates and Management Server.  This discrepancy We have 
modified Trillian to ensure consistent configuration of time zones across a 
cluster, and are preparing another run for XenServer and VMware.  KVM is not 
affected by this time zone issue because KVM hosts use the same CentOS template 
as CentOS based Management Servers -- creating time zone consistency by side 
effect.

Reports of each test run are available on PR #1692 [1].  We have kicked a new 
round of tests on KVM, VMware, and XenServer with the time zone fix and 
additional instrumentation to run down the VPC VR race condition.

Instead of directly forward merging these changes, we plan to open a PR for 
each forward merge.  Since we are very close to having 4.8 resolved, Rohit has 
open PR 1703 [2] for the 4.9 forward merge and kicked off a test run.  While we 
cannot close this PR until 1692 is complete, we are hoping to get a head start 
on any issues in the 4.9 branch.

Thank you again for your patience,
-John

[1]: https://github.com/apache/cloudstack/pull/1692
[2]: https://github.com/apache/cloudstack/pull/1703

> On Oct 5, 2016, at 4:32 AM, Haijiao <18602198...@163.com> wrote:
> 
> Though I am one of the silent majority, I would thank John the dev team for 
> your continuous effort, you keep ACS alive and better !
> 
> 
> Just heard one of biggest finance company in China running 10,000+ VMs on ACS 
> 4.4 for production/dev/QAS,  you guys should be proud of that.
> Salute to you!
> 
> 
> 
> 
> 
> 
> 
> 在2016年10月05 03时09分, "ilya"写道:
> 
> John and Team
> 
> Thanks for amazing work and contributing back.
> 
> Regards,
> ilya
> 
> On 10/3/16 9:48 PM, John Burwell wrote:
>> All,
>> 
>> A quick update on our progress to pass all smoke tests aka super green.  We 
>> have reduced the failures and errors for XenServer from 93 to 9 and for 
>> VMware from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently executing.  
>> Based on manual tests/fixes, we are expecting to be the first super green 
>> configuration.  We have also found the following additional defects:
>> 
>>  * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple Times
>>  * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly
>> 
>> 9528 is causing XenServer environments to fail to install and startup 
>> cleanly.  A lack of cleanup described in 9529 is causing XenServer to 
>> exhaust available resources before a test run completes.  We believe that 
>> resolution of these issues will address most, if not all, of the XenServer 
>> issues.
>> 
>> Thanks,
>> -John
>> 
>> [1]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
>> [2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
>> [3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9529
>> 
>>> 
>> john.burw...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
>> @shapeblue
>> 
>> 
>> 
>> On Sep 30, 2016, at 2:40 AM, John Burwell  wrote:
>>> 
>>> All,
>>> 
>>> Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing 
>>> the smoke tests for the 4.8, 4.9, and master branches against the following 
>>> environments:
>>> 
>>>* CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 

Re:Re: 4.8, 4.9, and master Testing Status

[Haijiao]

Though I am one of the silent majority, I would thank John the dev team for 
your continuous effort, you keep ACS alive and better !


Just heard one of biggest finance company in China running 10,000+ VMs on ACS 
4.4 for production/dev/QAS,  you guys should be proud of that.
Salute to you!







在2016年10月05 03时09分, "ilya"写道:

John and Team

Thanks for amazing work and contributing back.

Regards,
ilya

On 10/3/16 9:48 PM, John Burwell wrote:
> All,
>
> A quick update on our progress to pass all smoke tests aka super green.  We 
> have reduced the failures and errors for XenServer from 93 to 9 and for 
> VMware from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently executing.  
> Based on manual tests/fixes, we are expecting to be the first super green 
> configuration.  We have also found the following additional defects:
>
>   * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple Times
>   * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly
>
> 9528 is causing XenServer environments to fail to install and startup 
> cleanly.  A lack of cleanup described in 9529 is causing XenServer to exhaust 
> available resources before a test run completes.  We believe that resolution 
> of these issues will address most, if not all, of the XenServer issues.
>
> Thanks,
> -John
>
> [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
> [2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
> [3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9529
>
>>
> john.burw...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
> @shapeblue
>  
>  
>
> On Sep 30, 2016, at 2:40 AM, John Burwell  wrote:
>>
>> All,
>>
>> Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing 
>> the smoke tests for the 4.8, 4.9, and master branches against the following 
>> environments:
>>
>> * CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 
>> Storage
>> * CentOS 7.2 Management Server + XenServer 6.5SP1 + NFS 
>> Primary/Secondary Storage
>> * CentOS 7.2 Management Server + CentOS 7.2 KVM + NFS Primary/Secondary 
>> Storage
>>
>> Thus far, we have found seven (7) test case and/or CloudStack defects in the 
>> VMware run for the 4.8 branch [1].  We are currently triaging fifty-one (51) 
>> new issues from the XenServer run to determine which issues were 
>> environmental and defects.  This triage work should be completed today (30 
>> Sept 2016).  Finally, we are awaiting the results of the KVM run.  
>>
>> We are using PR #1692 [2] as the master tracking PR to fix all defects in 
>> the 4.8 branch.  Our goal is to get all non-skip tests to pass and then 
>> merge this PR to the 4.8, 4.9, and master.  For each bug, we are creating a 
>> JIRA ticket and adding a commit to the PR.  Currently, the branch for this 
>> PR is in the shapeblue repo (the branch started with a much smaller fix from 
>> Paul and we just kept using it).  However, if others are interested in 
>> picking up defects, we will move it to ASF repo.  Once the 4.8 branch is 
>> stabilized, we plan to re-execute these tests on the 4.9 and master branches 
>> as we expect that the 4.9 and master branches will have additional issues.
>>
>> Since we are in a test freeze, I propose that no further PRs are merged to 
>> the 4.8, 4.9, and master branches until they are stabilized.  The following 
>> PRs will be re-based, re-tested, and merged to 4.8, 4.9.1.0, and/or 4.10.0.0 
>> post-stabilization:
>>
>> * 1696
>> * 1694
>> * 1684
>>  * 1681
>> * 1680
>> * 1678
>> * 1677
>> * 1676
>> * 1674
>> * 1673
>> * 1642
>> * 1624
>> * 1615
>> * 1600
>> * 1545
>> * 1542
>>
>> I recognize that this a large backlog of contributions ready to merge, and 
>> apologize for asking folks to wait.  However, given current state of the 
>> release branches, merging them before we complete fixing the smoke tests 
>> would create a moving target that further delay stabilization.  
>>
>> Obviously, it is unlikely we will make the 10 October 2016 release date for 
>> the 4.8.2.0, 4.9.1.0, and 4.10.0.0 releases.  At this point, it is difficult 
>> to estimate the size of the schedule slip because we still have issues to 
>> triage and test runs to complete.  I have created a wiki page [2] to track 
>> progress on this effort.  
>>
>> Does this approach sound reasonable?  Any suggestions to speed up this 
>> process will be greatly appreciated as stabilizing and re-opening these 
>> branches stable ASAP is critical for the community.
>>
>> Thanks,
>> -John
>>
>> [1]: 
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9518?jql=project%20%3D%20CLOUDSTACK%20AND%20fixVersion%20in%20(4.8.2.0)%20AND%20labels%20in%20(4.8.2.0-smoke-test-failure)
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
>>
>>> On Sep 26, 2016, at 

Re: 4.8, 4.9, and master Testing Status

[ilya]

John and Team

Thanks for amazing work and contributing back.

Regards,
ilya

On 10/3/16 9:48 PM, John Burwell wrote:
> All,
> 
> A quick update on our progress to pass all smoke tests aka super green.  We 
> have reduced the failures and errors for XenServer from 93 to 9 and for 
> VMware from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently executing.  
> Based on manual tests/fixes, we are expecting to be the first super green 
> configuration.  We have also found the following additional defects:
> 
>   * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple Times 
>   * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly
> 
> 9528 is causing XenServer environments to fail to install and startup 
> cleanly.  A lack of cleanup described in 9529 is causing XenServer to exhaust 
> available resources before a test run completes.  We believe that resolution 
> of these issues will address most, if not all, of the XenServer issues.
> 
> Thanks,
> -John
> 
> [1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
> [2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
> [3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9529
> 
>>
> john.burw...@shapeblue.com 
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
> @shapeblue
>   
>  
> 
> On Sep 30, 2016, at 2:40 AM, John Burwell  wrote:
>>
>> All,
>>
>> Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing 
>> the smoke tests for the 4.8, 4.9, and master branches against the following 
>> environments:
>>
>>  * CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 
>> Storage
>>  * CentOS 7.2 Management Server + XenServer 6.5SP1 + NFS 
>> Primary/Secondary Storage
>>  * CentOS 7.2 Management Server + CentOS 7.2 KVM + NFS Primary/Secondary 
>> Storage
>>
>> Thus far, we have found seven (7) test case and/or CloudStack defects in the 
>> VMware run for the 4.8 branch [1].  We are currently triaging fifty-one (51) 
>> new issues from the XenServer run to determine which issues were 
>> environmental and defects.  This triage work should be completed today (30 
>> Sept 2016).  Finally, we are awaiting the results of the KVM run.  
>>
>> We are using PR #1692 [2] as the master tracking PR to fix all defects in 
>> the 4.8 branch.  Our goal is to get all non-skip tests to pass and then 
>> merge this PR to the 4.8, 4.9, and master.  For each bug, we are creating a 
>> JIRA ticket and adding a commit to the PR.  Currently, the branch for this 
>> PR is in the shapeblue repo (the branch started with a much smaller fix from 
>> Paul and we just kept using it).  However, if others are interested in 
>> picking up defects, we will move it to ASF repo.  Once the 4.8 branch is 
>> stabilized, we plan to re-execute these tests on the 4.9 and master branches 
>> as we expect that the 4.9 and master branches will have additional issues.
>>
>> Since we are in a test freeze, I propose that no further PRs are merged to 
>> the 4.8, 4.9, and master branches until they are stabilized.  The following 
>> PRs will be re-based, re-tested, and merged to 4.8, 4.9.1.0, and/or 4.10.0.0 
>> post-stabilization:
>>
>>  * 1696
>>  * 1694
>>  * 1684
>>  * 1681
>>  * 1680
>>  * 1678
>>  * 1677
>>  * 1676
>>  * 1674
>>  * 1673
>>  * 1642
>>  * 1624
>>  * 1615
>>  * 1600
>>  * 1545
>>  * 1542
>>
>> I recognize that this a large backlog of contributions ready to merge, and 
>> apologize for asking folks to wait.  However, given current state of the 
>> release branches, merging them before we complete fixing the smoke tests 
>> would create a moving target that further delay stabilization.  
>>
>> Obviously, it is unlikely we will make the 10 October 2016 release date for 
>> the 4.8.2.0, 4.9.1.0, and 4.10.0.0 releases.  At this point, it is difficult 
>> to estimate the size of the schedule slip because we still have issues to 
>> triage and test runs to complete.  I have created a wiki page [2] to track 
>> progress on this effort.  
>>
>> Does this approach sound reasonable?  Any suggestions to speed up this 
>> process will be greatly appreciated as stabilizing and re-opening these 
>> branches stable ASAP is critical for the community.
>>
>> Thanks,
>> -John
>>
>> [1]: 
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9518?jql=project%20%3D%20CLOUDSTACK%20AND%20fixVersion%20in%20(4.8.2.0)%20AND%20labels%20in%20(4.8.2.0-smoke-test-failure)
>> [2]: 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
>>
>>> On Sep 26, 2016, at 8:38 AM, Will Stevens  wrote:
>>>
>>> Yes, I think it is important that you or Rajani sign off on anything that
>>> gets in while branches are frozen so you guys can stay on top of what goes
>>> in.
>>>
>>> Thanks for all the hard work team.  :)
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> *CloudOps* 

RE: 4.8, 4.9, and master Testing Status

[Marty Godsey]

Is XenServer 7 on the roadmap? I have seen significant performance differences 
between my ACS cluster running XS 6.5 SP1 and a separate XS 7 cluster (non-ACS) 
connecting to the same storage array. Other than the XS hosts being different 
versions all is the same. Same switches, same storage array, etc..

I am just curious.

Regards,
Marty Godsey
nSource Solutions

-Original Message-
From: John Burwell [mailto:john.burw...@shapeblue.com] 
Sent: Tuesday, October 4, 2016 12:48 AM
To: dev@cloudstack.apache.org
Cc: Murali Reddy <murali.re...@shapeblue.com>; Rohit Yadav 
<rohit.ya...@shapeblue.com>; Paul Angus <paul.an...@shapeblue.com>; Boris 
Stoyanov <boris.stoya...@shapeblue.com>; Abhinandan Prateek 
<abhinandan.prat...@shapeblue.com>
Subject: Re: 4.8, 4.9, and master Testing Status

All,

A quick update on our progress to pass all smoke tests aka super green.  We 
have reduced the failures and errors for XenServer from 93 to 9 and for VMware 
from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently executing.  Based on 
manual tests/fixes, we are expecting to be the first super green configuration. 
 We have also found the following additional defects:

  * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple Times
  * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly

9528 is causing XenServer environments to fail to install and startup cleanly.  
A lack of cleanup described in 9529 is causing XenServer to exhaust available 
resources before a test run completes.  We believe that resolution of these 
issues will address most, if not all, of the XenServer issues.

Thanks,
-John

[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
[2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
[3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9529

> 
john.burw...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue
  
 

On Sep 30, 2016, at 2:40 AM, John Burwell <john.burw...@shapeblue.com> wrote:
> 
> All,
> 
> Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing 
> the smoke tests for the 4.8, 4.9, and master branches against the following 
> environments:
> 
>   * CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 
> Storage
>   * CentOS 7.2 Management Server + XenServer 6.5SP1 + NFS 
> Primary/Secondary Storage
>   * CentOS 7.2 Management Server + CentOS 7.2 KVM + NFS 
> Primary/Secondary Storage
> 
> Thus far, we have found seven (7) test case and/or CloudStack defects in the 
> VMware run for the 4.8 branch [1].  We are currently triaging fifty-one (51) 
> new issues from the XenServer run to determine which issues were 
> environmental and defects.  This triage work should be completed today (30 
> Sept 2016).  Finally, we are awaiting the results of the KVM run.  
> 
> We are using PR #1692 [2] as the master tracking PR to fix all defects in the 
> 4.8 branch.  Our goal is to get all non-skip tests to pass and then merge 
> this PR to the 4.8, 4.9, and master.  For each bug, we are creating a JIRA 
> ticket and adding a commit to the PR.  Currently, the branch for this PR is 
> in the shapeblue repo (the branch started with a much smaller fix from Paul 
> and we just kept using it).  However, if others are interested in picking up 
> defects, we will move it to ASF repo.  Once the 4.8 branch is stabilized, we 
> plan to re-execute these tests on the 4.9 and master branches as we expect 
> that the 4.9 and master branches will have additional issues.
> 
> Since we are in a test freeze, I propose that no further PRs are merged to 
> the 4.8, 4.9, and master branches until they are stabilized.  The following 
> PRs will be re-based, re-tested, and merged to 4.8, 4.9.1.0, and/or 4.10.0.0 
> post-stabilization:
> 
>   * 1696
>   * 1694
>   * 1684
>   * 1681
>   * 1680
>   * 1678
>   * 1677
>   * 1676
>   * 1674
>   * 1673
>   * 1642
>   * 1624
>   * 1615
>   * 1600
>   * 1545
>   * 1542
> 
> I recognize that this a large backlog of contributions ready to merge, and 
> apologize for asking folks to wait.  However, given current state of the 
> release branches, merging them before we complete fixing the smoke tests 
> would create a moving target that further delay stabilization.  
> 
> Obviously, it is unlikely we will make the 10 October 2016 release date for 
> the 4.8.2.0, 4.9.1.0, and 4.10.0.0 releases.  At this point, it is difficult 
> to estimate the size of the schedule slip because we still have issues to 
> triage and test runs to complete.  I have created a wiki page [2] to track 
> progress on this effort.  
> 
> Does this approach sound reasonabl

Re: 4.8, 4.9, and master Testing Status

[John Burwell]

All,

A quick update on our progress to pass all smoke tests aka super green.  We 
have reduced the failures and errors for XenServer from 93 to 9 and for VMware 
from 51 to 14.  A CentOS 6/CentOS 6 KVM run is currently executing.  Based on 
manual tests/fixes, we are expecting to be the first super green configuration. 
 We have also found the following additional defects:

  * CLOUDSTACK-9528 [2]: SSVM Downloads (built-in) Template Multiple Times 
  * CLOUDSTACK-9529 [3]: Marvin Tests Do Not Clean Up Properly

9528 is causing XenServer environments to fail to install and startup cleanly.  
A lack of cleanup described in 9529 is causing XenServer to exhaust available 
resources before a test run completes.  We believe that resolution of these 
issues will address most, if not all, of the XenServer issues.

Thanks,
-John

[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
[2]: https://issues.apache.org/jira/browse/CLOUDSTACK-9528
[3]: https://issues.apache.org/jira/browse/CLOUDSTACK-9529

> 
john.burw...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
@shapeblue
  
 

On Sep 30, 2016, at 2:40 AM, John Burwell  wrote:
> 
> All,
> 
> Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing 
> the smoke tests for the 4.8, 4.9, and master branches against the following 
> environments:
> 
>   * CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 
> Storage
>   * CentOS 7.2 Management Server + XenServer 6.5SP1 + NFS 
> Primary/Secondary Storage
>   * CentOS 7.2 Management Server + CentOS 7.2 KVM + NFS Primary/Secondary 
> Storage
> 
> Thus far, we have found seven (7) test case and/or CloudStack defects in the 
> VMware run for the 4.8 branch [1].  We are currently triaging fifty-one (51) 
> new issues from the XenServer run to determine which issues were 
> environmental and defects.  This triage work should be completed today (30 
> Sept 2016).  Finally, we are awaiting the results of the KVM run.  
> 
> We are using PR #1692 [2] as the master tracking PR to fix all defects in the 
> 4.8 branch.  Our goal is to get all non-skip tests to pass and then merge 
> this PR to the 4.8, 4.9, and master.  For each bug, we are creating a JIRA 
> ticket and adding a commit to the PR.  Currently, the branch for this PR is 
> in the shapeblue repo (the branch started with a much smaller fix from Paul 
> and we just kept using it).  However, if others are interested in picking up 
> defects, we will move it to ASF repo.  Once the 4.8 branch is stabilized, we 
> plan to re-execute these tests on the 4.9 and master branches as we expect 
> that the 4.9 and master branches will have additional issues.
> 
> Since we are in a test freeze, I propose that no further PRs are merged to 
> the 4.8, 4.9, and master branches until they are stabilized.  The following 
> PRs will be re-based, re-tested, and merged to 4.8, 4.9.1.0, and/or 4.10.0.0 
> post-stabilization:
> 
>   * 1696
>   * 1694
>   * 1684
>   * 1681
>   * 1680
>   * 1678
>   * 1677
>   * 1676
>   * 1674
>   * 1673
>   * 1642
>   * 1624
>   * 1615
>   * 1600
>   * 1545
>   * 1542
> 
> I recognize that this a large backlog of contributions ready to merge, and 
> apologize for asking folks to wait.  However, given current state of the 
> release branches, merging them before we complete fixing the smoke tests 
> would create a moving target that further delay stabilization.  
> 
> Obviously, it is unlikely we will make the 10 October 2016 release date for 
> the 4.8.2.0, 4.9.1.0, and 4.10.0.0 releases.  At this point, it is difficult 
> to estimate the size of the schedule slip because we still have issues to 
> triage and test runs to complete.  I have created a wiki page [2] to track 
> progress on this effort.  
> 
> Does this approach sound reasonable?  Any suggestions to speed up this 
> process will be greatly appreciated as stabilizing and re-opening these 
> branches stable ASAP is critical for the community.
> 
> Thanks,
> -John
> 
> [1]: 
> https://issues.apache.org/jira/browse/CLOUDSTACK-9518?jql=project%20%3D%20CLOUDSTACK%20AND%20fixVersion%20in%20(4.8.2.0)%20AND%20labels%20in%20(4.8.2.0-smoke-test-failure)
> [2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020
> 
>> On Sep 26, 2016, at 8:38 AM, Will Stevens  wrote:
>> 
>> Yes, I think it is important that you or Rajani sign off on anything that
>> gets in while branches are frozen so you guys can stay on top of what goes
>> in.
>> 
>> Thanks for all the hard work team.  :)
>> 
>> *Will STEVENS*
>> Lead Developer
>> 
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>> 
>> On Mon, Sep 26, 2016 at 2:10 AM, John Burwell 
>> wrote:
>> 
>>> All,
>>> 
>>> Per 

4.8, 4.9, and master Testing Status

[John Burwell]

All,

Using blueorganutan, Rohit, Murali, Boris, Paul, Abhi, and I are executing the 
smoke tests for the 4.8, 4.9, and master branches against the following 
environments:

* CentOS 7.2 Management Server + VMware 5.5u3 + NFS Primary/Secondary 
Storage
* CentOS 7.2 Management Server + XenServer 6.5SP1 + NFS 
Primary/Secondary Storage
* CentOS 7.2 Management Server + CentOS 7.2 KVM + NFS Primary/Secondary 
Storage

Thus far, we have found seven (7) test case and/or CloudStack defects in the 
VMware run for the 4.8 branch [1].  We are currently triaging fifty-one (51) 
new issues from the XenServer run to determine which issues were environmental 
and defects.  This triage work should be completed today (30 Sept 2016).  
Finally, we are awaiting the results of the KVM run.  

We are using PR #1692 [2] as the master tracking PR to fix all defects in the 
4.8 branch.  Our goal is to get all non-skip tests to pass and then merge this 
PR to the 4.8, 4.9, and master.  For each bug, we are creating a JIRA ticket 
and adding a commit to the PR.  Currently, the branch for this PR is in the 
shapeblue repo (the branch started with a much smaller fix from Paul and we 
just kept using it).  However, if others are interested in picking up defects, 
we will move it to ASF repo.  Once the 4.8 branch is stabilized, we plan to 
re-execute these tests on the 4.9 and master branches as we expect that the 4.9 
and master branches will have additional issues.

Since we are in a test freeze, I propose that no further PRs are merged to the 
4.8, 4.9, and master branches until they are stabilized.  The following PRs 
will be re-based, re-tested, and merged to 4.8, 4.9.1.0, and/or 4.10.0.0 
post-stabilization:

* 1696
* 1694
* 1684
* 1681
* 1680
* 1678
* 1677
* 1676
* 1674
* 1673
* 1642
* 1624
* 1615
* 1600
* 1545
* 1542

I recognize that this a large backlog of contributions ready to merge, and 
apologize for asking folks to wait.  However, given current state of the 
release branches, merging them before we complete fixing the smoke tests would 
create a moving target that further delay stabilization.  

Obviously, it is unlikely we will make the 10 October 2016 release date for the 
4.8.2.0, 4.9.1.0, and 4.10.0.0 releases.  At this point, it is difficult to 
estimate the size of the schedule slip because we still have issues to triage 
and test runs to complete.  I have created a wiki page [2] to track progress on 
this effort.  

Does this approach sound reasonable?  Any suggestions to speed up this process 
will be greatly appreciated as stabilizing and re-opening these branches stable 
ASAP is critical for the community.

Thanks,
-John

[1]: 
https://issues.apache.org/jira/browse/CLOUDSTACK-9518?jql=project%20%3D%20CLOUDSTACK%20AND%20fixVersion%20in%20(4.8.2.0)%20AND%20labels%20in%20(4.8.2.0-smoke-test-failure)
[2]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65873020

> On Sep 26, 2016, at 8:38 AM, Will Stevens  wrote:
> 
> Yes, I think it is important that you or Rajani sign off on anything that
> gets in while branches are frozen so you guys can stay on top of what goes
> in.
> 
> Thanks for all the hard work team.  :)
> 
> *Will STEVENS*
> Lead Developer
> 
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
> 
> On Mon, Sep 26, 2016 at 2:10 AM, John Burwell 
> wrote:
> 
>> All,
>> 
>> Per our release schedule [1], the 4.8, 4.9, and master branches are frozen
>> for testing.  There are some straggling PRs that Rajani and I are working
>> to merge.  Is it acceptable to everyone that for the next two (2) weeks,
>> all PRs require not only 2 LGTMs, but approval by Rajani or I to be merged
>> to these branches?  To be clear, we don’t have to perform the merges,
>> simply give a thumbs up.
>> 
>> Thanks,
>> -John
>> john.burw...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
>> @shapeblue
>> 
>> 
>> 
>> 


john.burw...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London VA WC2N 4HSUK
@shapeblue