Our problem is in the test coverage of the existing code base not in
new features! As \Paul said dev will code tests they can think of.
It's the ones they didn't think of that will break stuff. But we don't
mind if new features aren't fully functional. We mind regression the
most. You can not blame a new feature for regression if the dev
doesn't know he is breaking something. So: Coverage, coverage,
coverage. We have a big backlog of test coverage to be written. I am
not saying that the radical proposal cannot wait until 4.5. We
shouldn't wait till 4.5. We need steps now because the rc's staying in
long and the release coming late is a trend that will cross a line
next release. We can't stop it next release. It needs to be stopped
now. And IMHO it is by shortening iteration not by postponing other
measures. Oh, and +1 for radical proposal.

end of rant,
Daan

On Fri, Mar 14, 2014 at 9:33 AM, Paul Angus <paul.an...@shapeblue.com> wrote:
> I think it's a very difficult subject, because we're relying on the 
> developers to write tests which will show whether the code in their feature 
> is good, and think of every possible failure mode in order to ratify their 
> code.
>
> Obviously if they can think of the failure, they'll probably have coded for 
> it, it's the ones they didn't think of that are the problem.
>
> Without an automated suite of hypervisors and mgmt. vms to building the code 
> and testing the feature 'in the real world' I can't see how we could 
> otherwise (automated) test in a meaningful way.
>
> I believe that currently that's what the likes of myself, Geoff, the guys at 
> Schuberg and the others who test outside of devcloud add to this process - 
> testing the feature against real hypervisors with real storage etc.
>
> I would love it if these tests were already automated and part of the 
> development process and continuously repeat to check for regression.
>
> But until they are I believe that the RC windows (72 hrs) is too short, and 
> that all -1s should have a Jira ticket that can then be tracked as blockers 
> and that the next RC shouldn't start until those bugs are cleared, so 
> everyone can see where we stand during the RC process.
>
>
>
> Regards,
>
> Paul Angus
> Cloud Architect
> S: +44 20 3603 0540 | M: +447711418784 | T: @CloudyAngus
> paul.an...@shapeblue.com
>
> -----Original Message-----
> From: David Nalley [mailto:da...@gnsa.us]
> Sent: 13 March 2014 16:42
> To: dev@cloudstack.apache.org
> Subject: Release cadence
>
> The RC7 vote thread contained a lot of discussion around release cadence, and 
> I figured I'd move that to a thread that has a better subject so there is 
> better visibility to list participants who don't read every thread.
>
> When I look at things schedule wise, I see our aims and our reality.
> We have a relatively short development window (in the schedule) and we have 
> almost 50% of our time in the schedule allocated to testing.
> (over two months). However, it seems that a lot of testing - or at least a 
> lot of testing for  what became blockers to the release didn't appear to 
> happen until RCs were kicked out - and that's where our schedule has fallen 
> apart for multiple releases. The automated tests we have were clean when we 
> issued RCs, so we clearly don't have the depth needed from an automated 
> standpoint.
>
> Two problems, one cultural and one technical. The technical problem is that 
> our automated test suite isn't deep enough to give us a high level of 
> confidence that we should release. The cultural problem is that many of us 
> wait until the release period of the schedule to test.
>
> What does that have to do with release cadence? Well inherently not much; but 
> let me describe my concerns. As a project; the schedule is meaningless if we 
> don't follow it; and effectively the release date is held hostage. 
> Personally, I do want as few bugs as possible, but it's a balancing act where 
> people doubt our ability if we aren't able to ship. I don't think it matters 
> if we move to 6 month cycles, if this behavior continues, we'd miss the 6 
> month date as well and push to 8 or 9 months. See my radical proposition at 
> the bottom for an idea on dealing with this.
>
> I also find myself agreeing with Daan on the additional complexity.
> Increasing the window for release inherently increases the window for feature 
> development. As soon as we branch a release, master is open for feature 
> development again. This means a potential for greater change at each release. 
> Change is a risk to quality; or at least an unknown that we again have to 
> test. The greater that quantity of change, the greater the potential threat 
> to quality.
>
> Radical proposition:
>
> Because we have two problems, of different nature, we are in a difficult 
> situation. This is a possible solution, and I'd appreciate you reading and 
> considering it.  Feedback is welcome. I propose that after we enter the RC 
> stage that we not entertain any bugs as blockers that don't have automated 
> test cases associated with them. This means that you are still welcome to do 
> manual testing of your pet feature and the things that are important to you; 
> during the testing window (or anytime really). However, if the automation 
> suite isn't also failing then we consider the release as high enough quality 
> to ship.
> This isn't something we can codify, but the PMC can certainly adopt this 
> attitude as a group when voting. Which also means that we can deviate from 
> it. If you brought up a blocker for release - we should be immediately 
> looking at how we can write a test for that behavior.
> This should also mean several other behaviors need to become a valid part of 
> our process. We need to ensure that things are well tested before allowing a 
> merge. This means we need a known state of master, and we need to perform 
> testing that allows us to confirm that a patch does no harm. We also need to 
> insist on implementation of comprehensive tests for every inbound feature.
>
> Thoughts, comments, flames, death threats? :)
>
> --David
> Need Enterprise Grade Support for Apache CloudStack?
> Our CloudStack Infrastructure 
> Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the 
> best 24/7 SLA for CloudStack Environments.
>
> Apache CloudStack Bootcamp training courses
>
> **NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
> 18th-19th February 2014, Brazil. 
> Classroom<http://shapeblue.com/cloudstack-training/>
> 17th-23rd March 2014, Region A. Instructor led, 
> On-line<http://shapeblue.com/cloudstack-training/>
> 24th-28th March 2014, Region B. Instructor led, 
> On-line<http://shapeblue.com/cloudstack-training/>
> 16th-20th June 2014, Region A. Instructor led, 
> On-line<http://shapeblue.com/cloudstack-training/>
> 23rd-27th June 2014, Region B. Instructor led, 
> On-line<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended 
> solely for the use of the individual to whom it is addressed. Any views or 
> opinions expressed are solely those of the author and do not necessarily 
> represent those of Shape Blue Ltd or related companies. If you are not the 
> intended recipient of this email, you must neither take any action based upon 
> its contents, nor copy or show it to anyone. Please contact the sender if you 
> believe you have received this email in error. Shape Blue Ltd is a company 
> incorporated in England & Wales. ShapeBlue Services India LLP is a company 
> incorporated in India and is operated under license from Shape Blue Ltd. 
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is 
> operated under license from Shape Blue Ltd. ShapeBlue is a registered 
> trademark.



-- 
Daan

Reply via email to