David I certainly vote +1 on your ideas to improve testing in CS and enhancing and embracing automation as way of life. To put in my two cents , I would like to highlight that unit test automation is the base for automation strategy. Here is a good post on test pyramid http://martinfowler.com/bliki/TestPyramid.html. As developers if we can commit to comprehensive automated unit tests we can release early and often. CloudStack with its API layer is already well positioned for service/integration layer automation.
Thanks Animesh -----Original Message----- From: David Nalley [mailto:da...@gnsa.us] Sent: Wednesday, November 21, 2012 2:48 PM To: cloudstack-dev@incubator.apache.org Subject: Re: [DISCUSS] QA/Testing focus on 4.1 Hi folks: Just a heads up on where I currently stand before the holidays in the US begin. I have master building and they run unit tests - and also cobertura is reporting coverage reports in that job. https://builds.apache.org/job/cloudstack-master-maven/ I also have the RAT job running here: https://builds.apache.org/job/cloudstack-rat-master/ Both of these poll git and run if there have been commits. --David On Sat, Nov 10, 2012 at 7:00 AM, David Nalley <da...@gnsa.us> wrote: > Hi folks - haven't followed up on this in several weeks - also haven't > had a chance to do much - I've been on holiday and traveling to ACEU. > > I did have the opportunity to talk to Gavin McDonald of ASF Infra fame > while at ACEU this past week and discuss with him some ideas for our > testing. He did recommend we actually use builders local to ASF infra > for things we want to happen fast (compile tests, unit tests, etc) as > they are effectively on the same network, and thus things like > checkouts happen much faster than connections over the WAN. > > I am going to start having my weekly IRC time at 1800 UTC in > #cloudstack-dev on Wednesdays - all are welcome to join, I know it is > inconvenient for many - and so don't hesitate to do things outside of > that time - I am going to start working on migrating some of the > simpler stuff to builds.a.o. > > --David > > > On Wed, Oct 17, 2012 at 6:22 PM, David Nalley <da...@gnsa.us> wrote: > > Warning: This is a long ponderous email. Stop now and get your cup > > of coffee/tea/$beverage. > > > > Hi folks, > > > > I've been having some off hand conversations with folks in other > > open source communities and rereading a number of good books of late > > (most notably Continuous Delivery by Jez Humble) and I would like to > > toss this idea out and see what folks think of it: > > > > CloudStack is large, often complex, piece of software. There are few > > people who truly grok all of CloudStack, and because of the > > diversity of environments it can be deployed in it can be truly > > bewildering to test fully. That said, I noticed a number of things > > that became last minute, blocker bugs, and it wasn't overly esoteric > > configurations, it was code paths that we just hadn't excercised in > > testing. One if these would have even been easily detected and fixed > > we had been testing installation alone. Additionally, as a project > > we are dramatically growing. The number of people adding non-trivial > > amounts of code is growing, and that makes ad hoc QA even more difficult. > > > > So my proposal in short is that we focus on testing, beginning > > immediately and erect what is effectively a continuous deployment > > pipeline - such that our confidence in our codebase is such that we > > could arbitrarily decide to release if we so desired. I don't think > > that this is a one release type of project, indeed I think it's > > really more of a culture shift than a technical project. The big > > shift is that EVERYONE must be responsible for a quality release. To > > that end, I'd propose the following tenets if we choose to adopt this: > > > > * Tests become the Andon cord[1] for the entire project. When a test > > fails we stop - additional commits don't happen - we find out what > > is wrong and fix it. More on this in a bit. > > > > * Tests (specifically automated tests) become part of our culture. > > * New features should come replete with both unit and integration > > tests. I am tempted to suggest a certain percentage of coverage, but > > I worry that it is a red herring; particularly given our dismal > > current unit test coverage. > > * Blocker and critical bugs must have automated tests that get > > committed as part of being qualified for closing. > > > > * We dedicate a non-trivial portion of our energy to enhancing not > > just the quantity and quality of our tests, but also on making it > > highly automated, and capable of delivering fast feedback. Ideally > > we would know within minutes if a commit broke unit tests, within > > hours if a commit failed in integration testing. > > > > I also know that this isn't a new idea. Lots of people have been > > focused on automated testing as part of our ongoing development. The > > only difference here is that I am actively asking you not to solely > > depend on those folks to do the work for you, but to make testing a > > part of the problem that you have to solve here. To be clear the > > goal isn't to be perfect and problem free with every commit - things > > will break. (If you've followed things at all in CloudStack you'll > > know that I've broken builds more than once) > > > > Pipeline I'd like to see for 4.1: > > > > 1. RAT test (fail this and we have IP problems) 2. Compile test > > (does it actually compile) 3. Unit tests 4. Package building 5. > > Automated installation (multiple platforms, does it actually install > > from the packages) 6. Integration tests (aka Marvin running against > > virtualized or real CS deployments) > > > > Clearly the above isn't an end/all be all for testing, but perhaps > > we can get some of this going in the 4.1 timeframe. There are also > > clearly corner cases (building marvin, building api docs, building > > official documentation) that need to be included in the pipeline as > > well. But the principle is that we won't move on past our failure > > until that failure is resolved. > > > > Immediate Action Items: > > > > Whether we adopt this or not, I plan on showing up on IRC once a > > week to work on testing in some form or another for an hour or two. > > I will also be cajoling people to join me. I might be working on > > infrastructure tasks. Obviously we have people scattered across the > > globe, so it's not the only time to work on testing, but you are > > welcome to join me. > > > > I am curious to hear others thoughts, comments, or flames? Is this > > something we should espouse as we are close to 4.0.0 releasing and > > turning our focus on 4.1? > > > > --David > > > > [1] http://en.wikipedia.org/wiki/Andon_(manufacturing) >
