Great to see you kick this off Alex! I have a few ideas for this and had been looking for help as well ..
I had a draft lying around of an email you sent me long ago about tiered testing and I think your proposal fits very well on this: The tl;dr of that conversation was as below - improve simulator for runtime testability - customize to model and inject failures - make a habit of writing tests around bug reports (I started tagging tests since api_refactoring on JIRA already. look for the integration-test label) - make integration testing easier using factories and DSLs (from Chiradeep) (part 1 of this work was started on this in the marvin-refactor branch) More comments inline On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote: > Hi All, > > As most of you are aware, the master branch keeps getting broken by > checkins for various reasons. Committers need to be more > responsible about their checkins but I don't think we can depend on > that happening. There are various reasons. The most obvious to me > is that granting committership is not based on code competency. > (And I don't think it should.) Given that we need to build a BVT > system for ensure that checkins do not break the branch. > > Here's my proposal: > > Existing components that we'll use. > > - Citrix has contributed its testing to Apache. > > - Apache CloudStack has already a simulator that's been used for > scale testing. > > - Marvin > > - DevCloud-kvm > > Work Proposal: > > - Convert the Citrix testing into three phases: > > o Setup > > o Test > > o Verify I do the build, package, setup, test, verify in my integration test pipeline and a similar pipeline for a developer combined into easily runnable maven profiles/lifecycles/goals would be great to have. > - Add a Setup and Verify phase for the simulator > - Add all of the agent commands necessary for the simulator to pass > the testing. > - Add a Setup and Verify phase for devCloud-kvm The setups exists as configs in the marvin sandbox already. Just need to hook it up with mvn. For verify there is a simple python script testSetupSuccess.py already that checks two things - system VMs are up - built-in templates are downloaded This should be a good start IMO. Currently devcloud-kvm is a bit hard to run from a developer environment. But it's great to have in a continuous environment backed by a KVM host with a Linux 3.0 kernel. Marcus has written up some pretty good documentation for this. If someone can help bring up that setup I can assist in bringing up the tests using my devcloud-ci scripts. I'm bringing up devcloud after Kelven 'alerted' us of the memory fix. > > - Add two more profiles to pom > o Checkin-test-with-simulator: Runs the testing against the simulator > o Checkin-test-with-devCloud: Runs the testing against devcloud > > - All of the profiles will attempt to also check the merge list that > Chip has proposed.' > - We will also change marvin to easily add zones with > actual hardware. It will be based on a data driven document to do > the setup. This is currently partly doable using a properties file in the sandbox $ python advanced_env.py -i xen.properties -o xen.cfg gives out a advanced zone config for the properties specified. Is data driven similar or you are talking about a DSL that Edison mentioned at CCC12? > > For a developer to checkin: > > - S/he must writes marvin tests for their feature and add > it to the BVT. I made some progress on this in my marvin-refactor branch and will collect my thoughts in an FS that I am drafting. The marvin tests are difficult to write in their current form. I'm following Chiradeep's recommendation for providing factories and once those are baked a DSL language devs can quickly mock down tests against a simulator using the above mentioned mvn profiles. > > - S/he must run the checkin tests to verify everything works. > - If the feature contains a hardware/vpx component, simulation must > be added. > > At this point, everything is about the developer writing in their feature > branch and merging in. > > On infrastructure side: > > - We'll setup continuous BVT based on the simulator. I brought this up on the IRC and the list y'day, so +1 - happy to help > > - I again push that we must use Gerrit to test the code > before it gets merge into the branch but I'll leave that for someone > else to do that. > > Let me know what you guys think. I'll probably break out a bvt > branch to work on this. Anyone want to join me? Count me in! -- Prasanna.,
