First +1 on BVT. Second, should we consider the idea of having a staging area for people to check-in? Which is that making master always the stable(reasonable) branch for main development, but whenever people make check-ins, it goes into staging first, and we have maintainers(could be automatic) to run whatever test framework we have and only perform automatic merge to master from staging area after a successful test-run?
Kelven On 3/6/13 4:21 AM, "Prasanna Santhanam" <t...@apache.org> wrote: >Great to see you kick this off Alex! I have a few ideas for this and >had been looking for help as well .. > >I had a draft lying around of an email you sent me long ago about >tiered testing and I think your proposal fits very well on this: > >The tl;dr of that conversation was as below > >- improve simulator for runtime testability >- customize to model and inject failures >- make a habit of writing tests around bug reports >(I started tagging tests since api_refactoring on JIRA already. >look for the integration-test label) >- make integration testing easier using factories and DSLs (from > Chiradeep) >(part 1 of this work was started on this in the marvin-refactor >branch) > >More comments inline > >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote: >> Hi All, >> >> As most of you are aware, the master branch keeps getting broken by >> checkins for various reasons. Committers need to be more >> responsible about their checkins but I don't think we can depend on >> that happening. There are various reasons. The most obvious to me >> is that granting committership is not based on code competency. >> (And I don't think it should.) Given that we need to build a BVT >> system for ensure that checkins do not break the branch. >> >> Here's my proposal: >> >> Existing components that we'll use. >> >> - Citrix has contributed its testing to Apache. >> >> - Apache CloudStack has already a simulator that's been used >>for scale testing. >> >> - Marvin >> >> - DevCloud-kvm >> >> Work Proposal: >> >> - Convert the Citrix testing into three phases: >> >> o Setup >> >> o Test >> >> o Verify > >I do the build, package, setup, test, verify in my integration test >pipeline and a similar pipeline for a developer combined into easily >runnable maven profiles/lifecycles/goals would be great to have. > >> - Add a Setup and Verify phase for the simulator >> - Add all of the agent commands necessary for the simulator to >>pass the testing. >> - Add a Setup and Verify phase for devCloud-kvm > >The setups exists as configs in the marvin sandbox already. Just need >to hook it up with mvn. For verify there is a simple python script >testSetupSuccess.py already that checks two things > >- system VMs are up >- built-in templates are downloaded > >This should be a good start IMO. > >Currently devcloud-kvm is a bit hard to run from a developer >environment. But it's great to have in a continuous environment backed >by a KVM host with a Linux 3.0 kernel. Marcus has written up some >pretty good documentation for this. If someone can help bring up that >setup I can assist in bringing up the tests using my devcloud-ci >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the >memory fix. > >> >> - Add two more profiles to pom >> o Checkin-test-with-simulator: Runs the testing against the simulator >> o Checkin-test-with-devCloud: Runs the testing against devcloud >> >> - All of the profiles will attempt to also check the merge >>list that Chip has proposed.' > >> - We will also change marvin to easily add zones with >> actual hardware. It will be based on a data driven document to do >> the setup. >This is currently partly doable using a properties file in the sandbox >$ python advanced_env.py -i xen.properties -o xen.cfg > >gives out a advanced zone config for the properties specified. >Is data driven similar or you are talking about a DSL that Edison >mentioned at CCC12? > >> >> For a developer to checkin: >> >> - S/he must writes marvin tests for their feature and add >> it to the BVT. >I made some progress on this in my marvin-refactor branch and will >collect my thoughts in an FS that I am drafting. The marvin tests are >difficult to write in their current form. I'm following Chiradeep's >recommendation for providing factories and once those are baked a DSL >language devs can quickly mock down tests against a simulator using >the above mentioned mvn profiles. > >> >> - S/he must run the checkin tests to verify everything works. >> - If the feature contains a hardware/vpx component, simulation >>must be added. >> >> At this point, everything is about the developer writing in their >>feature branch and merging in. >> >> On infrastructure side: >> >> - We'll setup continuous BVT based on the simulator. >I brought this up on the IRC and the list y'day, so +1 - happy to help >> >> - I again push that we must use Gerrit to test the code >> before it gets merge into the branch but I'll leave that for someone >> else to do that. >> >> Let me know what you guys think. I'll probably break out a bvt >> branch to work on this. Anyone want to join me? > >Count me in! > >-- >Prasanna.,
