+1 to the idea of staging area. Each commit should be put in the staging area and on successful build and BVT run, it would get pushed to master. The staging area should maintain a queue of all the commits and process them in a sequential manner. If it is possible to set up all these with the existing infra that would be great.
> -----Original Message----- > From: Kelven Yang [mailto:kelven.y...@citrix.com] > Sent: Wednesday, March 06, 2013 11:52 PM > To: cloudstack-dev@incubator.apache.org > Subject: Re: [PROPOSAL] BVT for CloudStack checkins > > First +1 on BVT. > > Second, should we consider the idea of having a staging area for people to > check-in? Which is that making master always the stable(reasonable) branch > for main development, but whenever people make check-ins, it goes into > staging first, and we have maintainers(could be automatic) to run whatever > test framework we have and only perform automatic merge to master from > staging area after a successful test-run? > > Kelven > > On 3/6/13 4:21 AM, "Prasanna Santhanam" <t...@apache.org> wrote: > > >Great to see you kick this off Alex! I have a few ideas for this and > >had been looking for help as well .. > > > >I had a draft lying around of an email you sent me long ago about > >tiered testing and I think your proposal fits very well on this: > > > >The tl;dr of that conversation was as below > > > >- improve simulator for runtime testability > >- customize to model and inject failures > >- make a habit of writing tests around bug reports (I started tagging > >tests since api_refactoring on JIRA already. > >look for the integration-test label) > >- make integration testing easier using factories and DSLs (from > > Chiradeep) > >(part 1 of this work was started on this in the marvin-refactor > >branch) > > > >More comments inline > > > >On Wed, Mar 06, 2013 at 12:12:11AM +0530, Alex Huang wrote: > >> Hi All, > >> > >> As most of you are aware, the master branch keeps getting broken by > >> checkins for various reasons. Committers need to be more responsible > >> about their checkins but I don't think we can depend on that > >> happening. There are various reasons. The most obvious to me is > >> that granting committership is not based on code competency. > >> (And I don't think it should.) Given that we need to build a BVT > >> system for ensure that checkins do not break the branch. > >> > >> Here's my proposal: > >> > >> Existing components that we'll use. > >> > >> - Citrix has contributed its testing to Apache. > >> > >> - Apache CloudStack has already a simulator that's been used > >>for scale testing. > >> > >> - Marvin > >> > >> - DevCloud-kvm > >> > >> Work Proposal: > >> > >> - Convert the Citrix testing into three phases: > >> > >> o Setup > >> > >> o Test > >> > >> o Verify > > > >I do the build, package, setup, test, verify in my integration test > >pipeline and a similar pipeline for a developer combined into easily > >runnable maven profiles/lifecycles/goals would be great to have. > > > >> - Add a Setup and Verify phase for the simulator > >> - Add all of the agent commands necessary for the simulator to > >>pass the testing. > >> - Add a Setup and Verify phase for devCloud-kvm > > > >The setups exists as configs in the marvin sandbox already. Just need > >to hook it up with mvn. For verify there is a simple python script > >testSetupSuccess.py already that checks two things > > > >- system VMs are up > >- built-in templates are downloaded > > > >This should be a good start IMO. > > > >Currently devcloud-kvm is a bit hard to run from a developer > >environment. But it's great to have in a continuous environment backed > >by a KVM host with a Linux 3.0 kernel. Marcus has written up some > >pretty good documentation for this. If someone can help bring up that > >setup I can assist in bringing up the tests using my devcloud-ci > >scripts. I'm bringing up devcloud after Kelven 'alerted' us of the > >memory fix. > > > >> > >> - Add two more profiles to pom > >> o Checkin-test-with-simulator: Runs the testing against the simulator > >> o Checkin-test-with-devCloud: Runs the testing against devcloud > >> > >> - All of the profiles will attempt to also check the merge > >>list that Chip has proposed.' > > > >> - We will also change marvin to easily add zones with > >> actual hardware. It will be based on a data driven document to do > >> the setup. > >This is currently partly doable using a properties file in the sandbox > >$ python advanced_env.py -i xen.properties -o xen.cfg > > > >gives out a advanced zone config for the properties specified. > >Is data driven similar or you are talking about a DSL that Edison > >mentioned at CCC12? > > > >> > >> For a developer to checkin: > >> > >> - S/he must writes marvin tests for their feature and add > >> it to the BVT. > >I made some progress on this in my marvin-refactor branch and will > >collect my thoughts in an FS that I am drafting. The marvin tests are > >difficult to write in their current form. I'm following Chiradeep's > >recommendation for providing factories and once those are baked a DSL > >language devs can quickly mock down tests against a simulator using the > >above mentioned mvn profiles. > > > >> > >> - S/he must run the checkin tests to verify everything works. > >> - If the feature contains a hardware/vpx component, simulation > >>must be added. > >> > >> At this point, everything is about the developer writing in their > >>feature branch and merging in. > >> > >> On infrastructure side: > >> > >> - We'll setup continuous BVT based on the simulator. > >I brought this up on the IRC and the list y'day, so +1 - happy to help > >> > >> - I again push that we must use Gerrit to test the code > >> before it gets merge into the branch but I'll leave that for someone > >> else to do that. > >> > >> Let me know what you guys think. I'll probably break out a bvt > >> branch to work on this. Anyone want to join me? > > > >Count me in! > > > >-- > >Prasanna.,
