On Thu, Jun 06, 2013 at 09:04:55AM +0200, Ove Ewerlid wrote:
> On 06/06/2013 08:37 AM, Prasanna Santhanam wrote:
> >On Thu, Jun 06, 2013 at 08:29:26AM +0200, Ove Ewerlid wrote:
> >>On 06/06/2013 07:10 AM, Prasanna Santhanam wrote:
> >>>On Wed, Jun 05, 2013 at 05:39:16PM +0000, Edison Su wrote:
> >>>>I think we miss a VOTE from Jenkins, the vote from Jenkins should
> >>>>be taken as highest priority in each release. This kind of
> >>>>regression should be easily identified in Jenkins(If we have a
> >>>>regression test for each environment).
> >>>>
> >>>
> >>>+1 - need more people focussed on cloudstack-infra in general.
> >>
> >>The 41 regression with local storage, that required 2 or more hosts
> >>to duplicate, would be one example of an issue that would be
> >>detected by automatic testing provided the testing is done on a
> >>sufficiently big test fixture.
> >>
> >>Q: How many hosts are used in daily testing now?
> >
> >3 (2 in a cluster, 1 in a second pod) and 1 in a second zone -
> >totalling 4 hosts in the test rig.
> >
> >But I don't enable local storage on it. It's occupied testing XCP,
> >Xen and KVM with shared storage. The more configurations the longer
> >the test run time.
> >
>
> Not sure if you use multiple run queues, one queue with a more
> extensive job that runs ones per day to capture issues in a larger
> test fixture that is not suitable to build for every single commit.
> This test needs to complete within 24 hours.
>
We don't run tests for every commit. The tests run every four-five
hours for the three hypervisors. So each test run has collated a group
of commits made during the time window. Each test run splits into
multiple sub-jobs that are running in parallel.
The extensive jobs that test for regression run on Wednesday and
Saturday. These can take ~6hours to finish.
ASCII representation of how the jobs split up:
test-matrix
|___ test-packaging (new centos VM with latest packaged CloudStack)
|___ test-environment-refresh (kickstarts fresh hypervisors)
|_____test-setup-advanced-zone
|____test-smoke-matrix (Weekdays, except Wed)
|___ test#1
|___ test#2
|___ test#3
|___
|___ test#n
|____test-regression-matrix (Wed, Sat)
HTH
--
Prasanna.,
