Ok. Do we need to call a vote or something to change our rules to solidify that we should require at least two votes from each supported platform, whether they be automated tests or contributor tests?
On Thu, Jun 6, 2013 at 2:20 AM, Prasanna Santhanam <prasanna.santha...@citrix.com> wrote: > On Thu, Jun 06, 2013 at 09:04:55AM +0200, Ove Ewerlid wrote: >> On 06/06/2013 08:37 AM, Prasanna Santhanam wrote: >> >On Thu, Jun 06, 2013 at 08:29:26AM +0200, Ove Ewerlid wrote: >> >>On 06/06/2013 07:10 AM, Prasanna Santhanam wrote: >> >>>On Wed, Jun 05, 2013 at 05:39:16PM +0000, Edison Su wrote: >> >>>>I think we miss a VOTE from Jenkins, the vote from Jenkins should >> >>>>be taken as highest priority in each release. This kind of >> >>>>regression should be easily identified in Jenkins(If we have a >> >>>>regression test for each environment). >> >>>> >> >>> >> >>>+1 - need more people focussed on cloudstack-infra in general. >> >> >> >>The 41 regression with local storage, that required 2 or more hosts >> >>to duplicate, would be one example of an issue that would be >> >>detected by automatic testing provided the testing is done on a >> >>sufficiently big test fixture. >> >> >> >>Q: How many hosts are used in daily testing now? >> > >> >3 (2 in a cluster, 1 in a second pod) and 1 in a second zone - >> >totalling 4 hosts in the test rig. >> > >> >But I don't enable local storage on it. It's occupied testing XCP, >> >Xen and KVM with shared storage. The more configurations the longer >> >the test run time. >> > >> >> Not sure if you use multiple run queues, one queue with a more >> extensive job that runs ones per day to capture issues in a larger >> test fixture that is not suitable to build for every single commit. >> This test needs to complete within 24 hours. >> > > We don't run tests for every commit. The tests run every four-five > hours for the three hypervisors. So each test run has collated a group > of commits made during the time window. Each test run splits into > multiple sub-jobs that are running in parallel. > > The extensive jobs that test for regression run on Wednesday and > Saturday. These can take ~6hours to finish. > > ASCII representation of how the jobs split up: > > test-matrix > |___ test-packaging (new centos VM with latest packaged CloudStack) > |___ test-environment-refresh (kickstarts fresh hypervisors) > |_____test-setup-advanced-zone > |____test-smoke-matrix (Weekdays, except Wed) > |___ test#1 > |___ test#2 > |___ test#3 > |___ > |___ test#n > |____test-regression-matrix (Wed, Sat) > > > HTH > > -- > Prasanna.,
