Just as an FYI, we've dramatically increased Travis capacity. https://blogs.apache.org/infra/entry/apache_gains_additional_travis_ci
On Thursday, April 30, 2015, Abhinandan Prateek < abhinandan.prat...@shapeblue.com> wrote: > Yes, lot of sanity testing can be accomplished by the automated test > suites that we already have. > > There are still lot of tests that can be run using the travis-CI, (need to > see how many more without loading the travis system). Some of the companies > including Shapeblue are already running most or all of the test cases. > This means that the latest branches (4.5, 4.6) already have a level of > sanity. > > At this stage we should define the amount of coverage that automation > provides and also look at the areas where automation is failing, and work > towards covering it. We need this as we continue to find issues that are > not covered by automation. > > I agree that while releasing if issues are found then we fix them and also > write tests so that these do not reoccur. > > -abhi > > > On 30-Apr-2015, at 7:42 am, David Nalley <da...@gnsa.us <javascript:;>> > wrote: > > > > The problem is really two fold. > > Cutting an RC isn't terribly time consuming, once you've done it a > > dozen times or so, it goes by pretty quickly. > > > > The problem is the time it takes to do any material testing; and that > > we don't have a way of telling what the status of a candidate is > > without doing lots of manual testing. We've had multiple cases of > > having enough binding votes for 4.5.x candidates only for someone to > > find problems. This leads me (and this could just be my perspective) > > to let an RC sit for a while - get folks to test it. My experience as > > a person voting is that rapid churn of RCs leads me to not even > > attempt to setup a testing environment, much less do any testing. I > > found that by the time I could get around to testing an RC vote had > > been cancelled and it had been rerolled, sometimes twice. > > > > I've said this before, and I'll toss it out again. I think we have to > > start 'trusting' the automated tests. We find lots of problems > > manually, but we don't improve the testing situation for those blocker > > issues that stopped us from releasing. I'm of the opinion we should > > release quickly, very quickly. But if we have to spend a week just > > looking for bugs, that will never fly. So - let the standard be > > automated tests - and if there's a problem, we create an automated > > test for it, and solve in a week or two. That will force us to live > > and die by automated tests. > > > > --David > > > > On Wed, Apr 29, 2015 at 9:45 AM, Rohit Yadav <rohit.ya...@shapeblue.com > <javascript:;>> wrote: > >> Hi Wilder - Thanks we’ll need all the ammo we have :) I’ve already done > my testings wrt XS 6.2/6.5 and KVM qemu 2.0/2.1/2.2 so will test KVM 1.5 > and VMware 5.5. > >> > >> Hi Daan - makes sense, we’re “so" late with 4.5.1 that we need to focus > on 4.6.0 to avoid delaying it. I’ve asked David regarding co-piloting the > next 4.5.1 RC since he could be busy and I want to help with lifting some > weights. I’m planning to do basic tests with VMware today, publish new > systemvmtemplates tomorrow (I’m already done with KVM and Xen) and plan to > cut a 4.5.1 RC on Monday and if we don’t hit any blocker or regressions ACS > 4.5.1 should be released by the end of next week. > >> > >>> On 29-Apr-2015, at 2:49 pm, Wilder Rodrigues < > wrodrig...@schubergphilis.com <javascript:;>> wrote: > >>> > >>> Hi Rohit, > >>> > >>> I will join you in testing 4.5 form next week - just have to finish > some stuff. > >>> > >>> My environments will be: > >>> > >>> * Xen 6.2/6.5 > >>> * KVM (qemu 1.5.3 and 2.1) > >>> > >>> Let’s rock! > >>> > >>> Cheers, > >>> Wilder > >>> > >>> On 28 Apr 2015, at 18:33, Rohit Yadav <rohit.ya...@shapeblue.com > <javascript:;>> wrote: > >>> > >>>> Hi all, > >>>> > >>>> ### 4.5 Release Effort > >>>> > >>>> I’ve been exhaustively testing ACS 4.5 wrt Xen 6.2, Xen 6.5, KVM > (qemu 2.0, 2.3) and I feel we’re pretty good but we need testing efforts > and I want to help drive efforts to releasing ACS 4.5.1 -- if you’ve any > issues that you would like to get fixed that are either regressions or > blockers can you please share on this thread? If we don’t find any blockers > or regressions let us invest in ACS 4.5 testing and release it soon. For > other minor issues we can always fix them and release 4.5.2 etc if and when > needed in future. > >>>> > >>>> Regarding my 4.5 testing - I’ve tested basic vm life cycle operations > (deploy, start, stop, destroy/expunge, migrate to another host, migrate to > another storage pool, deploy using iso) for all cases and found no issues; > >>>> > >>>> KVM with basic zone (with/without security groups, SG blockers were > found and fixed) > >>>> KVM with Adv zone (with/without security groups, both isolated > network and VPC worked, networking issues were found and fixed) > >>>> XenServer 6.2 with basic and adv zones (with/without security groups, > no SG issues found) > >>>> XenServer 6.5 with basic and adv zones (with/without security groups, > SG blockers found and fixed) > >>>> > >>>> I found one issue that could be a blocker (if it’s not a hardware/env > issue) -- when using security groups in advance zone with a dedicated vlan > id (say vlan://500), I was unable to access the VMs (CPVM, SSVM, VR or user > vms) though all normal vm_life cycles seems to work. Though this worked for > me when I used the vlan://untagged. I tried to add vlan id 500 to my local > nics using vconfig but I still was n’t able to do access the CPVM or user > VMs. I guess my understanding of vlans with security groups is limited, so > if anyone knows about this feature or has used it - please help with some > regression testing. I plan to continue testing this week with VMWare and > fix any issues we find. > >>>> > >>>> > >>>> ### Testing against Xen, KVM, VMWare etc. > >>>> > >>>> With a recent improvement to allow cpu features (such as vmx) on KVM > hosts ( > https://github.com/apache/cloudstack/commit/58cc569273905c50d089f9fd82fe80028b4e9775), > it’s now possible to run KVM, ESX, XenServer, OVM3 (LXC, and hopefully > others) on KVM (in case of ESX a patched qemu would be needed: > http://people.apache.org/~bhaisaab/qemu/). Such a (ansible based) tool > aims to be reproduce such an environment for anyone and serve as a > developer kit (not to be confused with the DevCloud appliance, will share > more details of the developer kit later) and as an infrastructure to run > integration testing suite we have now. This would allow us to > build/test/run CloudStack against various nested-virtualized hosts running > as guests on KVM, using CloudStack. > >>>> > >>>> Meanwhile, Abhi is focusing on existing (integration) tests and has > increased the number of TravisCI tests: > https://travis-ci.org/apache/cloudstack/builds/60365947 > >>>> > >>>> Our goal with these efforts is to (1) have a developer kit that a > developer can have locally to build/test/develop CloudStack, (2) scale it > up for bigger baremetal servers to run long running integration tests and > have a better QA automation. > >>>> > >>>> Regards, > >>>> Rohit Yadav > >>>> Software Architect, ShapeBlue > >>>> M. +91 88 262 30892 | rohit.ya...@shapeblue.com <javascript:;> > >>>> Blog: bhaisaab.org | Twitter: @_bhaisaab > >>>> > >>>> > >>>> > >>>> Find out more about ShapeBlue and our range of CloudStack related > services > >>>> > >>>> IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > >>>> CSForge – rapid IaaS deployment framework< > http://shapeblue.com/csforge/> > >>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > >>>> CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > >>>> CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > >>>> CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > >>>> > >>>> This email and any attachments to it may be confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If you > are not the intended recipient of this email, you must neither take any > action based upon its contents, nor copy or show it to anyone. Please > contact the sender if you believe you have received this email in error. > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue > Services India LLP is a company incorporated in India and is operated under > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a > company incorporated in Brasil and is operated under license from Shape > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is > a registered trademark. > >>> > >> > >> Regards, > >> Rohit Yadav > >> Software Architect, ShapeBlue > >> M. +91 88 262 30892 | rohit.ya...@shapeblue.com <javascript:;> > >> Blog: bhaisaab.org | Twitter: @_bhaisaab > >> > >> > >> > >> Find out more about ShapeBlue and our range of CloudStack related > services > >> > >> IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > >> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/ > > > >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > >> CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > >> CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > >> CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > >> > >> This email and any attachments to it may be confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If you > are not the intended recipient of this email, you must neither take any > action based upon its contents, nor copy or show it to anyone. Please > contact the sender if you believe you have received this email in error. > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue > Services India LLP is a company incorporated in India and is operated under > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a > company incorporated in Brasil and is operated under license from Shape > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is > a registered trademark. > > Find out more about ShapeBlue and our range of CloudStack related services > > IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > > This email and any attachments to it may be confidential and are intended > solely for the use of the individual to whom it is addressed. Any views or > opinions expressed are solely those of the author and do not necessarily > represent those of Shape Blue Ltd or related companies. If you are not the > intended recipient of this email, you must neither take any action based > upon its contents, nor copy or show it to anyone. Please contact the sender > if you believe you have received this email in error. Shape Blue Ltd is a > company incorporated in England & Wales. ShapeBlue Services India LLP is a > company incorporated in India and is operated under license from Shape Blue > Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil > and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is > a company registered by The Republic of South Africa and is traded under > license from Shape Blue Ltd. ShapeBlue is a registered trademark. >
