Yes, lot of sanity testing can be accomplished by the automated test suites that we already have.
There are still lot of tests that can be run using the travis-CI, (need to see how many more without loading the travis system). Some of the companies including Shapeblue are already running most or all of the test cases. This means that the latest branches (4.5, 4.6) already have a level of sanity. At this stage we should define the amount of coverage that automation provides and also look at the areas where automation is failing, and work towards covering it. We need this as we continue to find issues that are not covered by automation. I agree that while releasing if issues are found then we fix them and also write tests so that these do not reoccur. -abhi > On 30-Apr-2015, at 7:42 am, David Nalley <da...@gnsa.us> wrote: > > The problem is really two fold. > Cutting an RC isn't terribly time consuming, once you've done it a > dozen times or so, it goes by pretty quickly. > > The problem is the time it takes to do any material testing; and that > we don't have a way of telling what the status of a candidate is > without doing lots of manual testing. We've had multiple cases of > having enough binding votes for 4.5.x candidates only for someone to > find problems. This leads me (and this could just be my perspective) > to let an RC sit for a while - get folks to test it. My experience as > a person voting is that rapid churn of RCs leads me to not even > attempt to setup a testing environment, much less do any testing. I > found that by the time I could get around to testing an RC vote had > been cancelled and it had been rerolled, sometimes twice. > > I've said this before, and I'll toss it out again. I think we have to > start 'trusting' the automated tests. We find lots of problems > manually, but we don't improve the testing situation for those blocker > issues that stopped us from releasing. I'm of the opinion we should > release quickly, very quickly. But if we have to spend a week just > looking for bugs, that will never fly. So - let the standard be > automated tests - and if there's a problem, we create an automated > test for it, and solve in a week or two. That will force us to live > and die by automated tests. > > --David > > On Wed, Apr 29, 2015 at 9:45 AM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: >> Hi Wilder - Thanks we’ll need all the ammo we have :) I’ve already done my >> testings wrt XS 6.2/6.5 and KVM qemu 2.0/2.1/2.2 so will test KVM 1.5 and >> VMware 5.5. >> >> Hi Daan - makes sense, we’re “so" late with 4.5.1 that we need to focus on >> 4.6.0 to avoid delaying it. I’ve asked David regarding co-piloting the next >> 4.5.1 RC since he could be busy and I want to help with lifting some >> weights. I’m planning to do basic tests with VMware today, publish new >> systemvmtemplates tomorrow (I’m already done with KVM and Xen) and plan to >> cut a 4.5.1 RC on Monday and if we don’t hit any blocker or regressions ACS >> 4.5.1 should be released by the end of next week. >> >>> On 29-Apr-2015, at 2:49 pm, Wilder Rodrigues >>> <wrodrig...@schubergphilis.com> wrote: >>> >>> Hi Rohit, >>> >>> I will join you in testing 4.5 form next week - just have to finish some >>> stuff. >>> >>> My environments will be: >>> >>> * Xen 6.2/6.5 >>> * KVM (qemu 1.5.3 and 2.1) >>> >>> Let’s rock! >>> >>> Cheers, >>> Wilder >>> >>> On 28 Apr 2015, at 18:33, Rohit Yadav <rohit.ya...@shapeblue.com> wrote: >>> >>>> Hi all, >>>> >>>> ### 4.5 Release Effort >>>> >>>> I’ve been exhaustively testing ACS 4.5 wrt Xen 6.2, Xen 6.5, KVM (qemu >>>> 2.0, 2.3) and I feel we’re pretty good but we need testing efforts and I >>>> want to help drive efforts to releasing ACS 4.5.1 -- if you’ve any issues >>>> that you would like to get fixed that are either regressions or blockers >>>> can you please share on this thread? If we don’t find any blockers or >>>> regressions let us invest in ACS 4.5 testing and release it soon. For >>>> other minor issues we can always fix them and release 4.5.2 etc if and >>>> when needed in future. >>>> >>>> Regarding my 4.5 testing - I’ve tested basic vm life cycle operations >>>> (deploy, start, stop, destroy/expunge, migrate to another host, migrate to >>>> another storage pool, deploy using iso) for all cases and found no issues; >>>> >>>> KVM with basic zone (with/without security groups, SG blockers were found >>>> and fixed) >>>> KVM with Adv zone (with/without security groups, both isolated network and >>>> VPC worked, networking issues were found and fixed) >>>> XenServer 6.2 with basic and adv zones (with/without security groups, no >>>> SG issues found) >>>> XenServer 6.5 with basic and adv zones (with/without security groups, SG >>>> blockers found and fixed) >>>> >>>> I found one issue that could be a blocker (if it’s not a hardware/env >>>> issue) -- when using security groups in advance zone with a dedicated vlan >>>> id (say vlan://500), I was unable to access the VMs (CPVM, SSVM, VR or >>>> user vms) though all normal vm_life cycles seems to work. Though this >>>> worked for me when I used the vlan://untagged. I tried to add vlan id 500 >>>> to my local nics using vconfig but I still was n’t able to do access the >>>> CPVM or user VMs. I guess my understanding of vlans with security groups >>>> is limited, so if anyone knows about this feature or has used it - please >>>> help with some regression testing. I plan to continue testing this week >>>> with VMWare and fix any issues we find. >>>> >>>> >>>> ### Testing against Xen, KVM, VMWare etc. >>>> >>>> With a recent improvement to allow cpu features (such as vmx) on KVM hosts >>>> (https://github.com/apache/cloudstack/commit/58cc569273905c50d089f9fd82fe80028b4e9775), >>>> it’s now possible to run KVM, ESX, XenServer, OVM3 (LXC, and hopefully >>>> others) on KVM (in case of ESX a patched qemu would be needed: >>>> http://people.apache.org/~bhaisaab/qemu/). Such a (ansible based) tool >>>> aims to be reproduce such an environment for anyone and serve as a >>>> developer kit (not to be confused with the DevCloud appliance, will share >>>> more details of the developer kit later) and as an infrastructure to run >>>> integration testing suite we have now. This would allow us to >>>> build/test/run CloudStack against various nested-virtualized hosts running >>>> as guests on KVM, using CloudStack. >>>> >>>> Meanwhile, Abhi is focusing on existing (integration) tests and has >>>> increased the number of TravisCI tests: >>>> https://travis-ci.org/apache/cloudstack/builds/60365947 >>>> >>>> Our goal with these efforts is to (1) have a developer kit that a >>>> developer can have locally to build/test/develop CloudStack, (2) scale it >>>> up for bigger baremetal servers to run long running integration tests and >>>> have a better QA automation. >>>> >>>> Regards, >>>> Rohit Yadav >>>> Software Architect, ShapeBlue >>>> M. +91 88 262 30892 | rohit.ya...@shapeblue.com >>>> Blog: bhaisaab.org | Twitter: @_bhaisaab >>>> >>>> >>>> >>>> Find out more about ShapeBlue and our range of CloudStack related services >>>> >>>> IaaS Cloud Design & >>>> Build<http://shapeblue.com/iaas-cloud-design-and-build//> >>>> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >>>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >>>> CloudStack Software >>>> Engineering<http://shapeblue.com/cloudstack-software-engineering/> >>>> CloudStack Infrastructure >>>> Support<http://shapeblue.com/cloudstack-infrastructure-support/> >>>> CloudStack Bootcamp Training >>>> Courses<http://shapeblue.com/cloudstack-training/> >>>> >>>> This email and any attachments to it may be confidential and are intended >>>> solely for the use of the individual to whom it is addressed. Any views or >>>> opinions expressed are solely those of the author and do not necessarily >>>> represent those of Shape Blue Ltd or related companies. If you are not the >>>> intended recipient of this email, you must neither take any action based >>>> upon its contents, nor copy or show it to anyone. Please contact the >>>> sender if you believe you have received this email in error. Shape Blue >>>> Ltd is a company incorporated in England & Wales. ShapeBlue Services India >>>> LLP is a company incorporated in India and is operated under license from >>>> Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company >>>> incorporated in Brasil and is operated under license from Shape Blue Ltd. >>>> ShapeBlue SA Pty Ltd is a company registered by The Republic of South >>>> Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a >>>> registered trademark. >>> >> >> Regards, >> Rohit Yadav >> Software Architect, ShapeBlue >> M. +91 88 262 30892 | rohit.ya...@shapeblue.com >> Blog: bhaisaab.org | Twitter: @_bhaisaab >> >> >> >> Find out more about ShapeBlue and our range of CloudStack related services >> >> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> >> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> >> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> >> CloudStack Software >> Engineering<http://shapeblue.com/cloudstack-software-engineering/> >> CloudStack Infrastructure >> Support<http://shapeblue.com/cloudstack-infrastructure-support/> >> CloudStack Bootcamp Training >> Courses<http://shapeblue.com/cloudstack-training/> >> >> This email and any attachments to it may be confidential and are intended >> solely for the use of the individual to whom it is addressed. Any views or >> opinions expressed are solely those of the author and do not necessarily >> represent those of Shape Blue Ltd or related companies. If you are not the >> intended recipient of this email, you must neither take any action based >> upon its contents, nor copy or show it to anyone. Please contact the sender >> if you believe you have received this email in error. Shape Blue Ltd is a >> company incorporated in England & Wales. ShapeBlue Services India LLP is a >> company incorporated in India and is operated under license from Shape Blue >> Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil >> and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a >> company registered by The Republic of South Africa and is traded under >> license from Shape Blue Ltd. ShapeBlue is a registered trademark. Find out more about ShapeBlue and our range of CloudStack related services IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> CloudStack Software Engineering<http://shapeblue.com/cloudstack-software-engineering/> CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> CloudStack Bootcamp Training Courses<http://shapeblue.com/cloudstack-training/> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
