On Wed, Oct 12, 2016 at 2:13 PM, Devananda van der Veen < devananda....@gmail.com> wrote:
> > > On 10/12/2016 05:01 AM, Dmitry Tantsur wrote: > > Hi folks! > > > > I'd like to propose a plan on how to simultaneously extend the coverage > of our > > jobs and reduce their number. > > > > Currently, we're running one instance per job. This was reasonable when > the > > coreos-based IPA image was the default, but now with tinyipa we can run > up to 7 > > instances (and actually do it in the grenade job). I suggest we use 6 > fake bm > > nodes to make a single CI job cover many scenarios. > > > > The jobs will be grouped based on driver (pxe_ipmitool and > agent_ipmitool) to be > > more in sync with how 3rd party CI does it. A special configuration > option will > > be used to enable multi-instance testing to avoid breaking 3rd party CI > systems > > that are not ready for it. > > > > To ensure coverage, we'll only leave a required number of nodes > "available", and > > deploy all instances in parallel. > > > > In the end, we'll have these jobs on ironic: > > gate-tempest-ironic-pxe_ipmitool-tinyipa > > gate-tempest-ironic-agent_ipmitool-tinyipa > > > > Each job will cover the following scenarious: > > * partition images: > > ** with local boot: > > ** 1. msdos partition table and BIOS boot > > ** 2. GPT partition table and BIOS boot > > ** 3. GPT partition table and UEFI boot <*> > > ** with netboot: > > ** 4. msdos partition table and BIOS boot <**> > > * whole disk images: > > * 5. with msdos partition table embedded and BIOS boot > > * 6. with GPT partition table embedded and UEFI boot <*> > > > > <*> - in the future, when we figure our UEFI testing > > <**> - we're moving away from defaulting to netboot, hence only one > scenario > > > > I suggest creating the jobs for Newton and Ocata, and starting with > Xenial right > > away. > > > > Any comments, ideas and suggestions are welcome. > > Huge +1 on this from me, as well. > > I am also in favor of some of the other suggestions on this thread, namely, > moving API testing over to functional tests so that those can be run more > quickly / with less resources / without affecting tempest scenario tests. > > I also think we should begin defining additional scenario tests to cover > things > we are not covering today (eg, adopt a running instance), as Vasyl already > pointed out. But I don't think that conflicts or prevents the changes > you're > suggesting, Dmitry. > > -Deva > > ++ Thanks for bringing this up Dmitry! Might I suggest, if we don't already have it, that this would be a good time to track (in a spreadsheet-like form), the jobs with the tests covered by each job (or desired but not covered yet). I can never remember what we are testing vs not testing. (e.g. I thought we had adoption of a running instance.) --ruby
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev