On Mon, 2016-10-10 at 10:25 -0400, Kamil Paral wrote:

> > Hi folks! Sending this to devel@ as well as test@ as there's been some
> > relevant discussion there recently. We've been kicking around a couple
> > of issues lately:
> > 
> > 1. Exactly what do we need to test and block on, in terms of writing
> > images to USB sticks?
> > 
> > 2. 'Default boot and install' table was supposed to require bare metal
> > optical media testing at release time, but this is not clear, and
> > openQA fills in results despite not testing real media on bare metal.
> > 
> > So here's my proposal to address both topics. It involves changes to
> > the Boot_default_install test:
> > 
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_install
> > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_Boot_default_install&diff=476645&oldid=476644
> > 
> > and the Installation matrix template page:
> > 
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
> > https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installation_test_matrix&diff=476646&oldid=476643
> > 
> > to summarize the changes - we improve the test case's wording to give
> > explicit instructions for all three media types we care about (virtual
> > disc attached to a VM, real optical disc in a real machine, real USB
> > stick in a real machine), we ditch the USB test matrices entirely, and
> > we give the 'Default boot and install' tables extra result columns so
> > they can now reflect results for VM, CD/DVD and USB testing (for BIOS
> > and UEFI in all three cases, for x86_64).

It's a lot of empty fields, which can't be automated by definition. 

Yep, yep it is.

> The new matrix version seems to be more time consuming that the
> previous version.

In fact, the count of 'empty spaces' is identical in both versions. In
the old version there were 24 empty spaces that humans were *supposed*
to be filling in (though this wasn't always really happening): 12 in
the "Default boot and install" table, and 12 in the USB table (it's a
4x3 table). In the new version, there are...24 empty spaces for humans
to fill in in the "Default boot and install (x86_64)" table.

The coverage is slightly *different*, but the amount of tests humans
are intended to run has not in fact changed. We lose the fudge factor
of openQA filling in the spaces I really intended humans to test on
real machines with real shiny discs, but that's kind of the *point* of
the change, so it's not a reason to not do it...

>  I wonder what we can do to cut down the time spent on this. Here are
> a few ideas:
> a) could we decouple boot from install?
> I know we've had a few problems in the past that occurred only on
> bare metal and not in VMs. But do I remember correctly that all of
> those were related to booting the installer/live system, and not the
> installation? What we do here is that we repeat the default
> installation over and over again, e.g. Workstation Live using BIOS
> and UEFI (which results in a slightly different partition layout),
> from different sources (DVD in VM, DVD on bare metal, USB). But do
> those environments (VM vs bare metal, DVD vs USB) actually matter, is
> there a good reason to repeat them that many times? What if we reduce
> all of this to:
> Default install
> ===============
>                         Result
> Workstation Live         [ ]
> Workstation netinst      [ ]
> Everything netinst       [ ]
> Server netinst           [ ]
> Server DVD               [ ]
> KDE Live                 [ ]
> which would be satisfiable using ANY environment and it would just
> check that anaconda can use that particular installation source
> correctly? The BIOS vs UEFI partitioning is already covered
> thoroughly in the "Guided storage configuration" table.

> The currently proposed "Default boot and install" section would stay
> the same, but be renamed to "Default boot" and would just check that
> the image *boots* into the installer in that particular environment.

I think this is going a bit too far. There are, IIRC, at least two
known places beyond just initial boot where things can go wrong in ways
related to the install medium.

One is install target selection. anaconda has code for filtering out
'protected' devices which is intended (in part) to prevent you trying
to install to the USB stick you're running the install from. That code
can go wrong in various ways, including over-aggressive filtering, and
the bugs can be media-specific (e.g. there's no problem in a VM or from
a real optical disc, but it goes wrong when booting from a USB stick).

The other is use of the on-medium repository. This is mainly for the
Server DVD image. When you install from that image the install should
use the packages from the medium itself as the base repository (and not
go out and get them from the network); I believe we've seen a case
before where the Server DVD written to USB would boot OK, and even
install OK, but it didn't actually use the USB stick as a repository,
it acted like a network install image. When written to a DVD it worked
OK. That was a long time ago, though, and I don't have the bug
reference handy.

This is why the test case has this text:

"If the image under test is intended to act as a package source, it
must be the default repository and installation must in fact use the
media as the package source."

We *could* reduce the test case to 'boot, select a target disk and
start the installation process if using a DVD image', I guess. Would
this sufficiently address your concerns?

Of course it's *possible* that some weird bug could show up which made
bootloader configuration not work properly in some media-dependent way,
or something. But I don't recall us running into such a bug before.

> > 
> > At present I'm not proposing any changes to the *release criteria* (we
> > can maybe consider that later). This is just a proposal for what we
> > will actually guarantee testing of via the validation process.
> If we go full steam ahead with optical disk testing as proposed (in
> the past, the "default boot and install" test case said you can
> choose either USB or DVD, now we have separate fields for both
> implying we need to test both) I'd suggest optical media being only
> in the Final criteria.

That was my intent, but it's a bit hard to express with the way we lay
out the wiki tables, unfortunately (hard to specify different
milestones for different environments). I can add an admon note above
the table, I guess, explaining precisely what's actually required for
each milestone. How does that sound?

Thanks for the feedback!
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
test mailing list -- test@lists.fedoraproject.org
To unsubscribe send an email to test-le...@lists.fedoraproject.org

Reply via email to