On 04/29/2014 09:55 AM, Lucas Meneghel Rodrigues wrote:
Hi folks:
As part of a research project we've been working on the past couple of
months, we'd like to ask for input about an important part of what
comprises virt-test, the following problem:
"given a set of input test parameters, we want to generate a test
matrix from those input parameters in a relatively compact fashion."
The solution, implemented way back in 2008 [1] and since then refined
is what we know today as the cartesian config format. As it stands
today, its API is documented here:
http://virt-test.readthedocs.org/en/latest/api/virttest.html#module-virttest.cartesian_config
The user documentation was very competently written by Chris here:
http://virt-test.readthedocs.org/en/latest/advanced/VirtTestDocumentation.html#cartesian-configuration
Over the years, we've had complaints about our solution/implementation:
1) Complex: Many people, myself included, took a non trivial amount of
time to grasp the format.
2) Fragile: The format is indentation based, messing up as much as a
single space will cause your statement to not be processed, leading to
long periods of staring at the screen in disbelief, trying to
understand what is wrong.
3) Inflexible: For performance reasons we have generators, and the
order of dictionaries is fixed, following the order of variants
definition inside the cartesian file. No way as of today of specifying
test1, test4, test2 to be executed in this order, if the variants were
defined in the right numeric sequence in the cartesian file.
4) There is no way to generate variants from inside tests. If I want
to test all cpu types in qemu, I need to know those beforehand and
write them in the config file. I can't discover those off from a test
and then generate new tests from this discovery, for example.
It's not all badness, though. Virt-Test requires an elevated number of
variants, creating literally millions of possible combinations, and
the cartesian config allows us to create all those scenarios on a
compact fashion. When looking at ways other projects implement param
multiplexing systems, the file format is remarkably similar to the
cartesian format, see:
http://docs.travis-ci.com/user/languages/python/#Choosing-Python-versions-to-test-against
From the feedback gathered so far, the work items are:
1) Complexity
Solution: Implement Paolo's RFC outlined here:
https://www.redhat.com/archives/virt-test-devel/2014-February/msg00073.html
Would be a start
A little confusing why we need use 'requires' and 'conflicts' to instead
of 'only' and 'no' for this RFC. Seems for cartesian config it doesn't
matter where the parameters from. No matter it is from cmd or cfg files,
they should be treated the same. So this propose can work without the
keywords change. From user side, we like to keep APIs as stable as it
can. The same for the definition of '@'. Maybe extend it to "hide and
default items for cartesian tuple" is better?
2) Frailty
Solution: Make the parser able to tell people wether they have messed
up their indentation, and be loud and fail instead of silently
ignoring the non-perfectly-aligned data.
This really sound great. We really need this as sometimes some parameter
is renamed it is really painful. And for some new one who is just
starting to use autotest. This really takes some time know how to deal
with it.
3) Inflexibility
Solution: Create a method that expands what the generators created,
and order according to the order the user asked for.
Great. Now we resolve this by writing a more complex test.cfg file to
setup orders in test loops with a new set of variants. But this is a
little complex and can not work with ./run. This feature really can help
to run a set up simple cases to cover a complex scenario.
4) Generate variants from tests
Solution: Well, no idea on this one yet :)
And there is one feature we'd like to add to the cartesian config:
- Set up boundary for some parameters like equal, max and min.
We used to update some code for it. And it is very useful for set up
boundaries for some hardware resource for guests, such like cpu, memory.
Different guests have different limitation for vcpu number, cpu thread
number and memory etc.. As there are more than one cfg files and
included in a certain order some parameters may be covered by the last
value set in cfg files. And with this feature we can set up the related
parameters' boundary in one of the cfg files, like guest-os.cfg for my
example. And we can ignore the unexpected test failed or error with a
warning message from cartesian config to show user that the cpu or
memory set up is out of boundary will set up to the request value.
Let us know whether you agree with my assessment, and if there's
something I missed, including a better format/way to solve the
original problem. If you read so far, thank you very much for your
patience!
Lucas
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
