Dne 16.5.2014 v 23:28 Ademar Reis napsal(a):
Changes from v1->v2
- changed 'filter: "only ..."' to "filter-only: ..."
- changed 'filter: "no ..."' to "filter-out: ..."
- changed 'filter: "depth ...' to "filter-depth: ..."
- added a section explaining filter behaviors and an example of
a multiplexation of the resulting tree.
- clarification about default values
- minor fixes/improvements
The problem
-----------
A multiplexer generates a combination of variables that basically
allows one test to be run on different scenarios and with
different parameters. In other words, users write just one test,
but the test runner runs n tests, with n being the size of the
multiplexed set.
The current cartesian format, as used in autotest is
unstructured. It's a collection of dictionaries and lists that
when combined and multiplexed easily explode into a huge data
set. A lot of effort has been put into optimizations and cleanups
along the years, but it still remains as one of the most complex
parts of the autotest framework.
It's relatively difficult to understand and given the global
nature of the variants/variables (no proper namespace or
compartmentalization), the format of the file and the size of the
variants set, it's very error prone.
It's currently very common to find global filters in the code
such as "only raw", "only up" or "only ide". Filters easily
become complex and difficult to understand and debug. Given
they're applied on top of the resulting (huge) set, they are
inherently slow to process and hard to understand and validate.
I believe this lack of structure is the root cause of many of the
problems or dislikes we have with the cartesian system.
...
OK, resurrecting this old thread with new idea.
### Introduction
Our goal is to multiplex setting for various platforms, hw layouts and
tests, ...
As example I'm using mix of yaml file similarly to Ademar's RFC, the
only change is that multiplexation of leafs would happen only on leafs
of lists of dicts. This helps us solve problem with the need to
filter-out tests, oss, ... (will talk about it later).
The final tree would be compound from 3 places:
1) default tree (big tree with all the variants)
- hw:
foo: 123
bar: [1,2,3]
variants:
- cpu:
cflags: '-O2'
amd:
@intel:
arm:
filter-only:
- '/env/debug'
- '/os/linux'
filter-out: ['/hw']
- disk:
scsi:
@virtio:
filter-only: ['/os/linux']
- nic_model:
@virtio:
nic_model: virtio
rtl8139:
nic_model: rtl8139
e1000:
nic_modle: e1000
- os:
linux:
dev-tools: 'gcc'
@fedora:
package-manager: 'yum'
init: 'systemd'
mint:
package-manager: 'apt-get'
init: 'init'
win:
bsod: true
dev-tools: ['Cygwin', 'MinGW', 'Visual Studio']
win7:
win8:
bsod: false
- env:
debug:
@prod:
filter-only: ['/os/win']
2) another files merged on cmdline (the same format (1)):
- overrides values and extends the existing nodes
- keyword to delete existing branch
3) each test has it's own small tree which will be merged into
/tests/$TESTNAME and list of key components to be multiplexed with
(either as file with the same name + .yaml or as variable inside the
python file __MUX__ = """..."""):
ping_test.yaml:
mux: ["/hw/nic_model"]
vms: "vm1 vm2"
# Other test default variables
variants:
- size:
@default:
long:
size = 65507
- interval:
@default:
short:
interval = 0
requires_root = true
=> would merge into /tests/ping_test node.
## Default variant
(test developer)
When nothing is specified, no multiplexation would happen, not even the
default mux tree is processed, default values will be used from the
global namespace or special hooks:
intel.virtio.virtio.fedora.prod.pingtest.default.default
## Full test
(feature developer, basic check)
When --mux-test=true|false is specified, multiplexer would take into
account the `mux` variable inside each specified test. So basically it
means all variants of the tests on one hw. (again, default mux tree is
not taken into account)
intel.virtio.virtio.fedora.prod.pingtest.default.default
intel.virtio.virtio.fedora.prod.pingtest.default.short
intel.virtio.virtio.fedora.prod.pingtest.long.default
intel.virtio.virtio.fedora.prod.pingtest.long.default
## All related components (Paolo, if I understood you correctly, this
one is for you ;-) )
(feature developer, check before submitting)
When --mux_related=true|false is specified, it multiplexes the nodes
specified in test's `mux:` node (so only parts of the tree specified in
`mux:` are processed and multiplexed):
intel.virtio.virtio.fedora.prod.pingtest.default.default
intel.virtio.rtl8139.fedora.prod.pingtest.default.default
intel.virtio.e1000.fedora.prod.pingtest.default.default
in case you use --mux-test and --mux-related it would run combinations
of above (12 variants).
## Multiplex all
(QA's who wants to test everything on everything unless it's disabled by
`filter-out`)
When --mux is specified, it process the full tree and multiplex leafs,
which are not in the same list item. This is a bit different than in
Ademar's proposal, but very similar to variants in autotest (with some
improvements). I'll talk about it in following chapter. For now let's
just say it'd execute 144 variants (If I'm not wrong).
This way you'd be able to specify tests in the old fashion, all variants
unless filter-out removes them.
### Multiplexation problem
Take a look at the ping_test using config from previous RFC:
hw:...
tests:
filter-out: ["/tests"]
ping_test:
vms: "vm1 vm2"
variants:
size:
default:
long:
size = 65507
interval:
default:
short:
interval = 0
requires_root = true
as you can see it makes sense to have tests-variants. But when we
filter-out tests, multiplexation won't happen and result would be:
ping_test.default
ping_test.long
ping_test.default
ping_test.short
the only way to make this work is to move the test's variants to the
global space:
hw: ...
tests:
ping_test:
variants:
filter-only: ["/tests/ping_test"]
size:
default:
...
where the filter-out is not set.
Alternatively we can specify the multiplexation domains similarly as we
did in autotest using key-word `variants`. This is what I proposed with
the list of dicts in yaml file (the `-` in the example above). The way
it works is that it discovers leafs and when it reaches `list of dicts`
(eg. "/hw/variants") it knows, that it has to multiplex each leaf from
"/hw/variants/cpu" with all leafs from "/hw/variants/disk" with all
leafs of "/hw/variants/nic_model".
It's very similar to autotests:
variants:
- cpu:
...
variants:
- disk:
...
variants:
- nic_model
The only difference is that autotest's version used the items, this yaml
version uses leafs. So for example you can use:
- hw:
- disk_format:
qcow:
format: qcow2
2:
2v3:
image_extra_params = "compat=1.1"
raw:
format = raw
...
(in autotest it had to be linear list, now you can do as complex
structure as you want with the variable inheritance and only leafs will
be used, which is kind-of cool.
Also this means you define the multiplexation so instead of multiplexing
everything which doesn't share the parent or you explicitly don't forbid
by `filter-only/out`, you specify domains and `filter-only/out` only the
parts which doesn't work together.
### Default parameters
params.get() should allow hooks for keywords. One that comes to my mind is:
params.get("/hw/cpu/arch") which would discover available architecture
(ppc vs. x86) and modify the result based on the hook. The hook would be
able to see the raw "/hw/cpu/arch" value so it would be able to deal
with problems (or None when not specified)
Similarly we'd be able to supply different versions, eg:
params.get("/sw/qemu/has_device") would return value based on system,
rather than cfg (well in fact we can see the raw value, so in case it's
set to false, we can return false even though the qemu supports it).
There is one problem, imagine this config:
- qemu_version: # hook
0:
filter-only: /use_devices/no
1:
2:
- use_devices: # hook
yes:
no:
- tests:
my_test:
mux: ["/use_devices"]
1) no multiplex, no problem
2) multiplex_test, no problem
3) multiplex related - in case use_devices is not available, the test
would either be executed twice for the same environment (hook overrides
yes to no), or it would skip the "yes" version or it would raise
exception that use_devices is not supported.
4) all multiplex - no problem, filter-only takes care of that.
In my opinion in this case the hook should just raise TestNAError() so
the test is skipped and we have no problem. Sometimes the other solution
based on the hook should apply. I was unable to find real problem, but
perhaps someone else could.
### Scheduler
while thinking about the variants, it struck me. Why not to use
multiplexer for sheduler too. At least as hinter. Because when you go
through the mux tree and you reach the hooks, you just need to ask the
hook what it requires for the value of the first, second, ... leaf. Then
you can add this into machine filter and schedule this particular
variant on machine, which supports all conditional branches.
So the only difference is, that the hooks should support booth ways.
1) get default value for this machine (and respecting the raw value)
2) return requirements for given leaf (eg. ppc64, arm, mem > 64G, ...)
### Summarize
There are 4 separate ideas described.
The first one is about different approaches of avocado users, which
should speed up the normal execution (for most users no tree would need
to be created or just very basic one).
The second is addressing the problem of multiplexation of abusing
filter-out to not multiplex non-parent leaves. By design it should
decrease number of multiplexations.
The third describes my view of special parameters handling. If we put
the hooks into single file (or split by parent namespaces) it should be
pretty readable and describes one potential problem.
The last one is a bit similar to the 3rd one, only from different
perspective. It tries to utilize the mux tree for scheduler.
_______________________________________________
Virt-test-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virt-test-devel
