Dne 6.8.2018 v 09:52 Sergey Bronnikov napsal(a): > пт, 3 авг. 2018 г. в 14:55, Lukáš Doktor <ldok...@redhat.com>: > >> Well the execution is actually simpler than you think, `avocado run >> basic.yaml` should does the trick. Anyway the yaml-file is slightly >> corrupted and even then some things won't probably result in what you >> expect. Let me explain it a bit and provide better example) >> >> Dne 3.8.2018 v 10:20 Sergey Bronnikov napsal(a): >>> Lukas, thanks for explanation! But I still have a question. >>> >>> Imagine we have an YAML config with a list of tests and their parameters: >>> >>> ```yaml >>> basic: !mux >>> >>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader >>> timeout: 360 >>> test_reference_resolver_args: !!python/dict >>> vt_extra_params: >>> - nic_model=rtl8139 >>> - extra_params=" -cpu host" >>> - drive_format=ahci >>> - mem=2048 >>> vt-machine-type: q35 >> >> this should be "vt_machine_type" >> >>> >>> shutdown: >>> test_reference: io-github-autotest-qemu.shutdown >> >> Here `test_reference_resolver_extra:` is missing. >> >>> vt_extra_params: >> >> this should be `avocado_vt_extra_params:` (see `avocado_vt/loader.py`) >> >>> - shutdown_method = shell >>> >>> reboot: >>> test_reference: io-github-autotest-qemu.reboot >> >> Here `test_reference_resolver_extra:` is missing. >> >>> vt_extra_params: >>> - param1=1 >>> - param2=2 >>> >>> stress: >>> test_reference: io-github-autotest-qemu.linux_stress >>> >>> iofuzz: >>> test_reference: io-github-autotest-qemu.iofuzz >> >> Here `test_reference_resolver_extra:` is missing. >> >>> vt_extra_params: >>> - param1=1 >>> - param2=2 >>> >>> ``` >>> >>> How to execute tests from this testplan? >> >> as mentioned earlier, it's recognized as test format, therefor simply >> `avocado run basic.yaml`. >> >> Also one thing, the vt discovery is quite slow (yeah we parse cartesian >> config on each invocation as it might have different values). You can put >> multiple `test_reference` into the same line and separate them by space in >> avocado-vt loader. But beware, it works as filters, therefor the discovered >> tests will be re-oredered accordingly to cartesian config (rather than how >> you put them there) and they'll occur only once. Therefor: >> >> test_reference: shutdown boot boot >> >> will generate: >> >> avocado list basic.yaml >> VT io-github-autotest-qemu.boot >> VT io-github-autotest-qemu.shutdown >> >> Anyway the speed of that is incomparably faster, which is why I use it >> when possible. >> > > > Finally with corrections I got a following testplan:
Dear Sergey, actually you unveiled a bug; fix can be found here: https://github.com/avocado-framework/avocado-vt/pull/1696 (in my CI I always use `--vt-extra-params` so I forgot it might not be set. Sorry about this). With this there is still one little remaining issue in your `basic.yaml`, see below... > > basic: !mux > test_reference_resolver_class: avocado_vt.loader.VirtTestLoader > timeout: 360 > test_reference_resolver_extra: !!python/dict This is supposed to be "test_reference_resolver_args" (you're modifying the `self.args` of Avocado for the purpose of this loader. Note this only affects things inside the loader, so you can't for example change the job-results-dir or so...) > vt_extra_params: > - mem=2048 > - nic_model=rtl8139 > - drive_format=ahci > - disk_bus=ahci > - extra_params=" -cpu host" > vt_machine_type: q35 > > shutdown: > test_reference: io-github-autotest-qemu.shutdown > test_reference_resolver_extra: !!python/dict > avocado_vt_extra_params: > - shutdown_method = shell > > reboot: > test_reference: io-github-autotest-qemu.reboot > test_reference_resolver_extra: !!python/dict > avocado_vt_extra_params: > - param1 = 1 > - param2 = 2 > > stress: > test_reference: io-github-autotest-qemu.linux_stress > > iofuzz: > test_reference: io-github-autotest-qemu.iofuzz > test_reference_resolver_extra: !!python/dict > avocado_vt_extra_params: > - param1 = 1 > - param2 = 2 > > ~$ avocado variants -m testplans/basic.yaml > Multiplex variants (4): > Variant shutdown-edec: /run/basic/shutdown > Variant reboot-c8ba: /run/basic/reboot > Variant stress-fd6a: /run/basic/stress > Variant iofuzz-e8bc: /run/basic/iofuzz > ~$ > > But it is still not accepted by 'avocado {list, run}': > ~$ avocado list testplans/basic.yaml > Unable to resolve reference(s) 'testplans/basic.yaml' with plugins(s) > 'file', 'vt', 'external', try running 'avocado list -V > testplans/basic.yaml' to see the details. > > Do you have any idea why it may happen? With that single change it works well for me (tried changing the "test_reference_resolver_args" and first test's "test_reference_resolver_extra" and it successfully propagated to test params). Regards, Lukáš >> autotest==0.16.4 > avocado-framework==63.0 > avocado-framework-plugin-varianter-yaml-to-mux==63.0 > avocado-plugins-vt==63.0 > > Sergey > > >> >> Regards, >> Lukáš >> >> PS: If this happens to work for you, would you please write an Avocado-vt >> documentation about this? I'd prefer someone external doing so because >> (apparently) I take many things as given... (you might as well improve the >> Avocado documentation, but looking at this I think the Avocado-vt one is >> missing more) >> >>> >>> $ avocado run -m basic.yaml >>> No test references provided nor any other arguments resolved into tests. >>> Please double check the executed command. >>> >>> >>> Sergey >>> >> >> > > >>> >>> чт, 2 авг. 2018 г. в 17:08, Lukáš Doktor <ldok...@redhat.com>: >>> >>>> Dne 1.8.2018 v 13:19 Sergey Bronnikov napsal(a): >>>>> Hello! >>>>> >>>>> I'm trying to find a way to organize my tests for running without >>>>> specifying all of them in command-line. Imagine we have a set of >>>> different >>>>> tests: sleep.py, passtest.py and fail.py. To run all of them with >>>> required >>>>> parameters (aka variants) we may execute a command: >>>>> avocado run sleep.py fail.py passtest.py -m basic.yaml >>>>> >>>>> Is there a way to run the same tests without explicit specifying them? >>>>> For example: avocado run -m basic.yaml >>>>> >>>>> Sergey >>>>> >>>> >>>> Dear Sergey, >>>> >>>> as a matter of fact, there are multiple ways. >>>> >>>> 1. filter-by-tag: >>>> >> http://avocado-framework.readthedocs.io/en/latest/WritingTests.html#categorizing-tests >>>> >>>> you can tag your tests eg. `sanity`, `release`, ... and then run >>>> `avocado run --filter-by-tag=sanity -- TEST_REPO` to only run tests >> tagged >>>> `sanity` (note the granularity is per-class or even per-test-method. >>>> >>>> 2. yaml loader optional plugin: >>>> >> http://avocado-framework.readthedocs.io/en/latest/optional_plugins/yaml_loader.html >>>> >>>> which is probably more what you're looking for, but currently only >>>> works on python2 (but should get py3 support, hopefully, soon). It works >>>> similarly to yaml_to_mux varianter plugin, but allows special keys like >>>> `test_reference` to specify what test will be used or even >>>> `test_reference_resolver_class` when your tests are not the default >> ones. >>>> This is for example useful when you want to include multiple >>>> external-runner tests with different external-runner. There are examples >>>> in: >>>> >> https://github.com/avocado-framework/avocado/tree/master/examples/yaml_to_mux_loader >>>> but let me share my slighly advanced example here: >>>> >>>> ```yaml >>>> # Generated s390x mux-suite >>>> !mux >>>> qemu_make: !mux >>>> mux_suite_test_name_prefix: 'qemu_make/git ' >>>> check: >>>> test_reference: >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-check >>>> install: >>>> test_reference: >>>> /home/jenkins/s390x/qemu-master/avocado-qemu-make-install >>>> >>>> unit: >>>> variants: !mux >>>> test_reference_resolver_class: >> avocado.core.loader.ExternalLoader >>>> timeout: 360 >>>> rpm-tcg: >>>> mux_suite_test_name_prefix: 'unit/rpm/tcg ' >>>> test_reference_resolver_extra: !!python/dict >>>> loader_options: >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-tcg >>>> rpm-kvm: >>>> mux_suite_test_name_prefix: 'unit/rpm/kvm ' >>>> test_reference_resolver_extra: !!python/dict >>>> loader_options: >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-rpm-kvm >>>> git-tcg: >>>> mux_suite_test_name_prefix: 'unit/git/tcg ' >>>> test_reference_resolver_extra: !!python/dict >>>> loader_options: >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-tcg >>>> git-kvm: >>>> mux_suite_test_name_prefix: 'unit/git/kvm ' >>>> test_reference_resolver_extra: !!python/dict >>>> loader_options: >>>> /home/jenkins/s390x/kvm-unit-test/avocado-runner-git-kvm >>>> tests: !mux >>>> cmm: >>>> test_reference: cmm >>>> diag10: >>>> test_reference: diag10 >>>> emulator: >>>> test_reference: emulator >>>> gs: >>>> test_reference: gs >>>> iep: >>>> test_reference: iep >>>> intercept: >>>> test_reference: intercept >>>> pfmf: >>>> test_reference: pfmf >>>> selftest-setup: >>>> test_reference: selftest-setup >>>> sieve: >>>> test_reference: sieve >>>> skey: >>>> test_reference: skey >>>> sthyi: >>>> test_reference: sthyi >>>> vector: >>>> test_reference: vector >>>> functional: !mux >>>> test_reference_resolver_class: avocado_vt.loader.VirtTestLoader >>>> s390x-rpm-libvirt/dvd: >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/dvd ' >>>> test_reference: >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads >>>> io-github-autotest-qemu.boot remove_guest.without_disk >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_type: libvirt >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - vga=none >>>> s390x-rpm-libvirt/url: >>>> mux_suite_test_name_prefix: 'functional/rpm/libvirt/url ' >>>> test_reference: >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads >>>> io-github-autotest-qemu.boot remove_guest.without_disk >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_type: libvirt >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - vga=none >>>> s390x-rpm-dvd: >>>> mux_suite_test_name_prefix: 'functional/rpm/dvd ' >>>> test_reference: >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>> migrate.default.tcp.with_post_copy >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - ping_pong=5 >>>> - vga=none >>>> s390x-rpm-url: >>>> mux_suite_test_name_prefix: 'functional/rpm/url ' >>>> test_reference: >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>> migrate.default.tcp.with_post_copy >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_dst_qemu_bin: /usr/libexec/qemu-kvm >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - ping_pong=5 >>>> - vga=none >>>> s390x-git-dvd: >>>> mux_suite_test_name_prefix: 'functional/git/dvd ' >>>> test_reference: >>>> unattended_install.cdrom.extra_cdrom_ks.default_install.aio_threads boot >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>> migrate.default.tcp.with_post_copy >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>> vt_dst_qemu_bin: >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - ping_pong=5 >>>> - vga=none >>>> s390x-git-url: >>>> mux_suite_test_name_prefix: 'functional/git/url ' >>>> test_reference: >>>> unattended_install.url.extra_cdrom_ks.default_install.aio_threads boot >>>> migrate.with_reboot.tcp migrate.with_reboot.exec.gzip_exec >>>> migrate.default.tcp.with_post_copy >>>> test_reference_resolver_args: !!python/dict >>>> vt_qemu_bin: >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>> vt_dst_qemu_bin: >>>> /home/jenkins/s390x/qemu-master/build/bin/qemu-system-s390x >>>> vt_extra_params: >>>> - automem=no >>>> - url=$URL >>>> - ping_pong=5 >>>> - vga=none >>>> ``` >>>> >>>> (feel free to ask for details, but basically it uses SIMPLE tests, >>>> EXTERNAL_RUNNER tests and then Avocado-vt tests. The test always >> receives >>>> all the params so you can override the standard arguments. If you >> specify >>>> "test_reference_resolver_args" it also updates the loader parameters, >> which >>>> is what I use for Avocado-vt tests). >>>> >>>> Theoretically there is yet another way and that is to create directory >>>> "sanity" and put symlinks to actual tests there, but the granularity >> there >>>> would be only per-test-class, which usually does not suffice. >>>> >>>> Anyway, hopefully at least one method suits you, but feel free to >>>> elaborate more, we can always improve. >>>> >>>> Kind regards, >>>> Lukáš >>>> >>>> >>> >> >> >> >
signature.asc
Description: OpenPGP digital signature