On Wed, 2017-05-17 at 17:49 -0400, Cleber Rosa wrote: > Introduction > ============ > > Avocado allows users to select tests to run based on free form > "tags". > These tags are given as "docstring directives", that is, special > entries on a class or function docstring. > This would be very helpful and becomes easy way to create a dynamic testsuite based on needs. +1 > As a user of an Avocado based test suite, I'd see value in not > **having** > to look at all the test tags before realizing that to not run tests > that > require "super user" permissions I should run:: > > $ avocado run test.py --filter-by-tags=-root > > Instead of:: > > $ avocado run test.py --filter-by-tags=-privileged > > Not even that, by having different tests as part of the same job, > the following very odd sequence of command line options may be > needed:: > > $ avocado run test.py test2.py --filter-by-tags=-root,-privileged > > So the goal here is to let users familiar with a given Avocado based > test, to have fair expectations when running another Avocado based > tests. > > This was initially going to be a documentation update, but I felt > that > it was not fair to make a formal proposal without without some > initial > brainstorming. > > Proposal > ======== > > To set the tone for my proposal, I'd like to make most things simple > and easy, while allowing for "everything else" to be doable. > > My general impression is that higher level information about the test > itself and its requirements are going to be the most commonly used > tags, so they must be easily set. Some examples follow. > > Simple (standalone) tags > ------------------------ > > Tags by functional area: > > * cpu - Exercises a system's CPU > * net - Exercises a system's network devices or networking stack > * storage - Exercises a system's local storage > * fs - Exercises a system's file system > > Tags by architecture: > > * x86_64 - Requires a x86_64 architecture > * ppc64 - Requries a ppc64 > > Tags by access privileges: > > * privileged - requires the test to be run with the most privileged, > unrestricted privileges. For Linux systems, this usually means > the > root account > > Composed (key:value) tags > ------------------------- > > The more specific tags can be achieved by composing a predefined key > with a value. For instance, to tag a test as needing a specific > CPU flag: > > * cpu_flag:vmx > > Or a specific PCI device: > > * pci_id:8086:08b2 > > Or even a software package: > > * package:gcc > > Or a package group altogether: > > * package_group:development-tools > > Some examples > ------------- > > * ``cpu,x86_64`` - The test exercises the CPU and requires a > ``x86_64`` based platform > > * ``net,privileged,pci_id:14e4:1657`` - The test exercises either a > network device or the network stack, needs super user privileges > and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe > (rev 01)" device. > > Looking at test tags > ==================== > > Currently, there's no way to actually list the test tags from the > command line, alongside the test name themselves. An RFE for fixing > this has been filed at https://trello.com/c/NXrbKEJC . > > Do users have to provide all the ``--filter-by-tags`` themselves? > ================================================================= > > The test runner can certainly help here, getting system information > when the job starts, and feeding them to the filtering. This is yet > another reason why coming up with a good set of guidelines for > tagging > tests is important. > > In some ways, this can be seen similar to a dependency resolution > mechanism for tests, only that at this point it will not resolve the > requirements. It will only filter out tests that can't (or > shouldn't) > be loaded on the current system. > > Effectively, instead of many in-tests checks, and many SKIPs/CANCELs, > the system information can be loaded once, and the only relevant > tests > will be part of the tests suite. > > A list of the tests that were filtered out from the job test suite > can > certainly be a useful part of the job results. > > As in every RFC, feedback is extremely welcome! >
I assume this has to be at the overall testcase level, can not be applied to multiplexed params. for example: certain multiplexer options to be filtered in the same testcase. As the proposal is to use doc-strings, I assume it can not be done? >
