On Fri, 2017-05-19 at 10:28 -0400, Cleber Rosa wrote: > > On 05/18/2017 01:07 AM, Satheesh Rajendran wrote: > > > > On Wed, 2017-05-17 at 17:49 -0400, Cleber Rosa wrote: > > > > > > Introduction > > > ============ > > > > > > Avocado allows users to select tests to run based on free form > > > "tags". > > > These tags are given as "docstring directives", that is, special > > > entries on a class or function docstring. > > > > > This would be very helpful and becomes easy way to > > create a dynamic testsuite based on needs. > > +1 > I'm not sure if it was clear enough, but this functionality already > exists in Avocado: > > http://avocado-framework.readthedocs.io/en/50.0/WritingTests.html#cat > egorizing-tests > > This proposal is about proposing a set of guidelines for using that > feature. > Oh Sure, need to explore on that, Thanks :-) > > > > > > > > As a user of an Avocado based test suite, I'd see value in not > > > **having** > > > to look at all the test tags before realizing that to not run > > > tests > > > that > > > require "super user" permissions I should run:: > > > > > > $ avocado run test.py --filter-by-tags=-root > > > > > > Instead of:: > > > > > > $ avocado run test.py --filter-by-tags=-privileged > > > > > > Not even that, by having different tests as part of the same job, > > > the following very odd sequence of command line options may be > > > needed:: > > > > > > $ avocado run test.py test2.py --filter-by-tags=-root,- > > > privileged > > > > > > So the goal here is to let users familiar with a given Avocado > > > based > > > test, to have fair expectations when running another Avocado > > > based > > > tests. > > > > > > This was initially going to be a documentation update, but I felt > > > that > > > it was not fair to make a formal proposal without without some > > > initial > > > brainstorming. > > > > > > Proposal > > > ======== > > > > > > To set the tone for my proposal, I'd like to make most things > > > simple > > > and easy, while allowing for "everything else" to be doable. > > > > > > My general impression is that higher level information about the > > > test > > > itself and its requirements are going to be the most commonly > > > used > > > tags, so they must be easily set. Some examples follow. > > > > > > Simple (standalone) tags > > > ------------------------ > > > > > > Tags by functional area: > > > > > > * cpu - Exercises a system's CPU > > > * net - Exercises a system's network devices or networking stack > > > * storage - Exercises a system's local storage > > > * fs - Exercises a system's file system > > > > > > Tags by architecture: > > > > > > * x86_64 - Requires a x86_64 architecture > > > * ppc64 - Requries a ppc64 > > > > > > Tags by access privileges: > > > > > > * privileged - requires the test to be run with the most > > > privileged, > > > unrestricted privileges. For Linux systems, this usually > > > means > > > the > > > root account > > > > > > Composed (key:value) tags > > > ------------------------- > > > > > > The more specific tags can be achieved by composing a predefined > > > key > > > with a value. For instance, to tag a test as needing a specific > > > CPU flag: > > > > > > * cpu_flag:vmx > > > > > > Or a specific PCI device: > > > > > > * pci_id:8086:08b2 > > > > > > Or even a software package: > > > > > > * package:gcc > > > > > > Or a package group altogether: > > > > > > * package_group:development-tools > > > > > > Some examples > > > ------------- > > > > > > * ``cpu,x86_64`` - The test exercises the CPU and requires a > > > ``x86_64`` based platform > > > > > > * ``net,privileged,pci_id:14e4:1657`` - The test exercises > > > either a > > > network device or the network stack, needs super user > > > privileges > > > and a "Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet > > > PCIe > > > (rev 01)" device. > > > > > > Looking at test tags > > > ==================== > > > > > > Currently, there's no way to actually list the test tags from the > > > command line, alongside the test name themselves. An RFE for > > > fixing > > > this has been filed at https://trello.com/c/NXrbKEJC . > > > > > > Do users have to provide all the ``--filter-by-tags`` themselves? > > > ================================================================= > > > > > > The test runner can certainly help here, getting system > > > information > > > when the job starts, and feeding them to the filtering. This is > > > yet > > > another reason why coming up with a good set of guidelines for > > > tagging > > > tests is important. > > > > > > In some ways, this can be seen similar to a dependency resolution > > > mechanism for tests, only that at this point it will not resolve > > > the > > > requirements. It will only filter out tests that can't (or > > > shouldn't) > > > be loaded on the current system. > > > > > > Effectively, instead of many in-tests checks, and many > > > SKIPs/CANCELs, > > > the system information can be loaded once, and the only relevant > > > tests > > > will be part of the tests suite. > > > > > > A list of the tests that were filtered out from the job test > > > suite > > > can > > > certainly be a useful part of the job results. > > > > > > As in every RFC, feedback is extremely welcome! > > > > > I assume this has to be at the overall testcase level, can not > > be applied to multiplexed params. > > for example: > > certain multiplexer options to be filtered in the same testcase. > > > > As the proposal is to use doc-strings, I assume it can not be done? > > > Exactly, it's done at the test level, before they're combined with > variants. > > I'm not quite sure that it can't be done, but definitely it's out of > the > scope of this specific RFC. > Sure, makes sense.
Regards, -Satheesh. > Cheers! >
