On Wed, Feb 14, 2018 at 2:09 PM, Brenda J. Butler <b...@mojatatu.com> wrote:
> Signed-off-by: Brenda J. Butler <b...@mojatatu.com>
Acked-by: Lucas Bates <luc...@mojatatu.com>
> tools/testing/selftests/tc-testing/README | 173
> tools/testing/selftests/tc-testing/TODO.txt | 25 +++-
> 2 files changed, 179 insertions(+), 19 deletions(-)
> diff --git a/tools/testing/selftests/tc-testing/README
> index 970ff294fec8..3a0336782d2d 100644
> --- a/tools/testing/selftests/tc-testing/README
> +++ b/tools/testing/selftests/tc-testing/README
> @@ -14,11 +14,11 @@ REQUIREMENTS
> * The kernel must have network namespace support
> -* The kernel must have veth support available, as a veth pair is created
> +* The kernel must have veth support available, as a veth pair is created
> prior to running the tests.
> -* All tc-related features must be built in or available as modules.
> - To check what is required in current setup run:
> +* All tc-related features being tested must be built in or available as
> + modules. To check what is required in current setup run:
> ./tdc.py -c
> @@ -44,10 +44,13 @@ using the -p option when running tdc:
> RUNNING TDC
> -To use tdc, root privileges are required. tdc will not run otherwise.
> +To use tdc, root privileges are required. This is because the
> +commands being tested must be run as root. The code that enforces
> +execution by root uid has been moved into a plugin (see PLUGIN
> +ARCHITECTURE, below).
> -All tests are executed inside a network namespace to prevent conflicts
> -within the host.
> +If nsPlugin is linked, all tests are executed inside a network
> +namespace to prevent conflicts within the host.
> Running tdc without any arguments will run all tests. Refer to the section
> on command line arguments for more information, or run:
> @@ -59,6 +62,33 @@ output captured from the failing test will be printed
> immediately following
> the failed test in the TAP output.
> +OVERVIEW OF TDC EXECUTION
> +One run of tests is considered a "test suite" (this will be refined in the
> +future). A test suite has one or more test cases in it.
> +A test case has four stages:
> + - setup
> + - execute
> + - verify
> + - teardown
> +The setup and teardown stages can run zero or more commands. The setup
> +stage does some setup if the test needs it. The teardown stage undoes
> +the setup and returns the system to a "neutral" state so any other test
> +can be run next. These two stages require any commands run to return
> +success, but do not otherwise verify the results.
> +The execute and verify stages each run one command. The execute stage
> +tests the return code against one or more acceptable values. The
> +verify stage checks the return code for success, and also compares
> +the stdout with a regular expression.
> +Each of the commands in any stage will run in a shell instance.
> USER-DEFINED CONSTANTS
> @@ -70,23 +100,132 @@ executed as part of the test. More will be added as
> test cases require.
> $TC qdisc add dev $DEV1 ingress
> +The NAMES values are used to substitute into the commands in the test cases.
> COMMAND LINE ARGUMENTS
> Run tdc.py -h to see the full list of available arguments.
> --p PATH Specify the tc executable located at PATH to be used on
> - test run
> --c Show the available test case categories in this test file
> --c CATEGORY Run only tests that belong to CATEGORY
> --f FILE Read test cases from the JSON file named FILE
> --l [CATEGORY] List all test cases in the JSON file. If CATEGORY is
> - specified, list test cases matching that category.
> --s ID Show the test case matching ID
> --e ID Execute the test case identified by ID
> --i Generate unique ID numbers for test cases with no existing
> - ID number
> +usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
> + [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]
> + [-d DEVICE] [-n NS] [-V]
> +Linux TC unit tests
> +optional arguments:
> + -h, --help show this help message and exit
> + -p PATH, --path PATH The full path to the tc executable to use
> + -v, --verbose Show the commands that are being run
> + -d DEVICE, --device DEVICE
> + Execute the test case in flower category
> + select which test cases: files plus directories; filtered by categories
> + plus testids
> + -D DIR [DIR ...], --directory DIR [DIR ...]
> + Collect tests from the specified directory(ies)
> + (default [tc-tests])
> + -f FILE [FILE ...], --file FILE [FILE ...]
> + Run tests from the specified file(s)
> + -c [CATG [CATG ...]], --category [CATG [CATG ...]]
> + Run tests only from the specified category/ies, or if
> + no category/ies is/are specified, list known
> + categories.
> + -e ID [ID ...], --execute ID [ID ...]
> + Execute the specified test cases with specified IDs
> + select action to perform on selected test cases
> + -l, --list List all test cases, or those only within the
> + specified category
> + -s, --show Display the selected test cases
> + -i, --id Generate ID numbers for new test cases
> + options for nsPlugin(run commands in net namespace)
> + -n NS, --namespace NS
> + Run commands in namespace NS
> + options for valgrindPlugin (run command under test under Valgrind)
> + -V, --valgrind Run commands under valgrind
> +PLUGIN ARCHITECTURE
> +There is now a plugin architecture, and some of the functionality that
> +was in the tdc.py script has been moved into the plugins.
> +The plugins are in the directory plugin-lib. The are executed from
> +directory plugins. Put symbolic links from plugins to plugin-lib,
> +and name them according to the order you want them to run.
> +bjb@bee:~/work/tc-testing$ ls -l plugins
> +total 4
> +lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py ->
> +lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py ->
> +-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
> +The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
> +must be called "SubPlugin" so tdc can find them. They are
> +distinguished from each other in the python program by their module
> +This base class supplies "hooks" to run extra functions. These hooks are as
> +pre- and post-suite
> +pre- and post-case
> +pre- and post-execute stage
> +adjust-command (runs in all stages and receives the stage name)
> +The pre-suite hook receives the number of tests and an array of test ids.
> +This allows you to dump out the list of skipped tests in the event of a
> +failure during setup or teardown stage.
> +The pre-case hook receives the ordinal number and test id of the current
> +The adjust-command hook receives the stage id (see list below) and the
> +full command to be executed. This allows for last-minute adjustment
> +of the command.
> +The stages are identified by the following strings:
> + - pre (pre-suite)
> + - setup
> + - command
> + - verify
> + - teardown
> + - post (post-suite)
> +To write a plugin, you need to inherit from TdcPlugin in
> +TdcPlugin.py. To use the plugin, you have to put the
> +implementation file in plugin-lib, and add a symbolic link to it from
> +plugins. It will be detected at run time and invoked at the
> +appropriate times. There are a few examples in the plugin-lib
> + - rootPlugin.py:
> + implements the enforcement of running as root
> + - nsPlugin.py:
> + sets up a network namespace and runs all commands in that namespace
> + - valgrindPlugin.py
> + runs each command in the execute stage under valgrind,
> + and checks for leaks.
> + This plugin will output an extra test for each test in the test file,
> + one is the existing output as to whether the test passed or failed,
> + and the other is a test whether the command leaked memory or not.
> + (This one is a preliminary version, it may not work quite right yet,
> + but the overall template is there and it should only need tweaks.)
> diff --git a/tools/testing/selftests/tc-testing/TODO.txt
> index 6a266d811a78..c40698557e2f 100644
> --- a/tools/testing/selftests/tc-testing/TODO.txt
> +++ b/tools/testing/selftests/tc-testing/TODO.txt
> @@ -5,6 +5,27 @@ tc Testing Suite To-Do list:
> - Add support for multiple versions of tc to run successively
> -- Improve error messages when tdc aborts its run
> +- Improve error messages when tdc aborts its run. Partially done - still
> + need to better handle problems in pre- and post-suite.
> -- Allow tdc to write its results to file
> +- Use python logger module for debug/verbose output
> +- Allow tdc to write its results to file.
> + Maybe use python logger module for this too.
> +- A better implementation of the "hooks". Currently, every plugin
> + will attempt to run a function at every hook point. Could be
> + changed so that plugin __init__ methods will register functions to
> + be run in the various predefined times. Then if a plugin does not
> + require action at a specific point, no penalty will be paid for
> + trying to run a function that will do nothing.
> +- Proper exception handling - make an exception class and use it
> +- a TestCase class, for easier testcase handling, searching, comparison
> +- a TestSuite class
> + and a way to configure a test suite,
> + to automate running multiple "test suites" with different requirements
> +- super simple test case example using ls, touch, etc