Signed-off-by: Brenda J. Butler <b...@mojatatu.com>
tools/testing/selftests/tc-testing/README | 173 +++++++++++++++++++++++++---
tools/testing/selftests/tc-testing/TODO.txt | 25 +++-
2 files changed, 179 insertions(+), 19 deletions(-)
diff --git a/tools/testing/selftests/tc-testing/README
index 970ff294fec8..3a0336782d2d 100644
@@ -14,11 +14,11 @@ REQUIREMENTS
* The kernel must have network namespace support
-* The kernel must have veth support available, as a veth pair is created
+* The kernel must have veth support available, as a veth pair is created
prior to running the tests.
-* All tc-related features must be built in or available as modules.
- To check what is required in current setup run:
+* All tc-related features being tested must be built in or available as
+ modules. To check what is required in current setup run:
@@ -44,10 +44,13 @@ using the -p option when running tdc:
-To use tdc, root privileges are required. tdc will not run otherwise.
+To use tdc, root privileges are required. This is because the
+commands being tested must be run as root. The code that enforces
+execution by root uid has been moved into a plugin (see PLUGIN
-All tests are executed inside a network namespace to prevent conflicts
-within the host.
+If nsPlugin is linked, all tests are executed inside a network
+namespace to prevent conflicts within the host.
Running tdc without any arguments will run all tests. Refer to the section
on command line arguments for more information, or run:
@@ -59,6 +62,33 @@ output captured from the failing test will be printed
the failed test in the TAP output.
+OVERVIEW OF TDC EXECUTION
+One run of tests is considered a "test suite" (this will be refined in the
+future). A test suite has one or more test cases in it.
+A test case has four stages:
+ - setup
+ - execute
+ - verify
+ - teardown
+The setup and teardown stages can run zero or more commands. The setup
+stage does some setup if the test needs it. The teardown stage undoes
+the setup and returns the system to a "neutral" state so any other test
+can be run next. These two stages require any commands run to return
+success, but do not otherwise verify the results.
+The execute and verify stages each run one command. The execute stage
+tests the return code against one or more acceptable values. The
+verify stage checks the return code for success, and also compares
+the stdout with a regular expression.
+Each of the commands in any stage will run in a shell instance.
@@ -70,23 +100,132 @@ executed as part of the test. More will be added as test
$TC qdisc add dev $DEV1 ingress
+The NAMES values are used to substitute into the commands in the test cases.
COMMAND LINE ARGUMENTS
Run tdc.py -h to see the full list of available arguments.
--p PATH Specify the tc executable located at PATH to be used on this
- test run
--c Show the available test case categories in this test file
--c CATEGORY Run only tests that belong to CATEGORY
--f FILE Read test cases from the JSON file named FILE
--l [CATEGORY] List all test cases in the JSON file. If CATEGORY is
- specified, list test cases matching that category.
--s ID Show the test case matching ID
--e ID Execute the test case identified by ID
--i Generate unique ID numbers for test cases with no existing
- ID number
+usage: tdc.py [-h] [-p PATH] [-D DIR [DIR ...]] [-f FILE [FILE ...]]
+ [-c [CATG [CATG ...]]] [-e ID [ID ...]] [-l] [-s] [-i] [-v]
+ [-d DEVICE] [-n NS] [-V]
+Linux TC unit tests
+ -h, --help show this help message and exit
+ -p PATH, --path PATH The full path to the tc executable to use
+ -v, --verbose Show the commands that are being run
+ -d DEVICE, --device DEVICE
+ Execute the test case in flower category
+ select which test cases: files plus directories; filtered by categories
+ plus testids
+ -D DIR [DIR ...], --directory DIR [DIR ...]
+ Collect tests from the specified directory(ies)
+ (default [tc-tests])
+ -f FILE [FILE ...], --file FILE [FILE ...]
+ Run tests from the specified file(s)
+ -c [CATG [CATG ...]], --category [CATG [CATG ...]]
+ Run tests only from the specified category/ies, or if
+ no category/ies is/are specified, list known
+ -e ID [ID ...], --execute ID [ID ...]
+ Execute the specified test cases with specified IDs
+ select action to perform on selected test cases
+ -l, --list List all test cases, or those only within the
+ specified category
+ -s, --show Display the selected test cases
+ -i, --id Generate ID numbers for new test cases
+ options for nsPlugin(run commands in net namespace)
+ -n NS, --namespace NS
+ Run commands in namespace NS
+ options for valgrindPlugin (run command under test under Valgrind)
+ -V, --valgrind Run commands under valgrind
+There is now a plugin architecture, and some of the functionality that
+was in the tdc.py script has been moved into the plugins.
+The plugins are in the directory plugin-lib. The are executed from
+directory plugins. Put symbolic links from plugins to plugin-lib,
+and name them according to the order you want them to run.
+bjb@bee:~/work/tc-testing$ ls -l plugins
+lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py ->
+lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py ->
+-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
+The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
+must be called "SubPlugin" so tdc can find them. They are
+distinguished from each other in the python program by their module
+This base class supplies "hooks" to run extra functions. These hooks are as
+pre- and post-suite
+pre- and post-case
+pre- and post-execute stage
+adjust-command (runs in all stages and receives the stage name)
+The pre-suite hook receives the number of tests and an array of test ids.
+This allows you to dump out the list of skipped tests in the event of a
+failure during setup or teardown stage.
+The pre-case hook receives the ordinal number and test id of the current test.
+The adjust-command hook receives the stage id (see list below) and the
+full command to be executed. This allows for last-minute adjustment
+of the command.
+The stages are identified by the following strings:
+ - pre (pre-suite)
+ - setup
+ - command
+ - verify
+ - teardown
+ - post (post-suite)
+To write a plugin, you need to inherit from TdcPlugin in
+TdcPlugin.py. To use the plugin, you have to put the
+implementation file in plugin-lib, and add a symbolic link to it from
+plugins. It will be detected at run time and invoked at the
+appropriate times. There are a few examples in the plugin-lib
+ - rootPlugin.py:
+ implements the enforcement of running as root
+ - nsPlugin.py:
+ sets up a network namespace and runs all commands in that namespace
+ - valgrindPlugin.py
+ runs each command in the execute stage under valgrind,
+ and checks for leaks.
+ This plugin will output an extra test for each test in the test file,
+ one is the existing output as to whether the test passed or failed,
+ and the other is a test whether the command leaked memory or not.
+ (This one is a preliminary version, it may not work quite right yet,
+ but the overall template is there and it should only need tweaks.)
diff --git a/tools/testing/selftests/tc-testing/TODO.txt
index 6a266d811a78..c40698557e2f 100644
@@ -5,6 +5,27 @@ tc Testing Suite To-Do list:
- Add support for multiple versions of tc to run successively
-- Improve error messages when tdc aborts its run
+- Improve error messages when tdc aborts its run. Partially done - still
+ need to better handle problems in pre- and post-suite.
-- Allow tdc to write its results to file
+- Use python logger module for debug/verbose output
+- Allow tdc to write its results to file.
+ Maybe use python logger module for this too.
+- A better implementation of the "hooks". Currently, every plugin
+ will attempt to run a function at every hook point. Could be
+ changed so that plugin __init__ methods will register functions to
+ be run in the various predefined times. Then if a plugin does not
+ require action at a specific point, no penalty will be paid for
+ trying to run a function that will do nothing.
+- Proper exception handling - make an exception class and use it
+- a TestCase class, for easier testcase handling, searching, comparison
+- a TestSuite class
+ and a way to configure a test suite,
+ to automate running multiple "test suites" with different requirements
+- super simple test case example using ls, touch, etc