[PATCH net-next] tc-testing: very simple example test cases

2017-10-31 Thread Brenda J. Butler
to test framework ok 2 2f simple test, no need for verify ok 3 3f simple test, no need for setup or teardown (or verify) $ Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- file AddingTestCases.txt clarification of what is expected/allowed in setup/teardown file exampl

[PATCH net-next] tc-testing: gitignore, ignore standard python artifacts

2017-10-31 Thread Brenda J. Butler
Ignore .pyc files, "python compiled" files, that get created when a python script is run. They should never be committed. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/.gitignore | 1 + 1 file changed, 1 insertion(+) diff --git

[PATCH net-next] tc-testing: correction to docstring in get_unique_item

2017-10-31 Thread Brenda J. Butler
Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc_helper.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/tc-testing/tdc_helper.py b/tools/testing/selftests/tc-testing/tdc_helper.py index c3254f

[PATCH net-next] tc-testing: split config file

2017-10-31 Thread Brenda J. Butler
environment for testing the test framework and/or the core tests. Also it makes it harder for any custom config to be submitted back to the kernel tdc. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc_config.py | 14 + .../tc-t

[PATCH net-next] tc-testing: better test case file error reporting

2017-10-31 Thread Brenda J. Butler
-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 23 ++- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index 550873

[PATCH net-next] tc-testing: better check if thing is list

2017-10-31 Thread Brenda J. Butler
-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc_helper.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/tc-testing/tdc_helper.py b/tools/testing/selftests/tc-testing/tdc_helper.py index c3254f861fb2..b4fbb0ce0cbe

[PATCH net-next] tc-testing: "show" action shows all selected tests

2017-10-31 Thread Brenda J. Butler
e test can be displayed with one command. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py| 7 --- tools/testing/selftests/tc-testing/tdc_helper.py | 12 +++- 2 files changed, 7 insertions(+), 12 deletions(-) diff --git

[PATCH net-next] tc-testing: correction to docstring in get_unique_item

2017-10-30 Thread Brenda J. Butler
The docstring says the function returns a "set" but it returns a "list". These are both python data types, so we should refer to the right one that is being returned. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- This patch was formerly sent as part of a 15-

[PATCH net] tc-testing: fix arg to ip command: -s -> -n

2017-10-30 Thread Brenda J. Butler
Fixes: 31c2611b66e0 ("selftests: Introduce a new test case to tc testsuite") Fixes: 76b903ee198d ("selftests: Introduce tc testsuite") Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- We want to run the command inside the container, not in the host. tools/te

[PATCH net-next 3/7] tools: tc-testing: Introduce plugin architecture

2018-02-14 Thread Brenda J. Butler
unctionality out of the tdc.py script and place it in plugins. To use the plugins, place the implementation in the plugins directory and run tdc.py. It will notice the plugins and use them. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/TdcPlugin

[PATCH net-next 7/7] tools: tc-testing: Update README and TODO

2018-02-14 Thread Brenda J. Butler
Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/README | 173 +--- tools/testing/selftests/tc-testing/TODO.txt | 25 +++- 2 files changed, 179 insertions(+), 19 deletions(-) diff --git a/tools/testing/selftests/tc-t

[PATCH net-next 2/7] tools: tc-testing: Refactor test-runner

2018-02-14 Thread Brenda J. Butler
Split the test_runner function into the loop part (test_runner) and the contents (run_one_test) for maintainability. It makes it a little easier to catch exceptions in an individual test, and keep going (and flush a bunch of tap results for the skipped tests). Signed-off-by: Brenda J. Butler &l

[PATCH net-next 6/7] tools: tc-testing: valgrindPlugin

2018-02-14 Thread Brenda J. Butler
Run the command under test under valgrind. Produce an extra set of tap output for the memory check on each test. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- .../tc-testing/plugin-lib/valgrindPlugin.py| 142 + 1 file changed, 142 insertions(+) creat

[PATCH net-next 1/7] tools: tc-testing: Command line parms

2018-02-14 Thread Brenda J. Butler
being constrained to run all tests in one collection or just one test. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- .../creating-testcases/AddingTestCases.txt | 35 +++- tools/testing/selftests/tc-testing/tdc.py | 209 + tools/testing/se

[PATCH net-next 0/7] tools: tc-testing: Plugin Architecture

2018-02-14 Thread Brenda J. Butler
To make tdc.py more general, we are introducing a plugin architecture. This patch set first organizes the command line parameters, then introduces the plugin architecture and some example plugins. Brenda J. Butler (7): tools: tc-testing: Command line parms tools: tc-testing: Refactor test

[PATCH net-next 4/7] tools: tc-testing: rootPlugin

2018-02-14 Thread Brenda J. Butler
Move the functionality that checks for root permissions into a plugin. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- .../selftests/tc-testing/plugin-lib/rootPlugin.py | 19 +++ tools/testing/selftests/tc-testing/tdc.py | 4 2 files chang

[PATCH net-next 5/7] tools: tc-testing: nsPlugin

2018-02-14 Thread Brenda J. Butler
Move the functionality of creating a namespace before the test suite and destroying it afterwards to a plugin. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- .../selftests/tc-testing/plugin-lib/nsPlugin.py| 141 + tools/testing/selftests/tc-testing/

[PATCH v2 net-next 2/2] tools: tc-testing: better error reporting

2018-02-23 Thread Brenda J. Butler
Do a better job with error handling - in pre- and post-suite, in pre- and post-case. Show a traceback for errors. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 84 +-- 1 file changed, 70 insertions(

[PATCH v2 net-next 0/2] tools: tc-testing: better error reporting

2018-02-23 Thread Brenda J. Butler
This patch set contains a bit of cleanup and better error reporting, esp. in pre- and post-suite, and pre- and post-case commands. Brenda J. Butler (2): tools: tc-testing: Fix indentation tools: tc-testing: better error reporting tools/testing/selftests/tc-testing/tdc.py | 88

[PATCH v2 net-next 1/2] tools: tc-testing: Fix indentation

2018-02-23 Thread Brenda J. Butler
Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index b3754b9aa302..ab28373dccd7

[PATCH net-next 0/2] tools: tc-testing: better error reporting

2018-02-23 Thread Brenda J. Butler
This patch set contains a bit of cleanup and better error reporting, esp. in pre- and post-suite, and pre- and post-case commands. Brenda J. Butler (2): tools: tc-testing: whitespace and .gitignore tools: tc-testing: better error reporting tools/testing/selftests/tc-testing/.gitignore | 1

[PATCH net-next 1/2] tools: tc-testing: whitespace and .gitignore

2018-02-23 Thread Brenda J. Butler
Fix indentation and add editor backup files to .gitignore Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/.gitignore | 1 + tools/testing/selftests/tc-testing/tdc.py | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a

[PATCH net-next 2/2] tools: tc-testing: better error reporting

2018-02-23 Thread Brenda J. Butler
Do a better job with error handling - in pre- and post-suite, in pre- and post-case. Show a traceback for errors. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 84 +-- 1 file changed, 70 insertions(

[PATCH v2 net-next 1/1] tools: tc-testing: Add notap option

2018-02-28 Thread Brenda J. Butler
Add a command line arg to suppress tap output. Handy in case all the tap output is being supplied by the plugins. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- v2: Drop the first patch that changes the format of the tap output. The second "notap" patch is r

[PATCH net-next 0/2] tools: tc-testing: notap option

2018-02-27 Thread Brenda J. Butler
When all the test results are being supplied by plugins, we may not want to see any test results for the command under test. So make a "no-tap" option, to suppress test result output for the command under test. Brenda J. Butler (2): tools: tc-testing: Better tap output tools: tc-te

[PATCH net-next 1/2] tools: tc-testing: Better tap output

2018-02-27 Thread Brenda J. Butler
Make it easier for automated test platforms to parse the test results. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/plugin-lib/valgrindPlugin.py | 2 +- tools/testing/selftests/tc-testing/tdc.py | 6 +++--- 2 files chan

[PATCH net-next 2/2] tools: tc-testing: Add notap option

2018-02-27 Thread Brenda J. Butler
Add a command line arg to suppress tap output. Handy in case all the tap output is being supplied by the plugins. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 36 +-- 1 file changed, 25 insertions(

[PATCH v2 net-next] tools: tc-testing: Can pause just before post-suite

2018-03-08 Thread Brenda J. Butler
With option -P, the test script will pause just before the post_suite functions are called. This allows the tester to inspect the system before it is torn down. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 9 + 1 file chan

[PATCH v2 net-next] tools: tc-testing: Can refer to $TESTID in test spec

2018-03-08 Thread Brenda J. Butler
in the test case: "cmdUnderTest": "some test command | tee -a $TESTID.out" Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 6 ++ 1 file changed, 6 insertions(+) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/

[PATCH net-next 1/1] tools: tc-testing: Can pause just before post-suite

2018-03-07 Thread Brenda J. Butler
At debug level 5 or above, the test script will pause just before the post_suite functions are called. This allows the tester to inspect the system before it is torn down. Signed-off-by: Brenda J. Butler <b...@mojatatu.com> --- tools/testing/selftests/tc-testing/tdc.py | 7 +++

[PATCH net-next 1/1] tools: tc-testing: Can refer to $TESTID in test spec

2018-03-07 Thread Brenda J. Butler
. For example, the testid can be given as a command line argument. As an example, if we wish to save the test output to a file named for the test case, we can write in the test case: "cmdUnderTest": "some test command | tee -a $TESTID.out" Signed-off-by: Brenda J. Butle