Travis' patches for --pinnedpubkey let me take a closer look into the python test suite today.
I wondered why some tests PASSed though there were serious failures in the appropriate .log files. An exception raised in the post_hook did not make the test fail. Patch 0001-... handles this now. The reason for those exceptions was a copy&paste of the TEST_NAME value. This value is taken as test directory. And with a parallel test suite we suddenly have several tests running at the same time using the same directory. I already stumbled upon this in the past myself - and I am sure this will cost some work in future as well. Easily circumvented by using the file name of the test as test name. This is patch 0002-... just as suggestion, to complete it we have to remove all occurrences of TEST_NAME from the python tests. The fix/patch 0001-... brought up a broken test case (fails in post_hook) which is fixed in 0003-... Please review & comment Tim
From a99b89d4649cd55bb725efe8397d44325bf41454 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim Rühsen?= <[email protected]> Date: Fri, 8 Apr 2016 15:29:24 +0200 Subject: [PATCH 1/3] Fail python tests when post_hook errors * testenv/test/base_test.py (__exit__): Return self.tests_passed (__test_cleanup): Set self.tests_passed to False on exception --- testenv/test/base_test.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/testenv/test/base_test.py b/testenv/test/base_test.py index c5b82be..9bb4151 100644 --- a/testenv/test/base_test.py +++ b/testenv/test/base_test.py @@ -162,6 +162,7 @@ class BaseTest: shutil.rmtree(self.get_test_dir()) except: print("Unknown Exception while trying to remove Test Environment.") + self.tests_passed = False def _exit_test(self): self.__test_cleanup() @@ -263,4 +264,4 @@ class BaseTest: traceback.print_tb(exc_tb) self.__test_cleanup() - return True + return self.tests_passed -- 2.8.0.rc3
From 4c975b4d2e7d5c63d940ebf11dab15e846c97175 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim Rühsen?= <[email protected]> Date: Fri, 8 Apr 2016 16:31:08 +0200 Subject: [PATCH 2/3] Use test file name as TEST_NAME * testenv/test/base_test.py (__init__): Use test file name for name --- testenv/test/base_test.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/testenv/test/base_test.py b/testenv/test/base_test.py index 9bb4151..82310b1 100644 --- a/testenv/test/base_test.py +++ b/testenv/test/base_test.py @@ -4,6 +4,7 @@ import shlex import traceback import re import time +import sys from subprocess import call from misc.colour_terminal import print_red, print_blue from exc.test_failed import TestFailed @@ -27,7 +28,7 @@ class BaseTest: Define the class-wide variables (or attributes). Attributes should not be defined outside __init__. """ - self.name = name + self.name = os.path.basename(os.path.realpath(sys.argv[0])) # if pre_hook == None, then {} (an empty dict object) is passed to # self.pre_configs self.pre_configs = pre_hook or {} -- 2.8.0.rc3
From 5a0b424b8e5bfcc6fb1d44b4cfb465bdbff16a66 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tim Rühsen?= <[email protected]> Date: Fri, 8 Apr 2016 16:32:07 +0200 Subject: [PATCH 3/3] Fix testenv/Test--rejected-log.py * testenv/Test--rejected-log.py: Add missing tabs in expected output --- testenv/Test--rejected-log.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/testenv/Test--rejected-log.py b/testenv/Test--rejected-log.py index a94ae3b..03f62c2 100755 --- a/testenv/Test--rejected-log.py +++ b/testenv/Test--rejected-log.py @@ -55,10 +55,10 @@ Disallow: /dummy.txt """ log = """\ -REASON U_URL U_SCHEME U_HOST U_PORT U_PATH U_PARAMS U_QUERY U_FRAGMENT P_URL P_SCHEME P_HOST P_PORT P_PATH P_PARAMS P_QUERY P_FRAGMENT -BLACKLIST http%3A//127.0.0.1%3A{{port}}/index.html SCHEME_HTTP 127.0.0.1 {{port}} index.html http%3A//127.0.0.1%3A{{port}}/secondpage.html SCHEME_HTTP 127.0.0.1 {{port}} secondpage.html -ROBOTS http%3A//127.0.0.1%3A{{port}}/dummy.txt SCHEME_HTTP 127.0.0.1 {{port}} dummy.txt http%3A//127.0.0.1%3A{{port}}/thirdpage.html SCHEME_HTTP 127.0.0.1 {{port}} thirdpage.html -SPANNEDHOST http%3A//no.such.domain/ SCHEME_HTTP no.such.domain 80 http%3A//127.0.0.1%3A{{port}}/thirdpage.html SCHEME_HTTP 127.0.0.1 {{port}} thirdpage.html +REASON\tU_URL\tU_SCHEME\tU_HOST\tU_PORT\tU_PATH\tU_PARAMS\tU_QUERY\tU_FRAGMENT\tP_URL\tP_SCHEME\tP_HOST\tP_PORT\tP_PATH\tP_PARAMS\tP_QUERY\tP_FRAGMENT +BLACKLIST\thttp%3A//127.0.0.1%3A{{port}}/index.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tindex.html\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/secondpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tsecondpage.html\t\t\t +ROBOTS\thttp%3A//127.0.0.1%3A{{port}}/dummy.txt\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tdummy.txt\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t +SPANNEDHOST\thttp%3A//no.such.domain/\tSCHEME_HTTP\tno.such.domain\t80\t\t\t\t\thttp%3A//127.0.0.1%3A{{port}}/thirdpage.html\tSCHEME_HTTP\t127.0.0.1\t{{port}}\tthirdpage.html\t\t\t """ dummyfile = "Don't care." -- 2.8.0.rc3
signature.asc
Description: This is a digitally signed message part.
