[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#9). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py A src/osmo_gsm_tester/report.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 290 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/9 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..16342c5 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import _prep -from osmo_gsm_tester import log, suite, config +from osmo_gsm_tester import log, suite, config, report config.ENV_CONF = './suite_test' @@ -22,13 +22,33 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(report.suite_to_text(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_error.py]' in output +assert "type:'AssertionError' message: AssertionError()" in output +assert 'assert False' in output + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_fail.py]' in output +assert "type:'EpicFail' message: This failure is expected" in output +assert "test.set_fail('EpicFail', 'This failure is expected')" in output + +print('\n- a test with a raised f
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#8). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py A src/osmo_gsm_tester/report.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 290 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/8 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..16342c5 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import _prep -from osmo_gsm_tester import log, suite, config +from osmo_gsm_tester import log, suite, config, report config.ENV_CONF = './suite_test' @@ -22,13 +22,33 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(report.suite_to_text(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_error.py]' in output +assert "type:'AssertionError' message: AssertionError()" in output +assert 'assert False' in output + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_fail.py]' in output +assert "type:'EpicFail' message: This failure is expected" in output +assert "test.set_fail('EpicFail', 'This failure is expected')" in output + +print('\n- a test with a raised f
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#7). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py A src/osmo_gsm_tester/report.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 291 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/7 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..16342c5 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import _prep -from osmo_gsm_tester import log, suite, config +from osmo_gsm_tester import log, suite, config, report config.ENV_CONF = './suite_test' @@ -22,13 +22,33 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(report.suite_to_text(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_error.py]' in output +assert "type:'AssertionError' message: AssertionError()" in output +assert 'assert False' in output + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_fail.py]' in output +assert "type:'EpicFail' message: This failure is expected" in output +assert "test.set_fail('EpicFail', 'This failure is expected')" in output + +print('\n- a test with a raised f
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#6). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py A src/osmo_gsm_tester/report.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 293 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/6 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..16342c5 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import _prep -from osmo_gsm_tester import log, suite, config +from osmo_gsm_tester import log, suite, config, report config.ENV_CONF = './suite_test' @@ -22,13 +22,33 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(report.suite_to_text(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_error.py]' in output +assert "type:'AssertionError' message: AssertionError()" in output +assert 'assert False' in output + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +output = report.suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_fail.py]' in output +assert "type:'EpicFail' message: This failure is expected" in output +assert "test.set_fail('EpicFail', 'This failure is expected')" in output + +print('\n- a test with a raised f
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#5). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A src/osmo_gsm_tester/trial_report.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 313 insertions(+), 91 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/5 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..c301c73 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 import os import _prep -from osmo_gsm_tester import log, suite, config +from osmo_gsm_tester import log, suite, config, trial_report config.ENV_CONF = './suite_test' @@ -22,13 +22,33 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(trial_report.TrialReportText(None).suite_to_text(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +output = trial_report.TrialReportText(None).suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_error.py]' in output +assert "type:'AssertionError' message: AssertionError()" in output +assert 'assert False' in output + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +output = trial_report.TrialReportText(None).suite_to_text(s) +assert 'FAIL: [test_suite] 1 failed ' in output +assert 'FAIL: [test_fail.py]' in output +assert "type:'EpicFail' message: This failure is expected" in output +assert "test.
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#4). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A src/osmo_gsm_tester/trial_report.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 15 files changed, 309 insertions(+), 90 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/4 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..3750a94 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -22,13 +22,30 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(str(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_error.py]' in str(s) +assert "type:'AssertionError' message: AssertionError()" in str(s) +assert 'assert False' in str(s) + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_fail.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s) + +print('\n- a test with a raised failure') +results = s.run_tests('test_fail_raise.py') +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_fail_raise.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "raise Failure('EpicFail', 'This failure is expected')" in
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#3). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 14 files changed, 265 insertions(+), 89 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/3 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..30c6915 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..3750a94 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -22,13 +22,30 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(str(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_error.py]' in str(s) +assert "type:'AssertionError' message: AssertionError()" in str(s) +assert 'assert False' in str(s) + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_fail.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s) + +print('\n- a test with a raised failure') +results = s.run_tests('test_fail_raise.py') +assert 'FAIL: [test_suite] 1 failed ' in str(s) +assert 'FAIL: [test_fail_raise.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "raise Failure('EpicFail', 'This failure is expected')" in str(s) print('\n- graceful exit.')
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder, I'd like you to reexamine a change. Please visit https://gerrit.osmocom.org/2669 to look at the new patch set (#2). Add JUnit XML reports; refactor test reporting * Add Junit output file support * Differentiate between an expected failure test and an error in the test, as described in JUnit. * In case of an error/exception during test, record and attach it to the Test object and continue running the tests, and show it at the end during the trial report. Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5 --- M .gitignore M selftest/suite_test.ok M selftest/suite_test.py A selftest/suite_test/test_suite/test_fail.py A selftest/suite_test/test_suite/test_fail_raise.py M selftest/trial_test.ok M src/osmo-gsm-tester.py M src/osmo_gsm_tester/suite.py M src/osmo_gsm_tester/test.py M src/osmo_gsm_tester/trial.py A suites/debug/error.py M suites/debug/fail.py A suites/debug/fail_raise.py A suites/debug/pass.py 14 files changed, 275 insertions(+), 88 deletions(-) git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/2 diff --git a/.gitignore b/.gitignore index 4119e7f..1ab84d9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ test_work state *.pyc +selftest/trial_test/ diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok index fda77dc..129529e 100644 --- a/selftest/suite_test.ok +++ b/selftest/suite_test.ok @@ -59,15 +59,30 @@ tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]] tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py] -pass: all 1 tests passed. +tst test_suite: PASS +pass: all 6 tests passed (5 skipped). - a test with an error tst test_suite: Suite run start [suite.py:[LINENR]] tst test_error.py:[LINENR] START [test_suite↪test_error.py] [suite.py:[LINENR]] tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]' [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]] -tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]] [suite.py:[LINENR]] -tst test_error.py:[LINENR]: ERR: AssertionError: [test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False] -FAIL: 1 of 1 tests failed: - test_error.py +tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]: assert False] +tst test_error.py:[LINENR] ERROR (AssertionError) [test_suite↪test_error.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]' [test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]] +tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] + +- a test with a raised failure +tst test_suite: Suite run start [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: I am 'test_suite' / 'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]] [test_fail_raise.py:[LINENR]] +tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This failure is expected')] +tst test_fail_raise.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail_raise.py] [suite.py:[LINENR]] +tst test_suite: FAIL [suite.py:[LINENR]] - graceful exit. diff --git a/selftest/suite_test.py b/selftest/suite_test.py index 315c683..1742ef2 100755 --- a/selftest/suite_test.py +++ b/selftest/suite_test.py @@ -22,13 +22,30 @@ print('- run hello world test') s = suite.SuiteRun(None, 'test_suite', s_def) results = s.run_tests('hello_world.py') -print(str(results)) +print(str(s)) log.style_change(src=True) #log.style_change(trace=True) print('\n- a test with an error') results = s.run_tests('test_error.py') -print(str(results)) +assert 'FAIL: [test_suite] failed->0 error->1 ' in str(s) +assert 'ERROR: [test_error.py]' in str(s) +assert "type:'AssertionError' message: AssertionError()" in str(s) +assert 'assert False' in str(s) + +print('\n- a test with a failure') +results = s.run_tests('test_fail.py') +assert 'FAIL: [test_suite] failed->1 error->0 ' in str(s) +assert 'FAIL: [test_fail.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s) + +print('\n- a test with a raised failure') +results = s.run_tests('test_fail_raise.py') +assert 'FAIL: [test_suite] failed->1 error->0 ' in str(s) +assert 'FAIL: [test_fail_raise.py]' in str(s) +assert "type:'EpicFail' message: This failure is expected" in str(s) +assert "raise Failure('EpicFail', 'This failure is expected')" in str(s