[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#8).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
A src/osmo_gsm_tester/report.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
15 files changed, 290 insertions(+), 91 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/8
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..16342c5 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
import os
import _prep
-from osmo_gsm_tester import log, suite, config
+from osmo_gsm_tester import log, suite, config, report
config.ENV_CONF = './suite_test'
@@ -22,13 +22,33 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(report.suite_to_text(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_error.py]' in output
+assert "type:'AssertionError' message: AssertionError()" in output
+assert 'assert False' in output
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_fail.py]' in output
+assert "type:'EpicFail' message: This failure is expected" in output
+assert "test.set_fail('EpicFail', 'This failure is expected')" in output
+
+print('\n- a test with a raised
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#7).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
A src/osmo_gsm_tester/report.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
15 files changed, 291 insertions(+), 91 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/7
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..16342c5 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
import os
import _prep
-from osmo_gsm_tester import log, suite, config
+from osmo_gsm_tester import log, suite, config, report
config.ENV_CONF = './suite_test'
@@ -22,13 +22,33 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(report.suite_to_text(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_error.py]' in output
+assert "type:'AssertionError' message: AssertionError()" in output
+assert 'assert False' in output
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_fail.py]' in output
+assert "type:'EpicFail' message: This failure is expected" in output
+assert "test.set_fail('EpicFail', 'This failure is expected')" in output
+
+print('\n- a test with a raised
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#6).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
A src/osmo_gsm_tester/report.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
15 files changed, 293 insertions(+), 91 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/6
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..16342c5 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
import os
import _prep
-from osmo_gsm_tester import log, suite, config
+from osmo_gsm_tester import log, suite, config, report
config.ENV_CONF = './suite_test'
@@ -22,13 +22,33 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(report.suite_to_text(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_error.py]' in output
+assert "type:'AssertionError' message: AssertionError()" in output
+assert 'assert False' in output
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+output = report.suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_fail.py]' in output
+assert "type:'EpicFail' message: This failure is expected" in output
+assert "test.set_fail('EpicFail', 'This failure is expected')" in output
+
+print('\n- a test with a raised
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#5).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A src/osmo_gsm_tester/trial_report.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
15 files changed, 313 insertions(+), 91 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/5
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..c301c73 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -1,7 +1,7 @@
#!/usr/bin/env python3
import os
import _prep
-from osmo_gsm_tester import log, suite, config
+from osmo_gsm_tester import log, suite, config, trial_report
config.ENV_CONF = './suite_test'
@@ -22,13 +22,33 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(trial_report.TrialReportText(None).suite_to_text(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+output = trial_report.TrialReportText(None).suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_error.py]' in output
+assert "type:'AssertionError' message: AssertionError()" in output
+assert 'assert False' in output
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+output = trial_report.TrialReportText(None).suite_to_text(s)
+assert 'FAIL: [test_suite] 1 failed ' in output
+assert 'FAIL: [test_fail.py]' in output
+assert "type:'EpicFail' message: This failure is expected" in output
+assert
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#4).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A src/osmo_gsm_tester/trial_report.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
15 files changed, 309 insertions(+), 90 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/4
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..3750a94 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -22,13 +22,30 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(str(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_error.py]' in str(s)
+assert "type:'AssertionError' message: AssertionError()" in str(s)
+assert 'assert False' in str(s)
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_fail.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s)
+
+print('\n- a test with a raised failure')
+results = s.run_tests('test_fail_raise.py')
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_fail_raise.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "raise Failure('EpicFail', 'This failure is expected')"
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#3).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
14 files changed, 265 insertions(+), 89 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/3
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..30c6915 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] FAIL (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..3750a94 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -22,13 +22,30 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(str(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_error.py]' in str(s)
+assert "type:'AssertionError' message: AssertionError()" in str(s)
+assert 'assert False' in str(s)
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_fail.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s)
+
+print('\n- a test with a raised failure')
+results = s.run_tests('test_fail_raise.py')
+assert 'FAIL: [test_suite] 1 failed ' in str(s)
+assert 'FAIL: [test_fail_raise.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "raise Failure('EpicFail', 'This failure is expected')" in str(s)
print('\n- graceful
[PATCH] osmo-gsm-tester[master]: Add JUnit XML reports; refactor test reporting
Hello Neels Hofmeyr, Jenkins Builder,
I'd like you to reexamine a change. Please visit
https://gerrit.osmocom.org/2669
to look at the new patch set (#2).
Add JUnit XML reports; refactor test reporting
* Add Junit output file support
* Differentiate between an expected failure test and an error in the
test, as described in JUnit.
* In case of an error/exception during test, record and attach it to the
Test object and continue running the tests, and show it at the end
during the trial report.
Change-Id: Iedf6d912b3ccea187a4ac6d5c6b70fe9d5c5
---
M .gitignore
M selftest/suite_test.ok
M selftest/suite_test.py
A selftest/suite_test/test_suite/test_fail.py
A selftest/suite_test/test_suite/test_fail_raise.py
M selftest/trial_test.ok
M src/osmo-gsm-tester.py
M src/osmo_gsm_tester/suite.py
M src/osmo_gsm_tester/test.py
M src/osmo_gsm_tester/trial.py
A suites/debug/error.py
M suites/debug/fail.py
A suites/debug/fail_raise.py
A suites/debug/pass.py
14 files changed, 275 insertions(+), 88 deletions(-)
git pull ssh://gerrit.osmocom.org:29418/osmo-gsm-tester refs/changes/69/2669/2
diff --git a/.gitignore b/.gitignore
index 4119e7f..1ab84d9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,4 @@
test_work
state
*.pyc
+selftest/trial_test/
diff --git a/selftest/suite_test.ok b/selftest/suite_test.ok
index fda77dc..129529e 100644
--- a/selftest/suite_test.ok
+++ b/selftest/suite_test.ok
@@ -59,15 +59,30 @@
tst hello_world.py:[LINENR]: two [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR]: three [test_suite↪hello_world.py:[LINENR]]
tst hello_world.py:[LINENR] PASS [test_suite↪hello_world.py]
-pass: all 1 tests passed.
+tst test_suite: PASS
+pass: all 6 tests passed (5 skipped).
- a test with an error
tst test_suite: Suite run start [suite.py:[LINENR]]
tst test_error.py:[LINENR] START [test_suite↪test_error.py]
[suite.py:[LINENR]]
tst test_error.py:[LINENR]: I am 'test_suite' / 'test_error.py:[LINENR]'
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]]
-tst test_error.py:[LINENR]: FAIL [test_suite↪test_error.py:[LINENR]]
[suite.py:[LINENR]]
-tst test_error.py:[LINENR]: ERR: AssertionError:
[test_suite↪test_error.py:[LINENR]] [test_error.py:[LINENR]: assert False]
-FAIL: 1 of 1 tests failed:
- test_error.py
+tst test_error.py:[LINENR]: ERR: AssertionError: [test_error.py:[LINENR]:
assert False]
+tst test_error.py:[LINENR] ERROR (AssertionError) [test_suite↪test_error.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail.py:[LINENR] START [test_suite↪test_fail.py] [suite.py:[LINENR]]
+tst test_fail.py:[LINENR]: I am 'test_suite' / 'test_fail.py:[LINENR]'
[test_suite↪test_fail.py:[LINENR]] [test_fail.py:[LINENR]]
+tst test_fail.py:[LINENR] FAIL (EpicFail) [test_suite↪test_fail.py]
[suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
+
+- a test with a raised failure
+tst test_suite: Suite run start [suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR] START [test_suite↪test_fail_raise.py]
[suite.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: I am 'test_suite' /
'test_fail_raise.py:[LINENR]' [test_suite↪test_fail_raise.py:[LINENR]]
[test_fail_raise.py:[LINENR]]
+tst test_fail_raise.py:[LINENR]: ERR: Failure: ('EpicFail', 'This failure is
expected') [test_fail_raise.py:[LINENR]: raise Failure('EpicFail', 'This
failure is expected')]
+tst test_fail_raise.py:[LINENR] FAIL (EpicFail)
[test_suite↪test_fail_raise.py] [suite.py:[LINENR]]
+tst test_suite: FAIL [suite.py:[LINENR]]
- graceful exit.
diff --git a/selftest/suite_test.py b/selftest/suite_test.py
index 315c683..1742ef2 100755
--- a/selftest/suite_test.py
+++ b/selftest/suite_test.py
@@ -22,13 +22,30 @@
print('- run hello world test')
s = suite.SuiteRun(None, 'test_suite', s_def)
results = s.run_tests('hello_world.py')
-print(str(results))
+print(str(s))
log.style_change(src=True)
#log.style_change(trace=True)
print('\n- a test with an error')
results = s.run_tests('test_error.py')
-print(str(results))
+assert 'FAIL: [test_suite] failed->0 error->1 ' in str(s)
+assert 'ERROR: [test_error.py]' in str(s)
+assert "type:'AssertionError' message: AssertionError()" in str(s)
+assert 'assert False' in str(s)
+
+print('\n- a test with a failure')
+results = s.run_tests('test_fail.py')
+assert 'FAIL: [test_suite] failed->1 error->0 ' in str(s)
+assert 'FAIL: [test_fail.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "test.set_fail('EpicFail', 'This failure is expected')" in str(s)
+
+print('\n- a test with a raised failure')
+results = s.run_tests('test_fail_raise.py')
+assert 'FAIL: [test_suite] failed->1 error->0 ' in str(s)
+assert 'FAIL: [test_fail_raise.py]' in str(s)
+assert "type:'EpicFail' message: This failure is expected" in str(s)
+assert "raise Failure('EpicFail', 'This failure is expected')" in
