Kit Yan Choi <[email protected]> added the comment:
I think Python does differentiate "test error" and "test failure" such that a
test outcome state can be one of these: success, failure, error, skipped. One
could refine these to six: expected success, unexpected success, expected
failure, unexpected failure, error, skipped.
For example, in the documentation for failureException:
* failureException: determines which exception will be raised when
the instance's assertion methods fail; test methods raising this
exception will be deemed to have 'failed' rather than 'errored'.
Another evidence: unittest.runner.TextTestResult, there are methods called
"addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and
"addUnexpectedSuccess".
For example, this test outcome is marked as "FAILED":
def test(self):
x = 1
y = 2
self.assertEqual(x + y, 4)
======================================================================
FAIL: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_main.py", line 9, in test
self.assertEqual(x + y, 4)
AssertionError: 3 != 4
But the test outcome for this test is "ERROR":
def test(self):
x = 1
y = 2 + z # NameError
self.assertEqual(x + y, 4)
======================================================================
ERROR: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_main.py", line 8, in test
y = 2 + z # NameError
NameError: global name 'z' is not defined
The issue here being "expectedFailure" converting "error" to "success", which
is not expected, and is causing decorated tests to become unmaintained. While
the rest of unittest differentiates "error" and "failure", expectedFailure does
not. This is either a bug in the behaviour of expectedFailure, or a bug in the
documentation for not being clear on the fact that unexpected error will be
considered as expected failure (which I think is wrong).
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue38296>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
