[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Choi]

Kit Choi  added the comment:

See issue38320 for documentation change request

--
nosy: +Kit Choi2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Yan Choi]

Kit Yan Choi  added the comment:

I think Python does differentiate "test error" and "test failure" such that a 
test outcome state can be one of these: success, failure, error, skipped. One 
could refine these to six: expected success, unexpected success, expected 
failure, unexpected failure, error, skipped.


For example, in the documentation for failureException:

* failureException: determines which exception will be raised when
the instance's assertion methods fail; test methods raising this
exception will be deemed to have 'failed' rather than 'errored'.


Another evidence: unittest.runner.TextTestResult, there are methods called 
"addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and 
"addUnexpectedSuccess".


For example, this test outcome is marked as "FAILED":

def test(self):
x = 1
y = 2
self.assertEqual(x + y, 4)


==
FAIL: test (test_main.T)
--
Traceback (most recent call last):
  File "test_main.py", line 9, in test
self.assertEqual(x + y, 4)
AssertionError: 3 != 4


But the test outcome for this test is "ERROR":

def test(self):
x = 1
y = 2 + z  # NameError  
self.assertEqual(x + y, 4)


==
ERROR: test (test_main.T)
--
Traceback (most recent call last):
  File "test_main.py", line 8, in test
y = 2 + z  # NameError
NameError: global name 'z' is not defined


The issue here being "expectedFailure" converting "error" to "success", which 
is not expected, and is causing decorated tests to become unmaintained. While 
the rest of unittest differentiates "error" and "failure", expectedFailure does 
not. This is either a bug in the behaviour of expectedFailure, or a bug in the 
documentation for not being clear on the fact that unexpected error will be 
considered as expected failure (which I think is wrong).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Terry J. Reedy]

Terry J. Reedy  added the comment:

A test either passes or fails.  Like a not operator, the expectedFailure 
decorator inverts the result.

https://docs.python.org/3/library/unittest.html#unittest.expectedFailure
@unittest.expectedFailure
Mark the test as an expected failure. If the test fails it will be 
considered a success. If the test passes, it will be considered a failure.

By itself, your 'test' method fails.  Decorated, it should and does pass.

As you suggested, using expectedFailure is a blunt instrument that can be 
misleading if not used carefully and not reviewed when editing the module 
tested.  It is only used 7 times in test_xyz.py modules in the lib/test 
directory and subdirectories.

--
nosy: +ezio.melotti, michael.foord
versions:  -Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Yan Choi]

Kit Yan Choi  added the comment:

Pining Chris based on previous discussion in issue16997 ... Hope that's okay.

I notice that the language in my initial message also conflates error and 
failure. My apologies on the carelessness.

Just to clarify:


@unittest.expectedFailure
def test(self):
THIS_VARIABLE_IS_UNDEFINED  # ---> NameError


should give: ERROR (errors=1)
currently gives: OK (expected failures=1)


By fixing this, we can help projects to maintain their tests decorated with 
expectedFailure so that the tests remaining to be meaningful.

--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Yan Choi]

Change by Kit Yan Choi :


--
nosy:  -Kit Choi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Yan Choi]

Kit Yan Choi  added the comment:

For your test:

class T(unittest.TestCase):
def test_f(self): raise TypeError()

If you run this test with unittest test runner, you should get this result:

E
==
ERROR: test_f (test_main.T)
--
Traceback (most recent call last):
  File "test_main.py", line 5, in test_f
def test_f(self): raise TypeError()
TypeError

--
Ran 1 test in 0.000s

FAILED (errors=1)


I expect to get this behaviour even if the test is decorated with 
unittest.expectedFailure. However, currently we get a success.


Scenario:


You create a class named Duck with a method "quack".  Then you added a test, 
and test needs to call Duck.quack.

Later on for whatever reason, you need to decorate the test with 
expectedFailure.  The test passes with the expected failure.

Then you rename the "quack" method to "walk", but you forget to update the 
test.  Now the test is actually failing with an AttributeError, but you won't 
notice it because expectedFailure silences it.

In this scenario, it is important to differentiate a "test error" and a "test 
failure". A test has four status: success, failure, error, skipped.  I expect 
unittest.expectedFailure to make "failure" a "success" and a "success" a 
"failure", and it should leave "error" and "skipped" unchanged.

Please consider reopening this issue.

--
nosy: +Kit Yan Choi

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Terry J. Reedy]

Terry J. Reedy  added the comment:

A function can fail to return an expected object by either returning a wrong 
object or raising an (unexpected) exception.  The assertXyz methods, which 
ultimately raise AssertionError or something similar, are mostly about catching 
the first kind of failure, but tests should also catch and report the second 
kind.  The traceback shows the kind of failure.  The assertXyx failures add 
additional details after the traceback.


import unittest

class T(unittest.TestCase):
def test_f(self): raise TypeError()

unittest.main()

# Properly results in

Traceback (most recent call last):
  File "F:\Python\a\tem4.py", line 4, in test_f
def test_f(self): raise TypeError()
TypeError

--
Ran 1 test in 0.050s

FAILED (errors=1)

--
nosy: +terry.reedy
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38296] unittest expectedFailure does not differentiate errors from failures

[Kit Choi]

New submission from Kit Choi :

I expect the following test to fail, because an "error" is not a "failure".
Unexpectedly, the test passes:

```
class TestFailure(unittest.TestCase):

@unittest.expectedFailure
def test_expected_failure(self):
raise TypeError()   # for example, a typo.
```

```
$ python -m unittest test_main
x
--
Ran 1 test in 0.000s

OK (expected failures=1)
```

This behaviour exists since Python 2.7, and is still true for the Python 3.8.0b1

--
components: Tests
messages: 353382
nosy: Kit Choi
priority: normal
severity: normal
status: open
title: unittest expectedFailure does not differentiate errors from failures
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com