[issue46523] Test suite skips failing tests when setUp[Class] fails

[Nikita Sobolev]

Change by Nikita Sobolev :


--
keywords: +patch
pull_requests: +29076
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30895

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46523] Test suite skips failing tests when setUp[Class] fails

[Ken Jin]

Change by Ken Jin :


--
title: Test suite skips failing tests -> Test suite skips failing tests when 
setUp[Class] fails

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46523] Test suite skips failing tests

[Ken Jin]

Ken Jin  added the comment:

I vote for option 2.

--
nosy: +ezio.melotti, gvanrossum, kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46523] Test suite skips failing tests

[Nikita Sobolev]

New submission from Nikita Sobolev :

Here's what happened. We had an error in `test_typing.py`, which was silently 
ignored.

```
 ==
ERROR: setUpClass (test.test_typing.NewTypeTests)
--
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
UserId = NewType('UserId', int)
NameError: name 'NewType' is not defined

--
Ran 396 tests in 0.085s

FAILED (errors=1, skipped=1)
test test_typing failed
```

Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

But, later the suite runner tried to rerun it:

```
0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: 
setUpClass)

1 re-run test:
test_typing

1 test run no tests:
test_typing
```

And since nothing matched `setUpClass` - no tests were executed and the CI went 
green instead of red.

What can we do?
1. Only schedule real `test_` item to be rerun, fail for everything else
2. Convert `setupClass` failure into the whole class rerun
3. Other options?

I would like to work on this, when we will decide which way is best.

--
components: Tests
messages: 411620
nosy: sobolevn
priority: normal
severity: normal
status: open
title: Test suite skips failing tests
type: behavior
versions: Python 3.10, Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com