#35849: ParallelTestSuite reports error occurred in arbitrary test instead of
setUpClass
-------------------------------------+-------------------------------------
Reporter: David Winiecki | Type: Bug
Status: new | Component: Testing
| framework
Version: 5.0 | Severity: Normal
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
When running tests in parallel, if an error occurs in a test suite
(`TestCase`) before the first test method runs, for example in
`setUpClass`, then the shell output incorrectly states that the error
occurred in an arbitrary test method, rather than, for example,
`setUpClass`.
This is a problem because a user reading the test output may not notice
the real error location (for example, `setUpClass`) in the traceback and
may not run the other tests in the test suite.
This is especially problematic if automation parses the error location to
automatically re-run tests (to address flaky, intermittently failing
tests), because the other tests in the suite may never be run.
Repro:
1. Create a Django project for testing:
Instructions to create the project yourself are below. **Or you can use
this existing project** in the `bugdemo/` directory at
https://github.com/dcki/django/blob/demo_report_test_error_bug/bugdemo/README.md
To create the project yourself:
A. Install Python.
B. Create a virtualenv and activate it: `mkdir my_project && cd my_project
&& python3 -m venv .venv && . .venv/bin/activate`
C. Install Django and create a Django project: `pip install django &&
django-admin startproject mysite .`
D. Create a tests directory and an `__init__.py` file inside: `mkdir
mysite/tests && touch mysite/tests/__init__.py`
E. Install tblib: `pip install tblib`
F. Add these tests to the project:
`mysite/tests/test_things.py`
{{{
from django.test import SimpleTestCase
class AlwaysFailTest(SimpleTestCase):
@classmethod
def setUpClass(cls) -> None:
super().setUpClass()
try:
raise Exception('Intentional error')
except:
super().tearDownClass()
raise
def test_should_pass_a(self) -> None:
pass
def test_should_pass_b(self) -> None:
pass
# Exists so that, when an attempt is made to run tests in parallel, and at
# least two TestCases are specified, then tests actually run in parallel.
# (As of this writing, Django runs tests serially if there is only one
test
# suite to run, even if `--parallel=2` is specified.)
class AlwaysPassTest(SimpleTestCase):
def test_should_pass_a(self) -> None:
pass
def test_should_pass_b(self) -> None:
pass
}}}
2. Run the tests: `python3 manage.py test --parallel=2`
Expected output:
{{{
Found 4 test(s).
System check identified no issues (0 silenced).
E..
======================================================================
ERROR: setUpClass (mysite.tests.test_things.AlwaysFailTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py",
line 166, in _handleClassSetUp
setUpClass()
File "/Users/dwiniecki/src/other/dcki-
django/bugdemo2/mysite/tests/test_things.py", line 9, in setUpClass
raise Exception('Intentional error')
^^^^^^^^^^^^^^^^^
Exception: Intentional error
----------------------------------------------------------------------
Ran 2 tests in 0.214s
FAILED (errors=1)
}}}
Actual output:
{{{
Found 4 test(s).
System check identified no issues (0 silenced).
E..
======================================================================
ERROR: test_should_pass_b
(mysite.tests.test_things.AlwaysFailTest.test_should_pass_b)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/Users/dwiniecki/.pyenv/versions/3.11.9/lib/python3.11/unittest/suite.py",
line 166, in _handleClassSetUp
setUpClass()
File "/Users/dwiniecki/src/other/dcki-
django/bugdemo/mysite/tests/test_things.py", line 9, in setUpClass
raise Exception('Intentional error')
^^^^^^^^^^^^^^^^^
Exception: Intentional error
----------------------------------------------------------------------
Ran 2 tests in 0.221s
FAILED (errors=1)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35849>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/010701929bfa5796-8fffade9-01ea-4f41-9dc2-77547ef19dad-000000%40eu-central-1.amazonses.com.
