#32140: `startTest` result hook not called until after tests already started
when
`--parallel` used
---------------------------------------------+------------------------
Reporter: bobwhitelock | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.1
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
Given a custom Django test runner that uses a custom result class, like
this:
{{{
from unittest.runner import TextTestRunner, TextTestResult
from django.test.runner import DiscoverRunner
class CustomResult(TextTestResult):
def startTest(self, test):
print(f"startTest called for {test}")
super().startTest(test)
class CustomRunner(TextTestRunner):
resultclass = CustomResult
class CustomDjangoRunner(DiscoverRunner):
test_runner = CustomRunner
}}}
And some tests, like this:
{{{
from django.test import TestCase
class FirstTest(TestCase):
def test_first(self):
print("Running test_first")
class SecondTest(TestCase):
def test_second(self):
print("Running test_second")
}}}
When the tests are run without `--parallel`, the expected behaviour
occurs, and the `startTest` hook is called just before starting the test:
{{{
$ ./manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
startTest called for test_first (parallel_test.tests.FirstTest)
Running test_first
.startTest called for test_second (parallel_test.tests.SecondTest)
Running test_second
.
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
Destroying test database for alias 'default'...
}}}
When the tests are run with `--parallel`, the `startTest` hook appears to
be called after the test has already been run:
{{{
$ ./manage.py test --parallel
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
System check identified no issues (0 silenced).
Running test_first
Running test_second
startTest called for test_second (parallel_test.tests.SecondTest)
.startTest called for test_first (parallel_test.tests.FirstTest)
.
----------------------------------------------------------------------
Ran 2 tests in 0.017s
OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
}}}
This is surprising, as I would expect the hook to a. be called in a
consistent place when running a test, whether this is in parallel or
serial, and b. be called before/as the test is started. This inconsistent
behaviour can lead to issues when writing custom Django test runners, as
people use this hook to run some code just as each test is started - for
example see https://github.com/xmlrunner/unittest-xml-reporting/issues/229
and https://stackoverflow.com/a/39758653/2620402.
See https://github.com/bobwhitelock/django_parallel_test_issue_repro for a
minimal repro of the issue including the code samples above.
I've confirmed this issue is present in both Django 3.1 and 2.2.
Happy to help with fixing this, but I'd need some pointers as to where to
start. Thanks!
--
Ticket URL: <https://code.djangoproject.com/ticket/32140>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/055.1d173dbc701ee6bdc85402e5c5fb8437%40djangoproject.com.
