#33264: DiscoverRunner doesn't counts unexpectedSuccess as an error
---------------------------------------------+------------------------
Reporter: Baptiste Mispelon | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: 3.2
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
---------------------------------------------+------------------------
I noticed today that my CI test suite was green even though its console
output printed:
{{{
FAILED (expected failures=13, unexpected successes=1)
}}}
It turns out that Django's default `DiscoverRunner` does not count
"unexpected successes" as errors when deciding the return code of its
`test` management command. [1]
So even though it prints "FAILED", the command returns with an exit code
of 0 which my CI (correctly) interprets as a success.
The fix seems quite simple:
{{{#!diff
diff --git a/django/test/runner.py b/django/test/runner.py
index 09ac4e142a..2e36514922 100644
--- a/django/test/runner.py
+++ b/django/test/runner.py
@@ -875,7 +875,7 @@ class DiscoverRunner:
teardown_test_environment()
def suite_result(self, suite, result, **kwargs):
- return len(result.failures) + len(result.errors)
+ return len(result.failures) + len(result.errors) +
len(result.unexpectedSuccesses)
def _get_databases(self, suite):
databases = {}
}}}
But I wonder if that would be considered backwards incompatible.
I also think this is a pretty serious issue and I wonder if backports
would be considered for this.
[1]
https://github.com/django/django/blob/60503cc747eeda7c61bab02b71f8f55a733a6eea/django/test/runner.py#L877-L878
--
Ticket URL: <https://code.djangoproject.com/ticket/33264>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/052.dcfb6b0cb8e2b25700bb5403e593e084%40djangoproject.com.
