#31431: Better Coverage Analysis integration with multiprocessed Django Test
(ParallelTestSuite)
-------------------------------------+-------------------------------------
Reporter: Ichlasul | Owner: nobody
Affan |
Type: New | Status: new
feature |
Component: Testing | Version: master
framework | Keywords: coverage; parallel
Severity: Normal | testing
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Currently, to use {{{Coverage.py}}} for line coverage analysis on Django
tests, we can use this command:
{{{
coverage run --omit='....' manage.py test .....
coverage report -m
}}}
This method is valid on single-processed testing. {{{Coverage.py}}} will
analyze which lines that Python interpreter has gone through when the
testing process runs. Unfortunately, on multi-processed testing, coverage
analysis result becomes invalid as it marks some lines as "uncovered",
given the same code and test suites.
Based on my initial observation, looks like the {{{Coverage.py}}} only
analyze lines run by main Python interpreter process. The child processes
are untouched. So, only tests run by the first process that are being
analyzed. For example, I have two different test suites:
- {{{AccountTests}}} for {{{account}}} app, and
- {{{HistoryTests}}} for {{{history}}} app.
Let's say the first test process will run {{{AccountTests}}}. The result
will only mark as "covered" on code lines in {{{account}}} app, and some
of the {{{models.py}}}, {{{urls.py}}}, and {{{apps.py}}} code of
{{{history}}} app. The latter happened because Django initialized App
registry, test models, and migrations using codes in {{{apps.py}}} and
{{{models.py}}}. The parent {{{urls.py}}} also imports {{{urls.py}}}
inside each app.
These are subfeatures that I propose in this ticket:
1. Integration of Coverage.py into Django Test engine, so we can use
coverage analysis by using: {{{python manage.py test --check-coverage
--coverage-omit='....' ......}}}. The execution of Coverage.py is now
handled by Django test using different test runner called
{{{CoverageTestRunner}}}, that extends the ordinary {{{DiscoverRunner}}}.
2. Implementation of {{{CoverageTestRunner}}} itself, by utilizing
{{{Coverage.py}}}'s own API. This includes initialization process for
coverage analysis inside the test runner.
3. Implementation of multiprocessed test runner support. It might be
implemented using different init worker than the usual
{{{ParallelTestSuite}}}, or there might be any better idea in the future.
When merging test results into parent process, it will also need to merge
the coverage analysis result of each processes.
I have no idea yet for a concrete Proof of Concept. The main idea is
initializing coverage analysis API for each of test processes, then merge
the results using {{{OR}}} operator. That said, a line counts as "covered"
when it is executed by minimum of one test process. Any suggestions about
this idea will be very much appreciated.
--
Ticket URL: <https://code.djangoproject.com/ticket/31431>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/057.0f92d6dd1625d0269d1910ec276a7fef%40djangoproject.com.
