[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Ned Batchelder]

Ned Batchelder added the comment:

Whoever takes this up next: let me know if I can help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Changes by Brett Cannon :


--
assignee: brett.cannon -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Brett Cannon added the comment:

Just an FYI that I have never managed to make this work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Changes by Brett Cannon :


--
pull_requests: +1948

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Changes by Brett Cannon :


--
assignee:  -> brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Brett Cannon added the comment:

I learned a few thing trying to make this work.

One is that the COVERAGE_PROCESS_START does need to point to an actual .ini 
file (which includes an empty file).

Two, 
https://bitbucket.org/ned/coveragepy/src/63dfe482a849e9cb0b82214242315a806a6a3d53/coverage/control.py?at=default=file-view-default#control.py-1265
 shows that stdlib coverage is left off by this, so even if you do turn on 
tracing in a subprocess, it will still ignore the stdlib. That means the 
coverage config file must turn on stdlib coverage in order or it to be picked 
up in the subprocesses.

Three, you need to run `coverage combine` to create a unified .coverage file 
for `coverage report` to pick up (I don't know if codecov explicitly needs the 
merge step.

When I have time I'll make a PR to test if this change actually helps speed 
things up.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Brett Cannon]

Brett Cannon added the comment:

There's also what changes might occur in the coverage results when we start 
using fullcoverage: https://github.com/python/core-workflow/issues/18

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Nick Coghlan]

Nick Coghlan added the comment:

To be more specific regarding `sitecustomize.py`:

$ echo "print('Hello from sitecustomize.py')" > Lib/sitecustomize.py
$ ./python -c "print('Hello from command line')"
Hello from sitecustomize.py
Hello from command line

Since we only need these instructions to work for a local checkout, we can rely 
on the `sitecustomize.py` hook.

It means we'll still miss coverage results from subprocess tests run in 
isolated mode or with site.py processing disabled, but those are both 
relatively rare and involve *not* running code that is normally run, so 
shouldn't impact the aggregate coverage results.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Nick Coghlan]

Nick Coghlan added the comment:

As a starting point, I'd suggest looking at what can be achieved without making 
any changes to CPython or its test suite:

1. Set COVERAGE_PROCESS_START in the environment where the tests are being run

2. Inject a sitecustomize.py file into Lib (and add `Lib/sitecustomize.py` to 
`.gitignore`)

There are cases that won't cover (like subprocesses with a custom environment), 
but it will provide a starting point for the tests that just pass the current 
environment through, and will also provide a way to notify 
test.support.script_helper of the expected value of COVERAGE_PROCESS_START in 
the future.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Jaysinh shukla]

Jaysinh shukla added the comment:

I found the regrtest wasn't displaying correct coverage for when the code is 
executed from call `Lib.test.support.script_helper.assert_python_ok`. I found 
`assert_python_ok` is using `subprocess` under the hood. It seems this problem 
is unobserved from a long time but it is having status `open`.

What should be the ideal situation for this? I request any core-developer to 
guide on this. I am interested in doing some more research on this issue. 
Thanks!

--
nosy: +jaysinh.shukla

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11978] Report correct coverage.py data for tests that invoke subprocesses

[Nick Coghlan]

New submission from Nick Coghlan ncogh...@gmail.com:

http://nedbatchelder.com/code/coverage/subprocess.html describes how to 
instruct a test suite that spawns subprocesses to correctly report coverage 
data for code covered only in those subprocesses.

regrtest currently doesn't do this, so modules that use subprocesses to run 
their tests may report inaccurate coverage numbers when using this tool (as 
recommended in the devguide). (Those numbers are irredeemably inaccurate when 
it comes to regrtest's own coverage assessment)

It may be better to build explicit invocation of coverage.py when regrtest is 
run under coverage.py into test.script_helper rather than trying to use the 
implicit mechanism though (as neither sitecustomize nor a .pth file are 
particularly good ideas when running Python's own test suite).

--
messages: 134967
nosy: brett.cannon, ncoghlan, nedbat
priority: normal
severity: normal
status: open
title: Report correct coverage.py data for tests that invoke subprocesses
type: feature request

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11978
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com