Wow. After carefully reading the pytest-cov plugin (only) code and the PyTest code that powers loading plugins, I figured it out. The plugin was getting loaded before pytest-cov started the Coverage analyzer. Since the plugin had module-level imports from other parts of our project, that meant that most of the project was imported (and, so, module-level imports and code and class definitions had already executed) before Coverage started. The fix was actually very simple: I moved the plugin into a different module and then moved its top-level imports for the same project to happen at the last second. Viola. Coverage is fixed. Now just the plugin module itself reports wonky coverage numbers, but I can live with that.
Thanks again for your help! Nick On Mon, Apr 23, 2018 at 5:42 PM, Nicholas Williams < nicholas+pyt...@nicholaswilliams.net> wrote: > Floris, your advice was invaluable. We have created a really fantastic > PyTest plugin for our service tests. However, I wanted to share it with > you, because it seems to have introduced some type of weird conflict with > the PyTest Coverage plugin, and I was hoping maybe you had some insights as > to what the problem might be: > > Normally, when I run PyTest tests with the PyTest Coverage plugin, > everything works as expected in regards to the coverage calculation and the > coverage reports. And, with this change to our project, everything STILL > works properly IF AND ONLY IF I run tests with `coverage run --source=pysoa > setup.py test` followed by `coverage report` (so, without the PyTest > Coverage plugin). However, with this change to our project, something VERY > weird happens to the coverage report if I rely on the PyTest Coverage > plugin. Now, ONLY LINES OF CODE WITHIN FUNCTIONS AND METHODS are marked as > covered. The following types of code, which were previously marked as > covered (and are clearly covered), are now marked as uncovered (again, only > when using the PyTest Coverage plugin): > > - module-level imports > - module-level code that executes on import > - module docstrings > - module-level class definitions (class ClassName(...):), or class > definitions within other class definitions > - code that executes within a class on definition > - class docstrings > - function definition lines (def func_name(...):), but not the code within > the functions, which is marked as covered > - method definition lines within classes (def method_name(...):), but not > the code within the functions, which is marked as covered > > While the entire pull request is large and daunting, the most relevant > part is the PyTest plugin itself, which is contained in a single file > "pytest_plugin.py" that you can search for on the page. Here it is: > https://github.com/eventbrite/pysoa/pull/87/files > > Unfortunately, I did not notice this problem with coverage until the bulk > of the code was written and tested, so determining what part of the code > caused it by removing bits of code is ... impracticable. I'm hoping one of > the pytest-cov experts in here will have some idea what the heck is going > on and can point me in the right direction. Perhaps I'm doing something > wrong, but this smells very strongly of a pytest-cov bug somehow. > > Thanks, > > Nick > > > On Fri, Apr 20, 2018 at 2:50 PM, Floris Bruynooghe <f...@devork.be> wrote: > >> On Thu 19 Apr 2018 at 16:50 -0500, Nicholas Williams wrote: >> >> > The tests work this way: >> > >> > - There's a test class that inherits from our specialized test class >> > (ServicePlanTestCase), which inherits from `unittest.TestCase` (for >> several >> > reasons, this detail cannot be altered) >> > - That test class can do setup and teardown like a normal test class, >> but >> > it has a static attribute that points to a directory >> > - That directory contains one or more files ending in a particular >> > extension, and those files each contain one or more tests defined using >> a >> > particular syntax >> >> I'm not even going to ask how this all came to be this way... but thanks >> for giving enough context. >> >> > Based on the hints you gave me, it sounds like I could do something like >> > this: >> > https://github.com/pytest-dev/pytest/blob/4678cbeb913385f00c >> c21b79662459a8c9fafa87/_pytest/unittest.py#L14-L22 >> > >> > Only, instead of checking for inheritance from TestCase, I'd check for >> > inheritance from our ServicePlanTestCase, and in that case I would >> return a >> > new collector object that inherits from _pytest.python.Class, and write >> > that new collector class to collect our tests. Am I barking up the right >> > tree now? >> >> pytest_pycollect_makeitem is supposed to create a single item, >> while from what you describe you still have two collections nested. You >> should probably create a custom collection node for your class, which >> then creates a different kind of collection node for each file which in >> turn creates the actual test nodes in it's .collect(). >> > >
_______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
