Re: [python-committers] Codecov and PR

[Brett Cannon]

On Wed, 26 Apr 2017 at 22:36 Terry Reedy  wrote:

> On 4/26/2017 1:45 PM, Brett Cannon wrote:
> >
> > On Tue, 25 Apr 2017 at 17:00 Terry Reedy  > > wrote:
>
> > While I use code coverage to improve automated unittesting, I am
> opposed
> > to turning a usable but limited and sometime faulty tool into a blind
> > robotic master that blocks improvements.  The prospect of this being
> > done has discouraged me from learning the new system.  (More on
> 'faulty
> > tool' later.)
> >
> > It should be stated that code coverage is not a blocking status check
> > for merging from our perspective (the **only** required check is that
> > Travis pass with it's test run).
>
> I have the impression that at one time you hoped to make it blocking.
> If that was wrong, I apologize for misunderstanding.  If you have
> changed your mind, then I am happy.
>

"Hope", sure; "currently plan to", no. IOW I have said a lot of things over
the 2.5 years I've been leading this workflow shift and I don't expect all
of them to pan out.


>
> I am otherwise in favor of both the measurement and report of coverage
> being improved.
>
> > The temptation to write artificial tests to satisfy an artificial
> goal
> > is real.  Doing so can eat valuable time better used for something
> else.
> >For instance:
> >
> >   def meth(self, arg):
> >   mod.inst.meth(arg, True, ob=self, kw='cut')
> >
> > Mocking mod.class.meth, calling meth, and checking that the mock is
> > called will satisfy the robot, but does not contribute much to the
> goal
> > of providing a language that people can use to solve problems.
>
> > My assumption is that there will be a test that meth() does the right
> > thing, mock or not. If we provide an API there should be some test for
> > it; using a mock or something else to do the test is an implementation
> > detail.
>
> My impression is that default mocks have a generic signature, so that
> merely checking that the mock is called will not catch an invalid call.
> I presume that one can do better with mocks, and I have with custom
> mocks I have written, but my point above was that coverage does not care.
>
> >  >> If it's not important enough to require tests >> it's not
> > important enough to be in Python.  ;)
> >
> > Modules in the test package are mostly not tested. ;)
> >
> >
> > :) But they are at least executed which is what we're really measuring
> > here and I think all Ethan and I are advocating for.
>
> I thought Ethan was advocating for more -- a specific unittest for each
> line.
>
> > E.g. I don't expect
> > test_importlib to be directly responsible for exercising all code in
> > importlib, just that Python's entire test suite exercise importlib as
> > much as possible as a whole.
>
> The advantage for importlib in this respect is that import statements
> cannot be mocked; only the objects imported, after importlib is finished.
>

Oh, you can mock import statements. :)


>
> There is lots of interaction between idlelib modules, but I would still
> like direct tests of each idlelib.xyz with a test_xyz.py.  Three years
> ago, there was no test.test_idle.  There now is, and it runs 35
> idlelib.idle_test.text_* files.  (There are 60 idlelib modules.)
>
> > The problem I have with just doing manual testing for things that can
> > easily be covered by a unit test -- directly or indirectly -- is there
> > are technically 85 people who can change CPython today. That means
> > there's potentially 85 different people who can screw up the code ;) .
>
> At the moment, I am the only one pushing idlelib patches, except when it
> gets included in one of Serhiy's multi-module refactoring patches (and
> he always nosies me).  That skews my view a bit.  However, with most of
> the critical issues fixed, I am anxious to automate what I can of what I
> now do by hand.
>
> > Making sure code is exercised somehow by tests at least minimizes how
> > badly someone like me might mess something thanks to me not realizing I
> > broke the code.
>
> I had not thought about the issue that way.  I should add a test_module
> for each remaining module, import the module, and at least create an
> instance of every tkinter widget defined therein, and see what other
> classes could be easily instantiated and what functions easily run.
>

That seems like a good starting point. Kind of like test_sundry but with
class instantiation on top of it.


> > Some practical issues with coverage and CodeCov:
>
> > 2. Some statements are only intended to run on certain systems,
> making
> > 100% coverage impossible unless one carefully puts all
> system-specific
> > code in "if system == 'xyz'" statements and uses system-specific
> > .coveragerc files to exclude code for 'other' systems.
>
> > True. We could have a discussion as to whether we want to use
> > Coverage.py's pragmas ... I'm sure we could discuss things with Ned
> > Bat

Re: [python-committers] Codecov and PR

[Terry Reedy]

On 4/27/2017 3:44 PM, Brett Cannon wrote:



On Wed, 26 Apr 2017 at 22:36 Terry Reedy > wrote:


On 4/26/2017 1:45 PM, Brett Cannon wrote:



 > E.g. I don't expect
 > test_importlib to be directly responsible for exercising all code in
 > importlib, just that Python's entire test suite exercise importlib as
 > much as possible as a whole.

The advantage for importlib in this respect is that import statements
cannot be mocked; only the objects imported, after importlib is
finished.


Oh, you can mock import statements. :)


Other than by pre-loading a mock module into sys.modules?
If so, please give a hint, as this could be useful to me.


At the moment, I am the only one pushing idlelib patches, except when it
gets included in one of Serhiy's multi-module refactoring patches (and
he always nosies me).


It turns out that Louie Lu's new tool revealed a couple of other 
patches, though just to tests that started failing.



I had not thought about the issue that way.  I should add a test_module
for each remaining module, import the module, and at least create an
instance of every tkinter widget defined therein, and see what other
classes could be easily instantiated and what functions easily run.


That seems like a good starting point. Kind of like test_sundry but with 
class instantiation on top of it.


I looked and saw that bdb is in 'untested'.  I also discovered
https://bugs.python.org/issue19417
to change that, with a 3+ year-old-patch.  I plan to review it.


 > I view 100% coverage as aspirational, not attainable. But if we
want an
 > attainable goal, what should we aim for? We're at 83.44% now

On what system?



Travis, where the Codecov run is driven from.


I meant OS, because


   I suspect that Tkinter, ttk, turtle, and IDLE
GUI-dependent tests make at least a 2% difference on GUI Windows versus
no-GUI *nix.


--
Terry Jan Reedy

___
python-committers mailing list
python-committers@python.org
https://mail.python.org/mailman/listinfo/python-committers
Code of Conduct: https://www.python.org/psf/codeofconduct/