Re: Autopkgtest smoke test for Python libraries

[Ben Finney]

Yaroslav Halchenko  writes:

> sorry if I came late to the discussion and missing the point, but
> pybuild/dh-python already has some tests discovery mechanisms

Have another read of the thread: this is not about running the upstream
test suite.

It is about making a standard “smoke test”, i.e. a simple “is this
package actually installed?” test.

Now that the code exists and seems to be meeting with approval, I'd like
it to be part of an appropriate Debhelper step or some other equivalent
“add this to the package while building it” hook.

-- 
 \  “Immorality: The morality of those who are having a better |
  `\  time.” —Henry L. Mencken |
_o__)  |
Ben Finney

Re: Autopkgtest smoke test for Python libraries

[Paul Wise]

On Wed, Mar 2, 2016 at 6:53 AM, Ben Finney wrote:

> Who can recommend a way to make this a more automated part of the Debian
> Python package build workflow?

Add things that should be changed to `cme fix` or pypi2deb

https://wiki.debian.org/AutomaticPackagingTools

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: Autopkgtest smoke test for Python libraries

[Ben Finney]

Martin Pitt  writes:

> Thanks to Matthias for pointing this out to me. I try to reply
> in-thread. Please keep CC:ing me as I'm not subscribed.

Done.

> > The goal is to perform a simple “smoke test” to verify the correct
> > installation of the Python libraries from the Debian packages, for all
> > relevant Python interpreters.
> > 
> > A secondary goal is to have the actual smoke test code be re-usable,
> > possibly incorporated into a standard Debian Python packager toolkit.
>
> Very nice! There's precedent for Perl, Ruby and DKMS packages which
> have a fairly standard way to run the upstream test suite. For Python
> there are some conventions, like "./setup.py test" or running
> nosetests, maybe it's worth experimenting with those after getting the
> initial "import works" smoketest in place.

I'll limit this work to just a smoke test, whatever that comes to mean
through consensus here.

As discussed elsewhere in this thread, I think running upstream's test
suite is beyond the scope of this simple smoke test. The autopkgtest
framework will allow adding more tests if someone else wants to add
them.

> The "generate" script [from ‘autodep8’] could iterate over all binary
> packages of the source and pick out the python-* or the python3-*
> ones, respectively.

Thanks, I will look into that.

> >   These scripts are very simple, but not simple enough that I was
> >   comfortable stuffing them directly into the ‘debian/tests/control’
> >   file. Is that a shortcoming of the Autopkgtest specification?
>
> Not sure what you mean with "shortcoming", you can put arbitrary shell
> code into Test-Command:.

It becomes very messy when there are more than a line or so; and the
test becomes anonymous in the build output.

That led me to add the extra indirection layer of some simple named
programs; but I'd prefer if that indirection layer could be removed
while keeping the test names and clean look of the code.

> That seems like an excellent starting point indeed.

Thanks for the positive feedback.

This belongs IMO in an existing “build Debian packages for Python” tool.
As it stands, I'll need to manually add a bundle of files to every
Python package I maintain.

Who can recommend a way to make this a more automated part of the Debian
Python package build workflow?

-- 
 \   “I have always wished for my computer to be as easy to use as |
  `\   my telephone; my wish has come true because I can no longer |
_o__)  figure out how to use my telephone.” —Bjarne Stroustrup |
Ben Finney

Re: Autopkgtest smoke test for Python libraries

[Barry Warsaw]

On Mar 01, 2016, at 06:10 PM, Martin Pitt wrote:

>You could use it for that purpose indeed, but that's not really what
>it is intended for. The idea is that the generic tests apply to all
>packages of a particular type, so you can change them centrally
>instead of having to modify and upload hundreds of packages.
>
>There is one more thing, though: The test machinery needs to be able
>to discover that a package has an autodep8 test (without having the
>source package already available, as getting and checking that is very
>expensive). So ideally all source packages which want to use that will
>get a "Testsuite: autopkgtest-pkg-python" header in debian/control,
>like for example libnet-oauth-perl. However, until these get uploaded
>we can add some custom code to debci to recognize those packages based
>on the name and dependencies (that's what we did for bootstrapping the
>perl and ruby autodep8 tests).

Ah cool, this is what I didn't understand from reading the README.  I was sort
of looking for the equivalent of README.package-tests.rst, i.e. what do I have
to do to my packages to opt-in to autodep8?  Or on the flip side, what do we
need to do globally to just enable it for all packages (where "all" == 80/20
of course :).

>As I said, running upstream tests works surprisingly well for
>Perl/Ruby, we had about 80% success rate. But they are a bit more
>rigid in structure apparently.

One of the problems is the wide variety of ways for invoking a package's test
suite in Python.  I've been pushing at those edges for a long time.  Ideally I
think the packaging metadata specs should define that, and it's on their
radar, just not any time soon.  My personal favorite is to look for a tox.ini
file and then just invoke `tox` but there are a few upstream features I still
want to make that really work nicely.  (Still, I encourage all upstream Python
authors to look closely at tox, it pretty much rules. :)

pybuild has a bunch of heuristics for trying to figure out how to invoke a
package's test suite, so that could be code/ideas that we could steal or
share.

>But indeed, ensuring that your module still imports already says a
>lot. New dependencies can still break your module in subtle ways, but
>at least things like new/removed Python versions, linker errors, wrong
>paths etc. are spotted.

Yep.  I think that's clearly the first step.

Cheers,
-Barry


pgpa5fEUGQypS.pgp
Description: OpenPGP digital signature

Re: Autopkgtest smoke test for Python libraries

[Martin Pitt]

Hey Barry,

Barry Warsaw [2016-03-01 10:48 -0500]:
> I gather that it's supposed to be used by the maintainer in a source package
> ($vcs repo) to jumpstart a debian/tests directory?

You could use it for that purpose indeed, but that's not really what
it is intended for. The idea is that the generic tests apply to all
packages of a particular type, so you can change them centrally
instead of having to modify and upload hundreds of packages.

There is one more thing, though: The test machinery needs to be able
to discover that a package has an autodep8 test (without having the
source package already available, as getting and checking that is very
expensive). So ideally all source packages which want to use that will
get a "Testsuite: autopkgtest-pkg-python" header in debian/control,
like for example libnet-oauth-perl. However, until these get uploaded
we can add some custom code to debci to recognize those packages based
on the name and dependencies (that's what we did for bootstrapping the
perl and ruby autodep8 tests).

> I'm not sure how useful things like "setup.py test" are for DEP-8.  In general
> I think a package's test suite are best exercised during package build[*].

During package build is of course good, but one of the points of
autopkgtest is to run the tests when one of your dependencies change
(and thus potentially break you).

> For the majority of Python packages, I view DEP-8 as ensuring that the user
> experience is what they expect, and that means at a minimum that the packages
> import and Do Something Simple.  A generic proof of non-broken import can mean
> printing the package's version or the module path, and that's something that
> autopkg8 could templatize.  Anything more than that requires some more
> detailed knowledge about what the package does.

As I said, running upstream tests works surprisingly well for
Perl/Ruby, we had about 80% success rate. But they are a bit more
rigid in structure apparently.

But indeed, ensuring that your module still imports already says a
lot. New dependencies can still break your module in subtle ways, but
at least things like new/removed Python versions, linker errors, wrong
paths etc. are spotted.

Martin
-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: PGP signature

Re: Autopkgtest smoke test for Python libraries

[Barry Warsaw]

On Mar 01, 2016, at 09:05 AM, Martin Pitt wrote:

>Very nice! There's precedent for Perl, Ruby and DKMS packages which
>have a fairly standard way to run the upstream test suite. For Python
>there are some conventions, like "./setup.py test" or running
>nosetests, maybe it's worth experimenting with those after getting the
>initial "import works" smoketest in place.
>
>This situation (testing many related packages in a similar way) is
>what autodep8 is for. When run in a Debian source package tree, it
>checks if it knows something about this type of package, and if so,
>synthesizes a debian/tests/control. Have a look at

I didn't know about autodep8, thanks for the link!

I gather that it's supposed to be used by the maintainer in a source package
($vcs repo) to jumpstart a debian/tests directory?  But after that, would you
use autodep8 again?

I'll try to clear some time to look at adding Python support; shouldn't be too
difficult I think.

I'm not sure how useful things like "setup.py test" are for DEP-8.  In general
I think a package's test suite are best exercised during package build[*].
For the majority of Python packages, I view DEP-8 as ensuring that the user
experience is what they expect, and that means at a minimum that the packages
import and Do Something Simple.  A generic proof of non-broken import can mean
printing the package's version or the module path, and that's something that
autopkg8 could templatize.  Anything more than that requires some more
detailed knowledge about what the package does.

I've tended to taking a pretty dumb approach since I like using Test-Command,
but Ben's smoketests in lockfile are certainly more thorough.  For an autodep8
plugin, I'd probably fall somewhere in between.

>Not sure what you mean with "shortcoming", you can put arbitrary shell
>code into Test-Command:. However, the spirit is that these should be
>very short, usually one line. I suggest that they should mostly just
>call something like "/usr/share/python-something/run-autopkgtest $py $pkgname"
>and you maintain the actual test code centrally in e. g.
>python-defaults.

Oh, that's a good idea.  Then when we come up with better generic tests, we
only need to update one package, not everything.

>That seems like an excellent starting point indeed. Making sure that
>all modules are importable will catch missing builds for particular
>Python versions, regressions in newer Python versions, missing
>dependencies, packaging errors, etc.

+1

>As said above, maybe in the future we want to try and run upstream test
>suites, but this will need a lot more heuristics (starting with additional
>test deps).

I'm skeptical about running the upstream test suite in DEP-8, but baby steps!

Cheers,
-Barry

[*] Although it's true for a small handful of packages, it's not feasible to
run their test suite during package build for various reasons.  In those cases
I've sometimes added DEP-8 tests for running the test suite, but I've also
found those to be rather fragile, so I'm on the fence about gating
e.g. package promotion based on their passing their own test suite.


pgpFtylcbWZXz.pgp
Description: OpenPGP digital signature

Re: Autopkgtest smoke test for Python libraries

[Martin Pitt]

Hello all,

Thanks to Matthias for pointing this out to me. I try to reply
in-thread. Please keep CC:ing me as I'm not subscribed.

> The goal is to perform a simple “smoke test” to verify the correct
> installation of the Python libraries from the Debian packages, for all
> relevant Python interpreters.
> 
> A secondary goal is to have the actual smoke test code be re-usable,
> possibly incorporated into a standard Debian Python packager toolkit.

Very nice! There's precedent for Perl, Ruby and DKMS packages which
have a fairly standard way to run the upstream test suite. For Python
there are some conventions, like "./setup.py test" or running
nosetests, maybe it's worth experimenting with those after getting the
initial "import works" smoketest in place.

This situation (testing many related packages in a similar way) is
what autodep8 is for. When run in a Debian source package tree, it
checks if it knows something about this type of package, and if so,
synthesizes a debian/tests/control. Have a look at

  http://anonscm.debian.org/cgit/collab-maint/autodep8.git/tree/

which has a README, and the "detect" and "generate" scripts in
support/.

> * debian/tests/control
> 
>   Specifies two tests to run, ‘smoke-python2’ and ‘smoke-python3’.
> 
>   The dependencies explicitly specify the Python 2 and Python 3 library
>   modules respectively. Can this be automated better?

The "generate" script could iterate over all binary packages of the
source and pick out the python-* or the python3-* ones, respectively.

> * debian/tests/smoke-python2, debian/tests/smoke-python3
> 
>   These are Bash programs that just run a Python module under every
>   available Python interpreter.
> 
>   Running ‘py{,3}versions -i’ gets all the installed Python
>   interpreters. Can this fail, e.g. can the set of interpreters be
>   different from what interpreters the package was built for?

Yes, it can, and that's IMHO a situation where you *want* the test to
fail, as your package is actually broken (or at least in need of a
binNMU).

>   These scripts are very simple, but not simple enough that I was
>   comfortable stuffing them directly into the ‘debian/tests/control’
>   file. Is that a shortcoming of the Autopkgtest specification?

Not sure what you mean with "shortcoming", you can put arbitrary shell
code into Test-Command:. However, the spirit is that these should be
very short, usually one line. I suggest that they should mostly just
call something like "/usr/share/python-something/run-autopkgtest $py $pkgname"
and you maintain the actual test code centrally in e. g.
python-defaults.

> * debian/tests/smoke_test.py
> 
>   A module that runs a simple suite of smoke tests: emit the interpreter
>   description, fetch the Python distribution of the installed library
>   and emit that, import the modules and emit those.
> 
>   Implemented as a command-line program so that the distribution name,
>   and the set of module names, are specified by the caller. This
>   hopefully makes ‘smoke_test.py’ reuseable for many Python library
>   packages.
> 
>   Does this module meet your idea of a simple “smoke test” for Python
>   library packages? Should it do more?

That seems like an excellent starting point indeed. Making sure that
all modules are importable will catch missing builds for particular
Python versions, regressions in newer Python versions, missing
dependencies, packaging errors, etc. As said above, maybe in the
future we want to try and run upstream test suites, but this will need
a lot more heuristics (starting with additional test deps).

Thanks,

Martin

-- 
Martin Pitt| http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)


signature.asc
Description: PGP signature

Autopkgtest smoke test for Python libraries

[Ben Finney]

Howdy all,

I am experimenting with Autopkgtest[0] for Python library packages.
Perhaps others can use what I've made and/or make suggestions for
improvement.

The ‘python-lockfile’ Debian packaging VCS now has a smoke test suite
.

The goal is to perform a simple “smoke test” to verify the correct
installation of the Python libraries from the Debian packages, for all
relevant Python interpreters.

A secondary goal is to have the actual smoke test code be re-usable,
possibly incorporated into a standard Debian Python packager toolkit.

* debian/tests/control

  Specifies two tests to run, ‘smoke-python2’ and ‘smoke-python3’.

  The dependencies explicitly specify the Python 2 and Python 3 library
  modules respectively. Can this be automated better?

* debian/tests/smoke-python2, debian/tests/smoke-python3

  These are Bash programs that just run a Python module under every
  available Python interpreter.

  Running ‘py{,3}versions -i’ gets all the installed Python
  interpreters. Can this fail, e.g. can the set of interpreters be
  different from what interpreters the package was built for?

  These scripts are very simple, but not simple enough that I was
  comfortable stuffing them directly into the ‘debian/tests/control’
  file. Is that a shortcoming of the Autopkgtest specification?

* debian/tests/smoke_test.py

  A module that runs a simple suite of smoke tests: emit the interpreter
  description, fetch the Python distribution of the installed library
  and emit that, import the modules and emit those.

  Implemented as a command-line program so that the distribution name,
  and the set of module names, are specified by the caller. This
  hopefully makes ‘smoke_test.py’ reuseable for many Python library
  packages.

  Does this module meet your idea of a simple “smoke test” for Python
  library packages? Should it do more?


Is there interest in getting this work into one of the Debian packaging
tookits for Python packages? How should that be done?


[0] Doesn't seem to have a coherent website or documentation index. Try
.

-- 
 \ “The Vatican is not a state.… a state must have territory. This |
  `\ is a palace with gardens, about as big as an average golf |
_o__) course.” —Geoffrey Robertson, 2010-09-18 |
Ben Finney