Re: Test suite in github but missing from pypi tarballs

[Edward Betts]

Here is the progress for each package.

> csvkit https://github.com/onyxfish/csvkit

Done, just waiting for a new pypi release. See
https://github.com/wireservice/csvkit/issues/20

> django-jinja   https://github.com/niwibe/django-jinja

Upstream not interested, claims the tests are hard to run.
https://github.com/niwinz/django-jinja/issues/135

> dockerpty  https://github.com/d11wtq/dockerpty

https://github.com/d11wtq/dockerpty/pull/21

> pdfrw  https://github.com/pmaupin/pdfrw

https://github.com/pmaupin/pdfrw/pull/53

> pep8-naminghttps://github.com/flintwork/pep8-naming

https://github.com/PyCQA/pep8-naming/pull/24

> pkgconfig  https://github.com/matze/pkgconfig

https://github.com/PyCQA/pep8-naming/pull/24

> pydot  https://github.com/erocarrera/pydot

https://github.com/erocarrera/pydot/pull/120

> pylast http://github.com/pylast/pylast

https://github.com/pylast/pylast/pull/169

> pysrt  https://github.com/byroot/pysrt

https://github.com/byroot/pysrt/issues/42
https://github.com/byroot/pysrt/pull/59

> python-argshttps://github.com/kennethreitz/args

https://github.com/kennethreitz/args/pull/19

> python-astor   https://github.com/berkerpeksag/astor

https://github.com/berkerpeksag/astor/pull/52

> python-cachecontrolhttps://github.com/ionrock/cachecontrol

https://github.com/ionrock/cachecontrol/pull/120

> python-easywebdav  https://github.com/amnong/easywebdav

https://github.com/amnong/easywebdav/pull/47

> python-gnutls  https://github.com/AGProjects/python-gnutls

https://github.com/AGProjects/python-gnutls/pull/1

> python-hpilo   https://github.com/seveas/python-hpilo

https://github.com/seveas/python-hpilo/pull/112

> python-humanizehttps://github.com/jmoiron/humanize

https://github.com/jmoiron/humanize/issues/33
https://github.com/jmoiron/humanize/pull/34

> python-jsonpifyhttps://github.com/wcdolphin/flask-jsonpify

https://github.com/corydolphin/flask-jsonpify/pull/13

> txws   https://github.com/MostAwesomeDude/txWS

https://github.com/MostAwesomeDude/txWS/pull/29

> vcversionerhttps://github.com/habnabit/vcversioner

https://github.com/habnabit/vcversioner/pull/7

> whichcraft https://github.com/pydanny/whichcraft

https://github.com/pydanny/whichcraft/pull/1

-- 
Edward.

Re: Test suite in github but missing from pypi tarballs

[Edward Betts]

Here is a list of packages that include a test suite on github but not in
the DPMT git.

csvkit https://github.com/onyxfish/csvkit
django-haystackhttps://github.com/toastdriven/django-haystack
django-jinja   https://github.com/niwibe/django-jinja
django-recurrence  https://github.com/django-recurrence/django-recurrence
django-webpack-loader  https://github.com/owais/django-webpack-loader
djoser https://github.com/gizmag/django-fsm-log
dockerpty  https://github.com/d11wtq/dockerpty
drf-generators https://github.com/Brobin/drf-generators
mod-wsgi   https://github.com/GrahamDumpleton/mod_wsgi
pdfrw  https://github.com/pmaupin/pdfrw
pep8-naminghttps://github.com/flintwork/pep8-naming
pkgconfig  https://github.com/matze/pkgconfig
pydot  https://github.com/erocarrera/pydot
pylast http://github.com/pylast/pylast
pysrt  https://github.com/byroot/pysrt
python-argshttps://github.com/kennethreitz/args
python-astor   https://github.com/berkerpeksag/astor
python-cachecontrolhttps://github.com/ionrock/cachecontrol
python-easywebdav  https://github.com/amnong/easywebdav
python-exifhttps://github.com/ianare/exif-py
python-gnutls  https://github.com/AGProjects/python-gnutls
python-hpilo   https://github.com/seveas/python-hpilo
python-humanizehttps://github.com/jmoiron/humanize
python-jsonpifyhttps://github.com/wcdolphin/flask-jsonpify
python-model-mommy http://github.com/vandersonmota/model_mommy
python-mrjob   http://github.com/Yelp/mrjob
python-pgspecial   https://github.com/dbcli/pgspecial
python-pip https://github.com/pypa/pip
python-pretend https://github.com/alex/pretend
python-rplyhttps://github.com/alex/rply
python-scp https://github.com/jbardin/scp.py
python-sentinels   https://github.com/vmalloc/sentinels
python-sunlighthttps://github.com/sunlightlabs/python-sunlight
python-trezor  https://github.com/trezor/python-trezor
python-zxcvbn  https://github.com/dropbox/python-zxcvbn
responses  https://github.com/getsentry/responses
subliminal https://github.com/Diaoul/subliminal
txws   https://github.com/MostAwesomeDude/txWS
vcversionerhttps://github.com/habnabit/vcversioner
whichcraft https://github.com/pydanny/whichcraft

This is the code I ran on git.debian.org to generate the list.

#!/usr/bin/python3

from subprocess import Popen, PIPE
from urllib.request import urlopen
from urllib.error import HTTPError
import os
import re

re_github_url = re.compile('(https?://[^/ ]*github.com/[^/]+/[^/ ]+)')

location = '/git/python-modules/packages/'

def repo_includes_tests(repo):
cmd = ['git', '--git-dir=' + location + repo, 'ls-tree', '-r', 'HEAD', 
'--name-only']
with Popen(cmd, stdout=PIPE, stderr=PIPE) as proc:
for line in proc.stdout:
filename = line.decode('utf-8')[:-1]
if not filename.endswith('.py'):
continue
if filename.startswith('test') or '/test' in filename:
return True
return False

def show_debian_file(repo, filename):
cmd = ['git', '--git-dir=' + location + repo, 'show', 'HEAD:debian/' + 
filename]
return Popen(cmd, stdout=PIPE, stderr=PIPE).stdout.read().decode('utf-8')

def github_urls(content):
github_urls = set()
for line in content.splitlines():
if 'github.com' in line:
m = re_github_url.search(line)
if not m:
print(line)
github_urls.add(m.group(1))
return github_urls

for repo in sorted(os.listdir(location)):
if not repo.endswith('.git') or repo_includes_tests(repo):
continue

urls = (github_urls(show_debian_file(repo, 'copyright'))
or github_urls(show_debian_file(repo, 'watch')))

if len(urls) != 1:
continue
(url,) = urls
try:
page = urlopen(url).read()
except HTTPError:
continue
if ' title="test' not in page.decode('utf-8'):
continue  # no tests in github
print('{:23s} {}'.format(repo[:-4], url))

Re: Test suite in github but missing from pypi tarballs

[Brian May]

Tristan Seligmann  writes:

> With my upstream developer hat on: source packages on PyPI are meant for
> end users to install via pip. They often include generated artifacts, and
> don't include things that aren't intended for installation via pip (tests
> being just one of these).
>
> For distribution packaging purposes, the GitHub tags are generally
> preferrable. GitHub makes archives of tagged releases available as
> tarballs, so this is generally a simple tweak to debian/watch.

Forgot to mention: In PyPI there is an application called
"check-manifest" that will check to ensure all files in git will be
included in the PyPI release, and suggest changes to MANIFEST.in.
-- 
Brian May 

Re: Test suite in github but missing from pypi tarballs

[Brian May]

Tristan Seligmann  writes:

> With my upstream developer hat on: source packages on PyPI are meant for
> end users to install via pip. They often include generated artifacts, and
> don't include things that aren't intended for installation via pip (tests
> being just one of these).
>
> For distribution packaging purposes, the GitHub tags are generally
> preferrable. GitHub makes archives of tagged releases available as
> tarballs, so this is generally a simple tweak to debian/watch.

I use to have this attitude too. However now I have reconsidered.

Far better to to retrieve source packages from PyPI.

For example, it is possible to have signed uploads to PyPI and have
uuscan automatically check the signature. This is not the case for
github.

packages that use setuptools-scm is another example - as a (over?)
simplification this requires files in the PyPI distribution, although
perhaps that is a topic best left for another discussion.
-- 
Brian May 

Re: Test suite in github but missing from pypi tarballs

[Tristan Seligmann]

Resending to the list since I accidentally replied off-list!

On Thu, 21 Apr 2016, 20:14 Tristan Seligmann, 
wrote:

> On Thu, 21 Apr 2016 at 19:34 Barry Warsaw  wrote:
>
>> I also don't particularly like relying on GitHub generated tarballs.
>> Despite
>> popular believe, not every upstream uses GH or even git, signs their tags
>> or
>> even tags their releases.  But almost *every* Python package releases
>> tarballs
>> to PyPI.
>
>
> There's no particular reason why every Python package has to distribute
> the Debian orig tarball via PyPI, though. *My* projects are almost all on
> GitHub these days, so I offer the git tags and github archives as an option
> to get the "full source"; in the past, I have hosted projects elsewhere,
> and released tarballs elsewhere, and there is of course nothing wrong with
> this.
>
> If you want to make your PyPI sdists do double-duty, there's usually
> nothing terribly wrong with this, but given that they must include
> generated artifacts (like .c source generated by Cython) in order to work
> well for pip, and pip is typically how people consume PyPI, I don't view
> these as the ideal "upstream source" for Debian.
>
> And of course, there are extreme examples like *cryptography*: The
> python-cryptography source package is 384K, the python-cryptography-vectors
> source package (necessary for running the cryptography tests) is 26MB!
>
> FWIW: I should note that I'm a fan of having my tests as a subpackage of
> the main package (ie. mything.tests) so they can be run by an end-user to
> verify that an installed copy of the package is actually working. I do this
> in most of the projects I'm upstream for; so the tests *are* included in
> my sdist, but other files are not (like documentation).
>

Re: Test suite in github but missing from pypi tarballs

[Fred Drake]

On Apr 21, 2016 13:16, "Jeremy Stanley"  wrote:
> Agreed, as long as "closely" is interpreted in ways consistent with,
> say, tarballs for C-based projects. Consider `setup.py sdist`
> similar to `make dist` where the dist target of some projects may
> still run additional commands to generate metadata or other files
> not tracked in revision control prior to invoking tar/gzip.

That's exactly what I mean.

   -Fred

Re: Test suite in github but missing from pypi tarballs

[Barry Warsaw]

On Apr 21, 2016, at 04:52 PM, Thomas Goirand wrote:

>It's best that the test suite goes within the project. So if project is
>called foo, then best is to get the test folder in foo/tests. This way,
>you don't even need to fix the MANIFEST.in.

+1 - as an upstream I always do this because I like having the tests as close
to the code its testing as possible.  It generally does no harm to include
them in the binary package so why bother, but yes in some cases you might want
to split them out.

Cheers,
-Barry

Re: Test suite in github but missing from pypi tarballs

[Barry Warsaw]

On Apr 21, 2016, at 02:54 PM, Tristan Seligmann wrote:

>With my upstream developer hat on: source packages on PyPI are meant for
>end users to install via pip. They often include generated artifacts, and
>don't include things that aren't intended for installation via pip (tests
>being just one of these).
>
>For distribution packaging purposes, the GitHub tags are generally
>preferrable. GitHub makes archives of tagged releases available as
>tarballs, so this is generally a simple tweak to debian/watch.

With my own upstream hat perched precariously on my head, I disagree.  I
generally prefer to include the test suite in the PyPI tarballs and I also
submit bug reports to upstream that don't include them.  Most have been very
accommodating, especially when you can provide a simple patch to their
MANIFEST.in.

I agree with Fred that PyPI tarballs have multiple intended consumers.

I also don't particularly like relying on GitHub generated tarballs.  Despite
popular believe, not every upstream uses GH or even git, signs their tags or
even tags their releases.  But almost *every* Python package releases tarballs
to PyPI.

There are a few very limited cases where upstreams have balked at supplying
their test suites in their PyPI tarballs.  One that I agree with is pip[*]
because it's downloaded like a jillion times a day so in that case, have a
more svelte tarball makes sense.  I think Donald and I talked about them
spinning a pip-dev tarball for distro consumers, but for now I just override
dh_auto_test in d/rules.

Cheers,
-Barry

[*] https://github.com/pypa/pip/issues/3370

Re: Test suite in github but missing from pypi tarballs

[Jeremy Stanley]

On 2016-04-21 11:23:20 -0400 (-0400), Fred Drake wrote:
> On Thu, Apr 21, 2016 at 10:54 AM, Tristan Seligmann
[...]
> > For distribution packaging purposes, the GitHub tags are generally
> > preferrable. GitHub makes archives of tagged releases available as tarballs,
> > so this is generally a simple tweak to debian/watch.
> 
> I'd generally be worried if the source package doesn't closely match a
> tag in whatever VCS a project is using, but I don't think that's
> essential, release processes being what they are.
[...]

Agreed, as long as "closely" is interpreted in ways consistent with,
say, tarballs for C-based projects. Consider `setup.py sdist`
similar to `make dist` where the dist target of some projects may
still run additional commands to generate metadata or other files
not tracked in revision control prior to invoking tar/gzip.
-- 
Jeremy Stanley

Re: Test suite in github but missing from pypi tarballs

[Thomas Goirand]

On 04/21/2016 05:20 PM, Matthias Klose wrote:
> On 21.04.2016 16:52, Thomas Goirand wrote:
>> On 04/21/2016 04:10 PM, Edward Betts wrote:
>>> Recently I've come across some Python libraries that have a test
>>> suite in
>>> their github repo but don't include it in the tarball they upload to
>>> pypi.
>>>
>>> Debian binary packages don't normally include the test suite.
>>
>> Why? It's my view that it's a good idea to include it, if it is located
>> within the lib itself.
> 
> there is no benefit to include the testsuite if it cannot be run in the
> install location.

Probably I'm too much influenced by what I do in OpenStack, where there
is not only unit tests, but also functional tests. And it makes a lot of
sense to include functional tests, so that they can be launched once the
package is installed (and connected to other services).

> Also for large testsuites a separate binary package would be preferred,

Good point! I probably should start doing that in my packages, it would
make a lot of sense to do so.

Cheers,

Thomas Goirand (zigo)

Re: Test suite in github but missing from pypi tarballs

[Fred Drake]

On Thu, Apr 21, 2016 at 11:48 AM, Ian Cordasco
 wrote:
> In the long history of both Python and its packaging this is
> absolutely true (all you need is an archive and a setup.py) but
> Python's packaging has evolved and improved for its users through
> setuptools, pip, and wheel (even though I'm sure there are many on
> this list who vehemently disagree). So tar files, for the *majority*
> of PyPI are there to help users with ancient, barely working, versions
> of pip, buildout, etc.

My reaction is really about the comment that source packages are in
PyPI "for pip". You definitely understand that it's broader than that,
but careful language is important to avoid confusing someone new,
leading to poor understanding across the community.

Some of us still see pip as a youngling upstart.  :-)

> There's also a lot of data that could be in a VCS tag that is
> orthogonal to what needs to be in a package for a user or
> redistributor. For example, few if any people are going to want a file
> that configures the (potentially closed-source) continuous integration
> service(s) that the project uses and it's inconsequential to a
> redistributor if a .gitignore is in the archive too.

Which is why I don't consider slavish use of "git archive" (or similar
from some other VCS) to be valuable.


  -Fred

-- 
Fred L. Drake, Jr.
"A storm broke loose in my mind."  --Albert Einstein

Re: Test suite in github but missing from pypi tarballs

[Ian Cordasco]

On Thu, Apr 21, 2016 at 10:23 AM, Fred Drake  wrote:
> On Thu, Apr 21, 2016 at 10:54 AM, Tristan Seligmann
>  wrote:
>> With my upstream developer hat on: source packages on PyPI are meant for end
>> users to install via pip. They often include generated artifacts, and don't
>> include things that aren't intended for installation via pip (tests being
>> just one of these).
>
> Interesting.  In many years of working with Python (since before
> either PyPI or pip), I have a slightly different perspective.
>
> Source packages (whether on PyPI or not) with a setup.py (or whatever
> comes next) are intended for both automated or developer consumption.
> pip is just one possible tool; buildout is another, and others as
> well.

In the long history of both Python and its packaging this is
absolutely true (all you need is an archive and a setup.py) but
Python's packaging has evolved and improved for its users through
setuptools, pip, and wheel (even though I'm sure there are many on
this list who vehemently disagree). So tar files, for the *majority*
of PyPI are there to help users with ancient, barely working, versions
of pip, buildout, etc.

>> For distribution packaging purposes, the GitHub tags are generally
>> preferrable. GitHub makes archives of tagged releases available as tarballs,
>> so this is generally a simple tweak to debian/watch.
>
> I'd generally be worried if the source package doesn't closely match a
> tag in whatever VCS a project is using, but I don't think that's
> essential, release processes being what they are.

There's also a lot of data that could be in a VCS tag that is
orthogonal to what needs to be in a package for a user or
redistributor. For example, few if any people are going to want a file
that configures the (potentially closed-source) continuous integration
service(s) that the project uses and it's inconsequential to a
redistributor if a .gitignore is in the archive too. Some of this will
just needlessly inflate the package size.

> This is really independent from the debate as to whether a test suite
> should be included with the package, though.

Agreed. It's just interesting.

Re: Test suite in github but missing from pypi tarballs

[Tristan Seligmann]

On Thu, 21 Apr 2016 at 16:47 Edward Betts  wrote:

> Debian binary packages don't normally include the test suite. Some Python
>
library developers are treating the pypi releases in a similar way, as if
> they're just for deployment. They think anybody who needs the test suite is
> doing development and will clone from the github repo.
>

With my upstream developer hat on: source packages on PyPI are meant for
end users to install via pip. They often include generated artifacts, and
don't include things that aren't intended for installation via pip (tests
being just one of these).

For distribution packaging purposes, the GitHub tags are generally
preferrable. GitHub makes archives of tagged releases available as
tarballs, so this is generally a simple tweak to debian/watch.

Re: Test suite in github but missing from pypi tarballs

[Thomas Goirand]

On 04/21/2016 04:10 PM, Edward Betts wrote:
> Recently I've come across some Python libraries that have a test suite in
> their github repo but don't include it in the tarball they upload to pypi.
> 
> Debian binary packages don't normally include the test suite.

Why? It's my view that it's a good idea to include it, if it is located
within the lib itself.

> Some Python
> library developers are treating the pypi releases in a similar way, as if
> they're just for deployment. They think anybody who needs the test suite is
> doing development and will clone from the github repo.
> 
> It is very useful to run the test suite when building a Debian package. It
> would be better if upstream would include the test suite in their release
> tarballs.
> 
> It is often just a one line change to MANIFEST.in to have the test suite
> included. If the tests are in a directory called 'tests', then this is the
> line that needs to added:
> 
>   recursive-include tests *.py

It's best that the test suite goes within the project. So if project is
called foo, then best is to get the test folder in foo/tests. This way,
you don't even need to fix the MANIFEST.in. Then you can even use it as
an autopkgtest thing.

> I've been sending github pull requests to encourage library authors to make
> this change.
> 
>   https://github.com/RPi-Distro/python-gpiozero/pull/284
>   https://github.com/john-kurkowski/tldextract/pull/97
> 
> You might want to take a few minutes to check any Python packages you've
> worked on, see if there is a test suite in the github repo that hasn't made it
> to the release tarball and send a pull request to fix it.

IMO, don't bother, and just re-generate a tarball from git, using "git
archive", which is my preferred way to use upstream sources.

Cheers,

Thomas Goirand (zigo)