Re: PyPI source or github source?

[Brian May]

Thomas Goirand  writes:

> IMO, upstream are right that the PyPi releases should be minimal. They
> are, from my view point, a binary release, not a source release.

To go against this, often PyPI releases contain *.pyc files. Looking at
django-filter 0.13.0 right now.
-- 
Brian May 

Re: PyPI source or github source?

[Thomas Goirand]

On 03/13/2017 10:40 PM, Ghislain Vaillant wrote:
> Do you get rid of the useless dotfiles (gitignore, ci settings, tox...)
> or leave them alone then?

It is a battle that needs a lot of efforts for not so many rewards, so I
just leave them alone. However, I could probably just add a few
parameters to the "git archive" command within openstack-pkg-tools, so
that it would *always* get rid of them.

The issue I had was, however, with the .gitreview. Indeed, since the
packaging of OpenStack also uses Gerrit, the .gitreview file is needed.
That's not a problem with Stretch (since it ignores it by default), but
it was when backporting to Jessie. The solution is to add:

extend-diff-ignore = "^[.]gitreview$"

in debian/source/options, so that there's no error with diff in that
file when building (this isn't needed for >= Stretch).

Apart from that, no problem with other upstream files.

Cheers,

Thomas Goirand (zigo)

Re: PyPI source or github source?

[Brian May]

Ghislain Vaillant  writes:

> Do you get rid of the useless dotfiles (gitignore, ci settings, tox...)
> or leave them alone then?

So far .. they have never caused any problems. So leave them as is.

Think you can leave tox, that can still be useful.
-- 
Brian May 

Re: PyPI source or github source?

[Ghislain Vaillant]

On Tue, 2017-03-14 at 08:32 +1100, Brian May wrote:
> Scott Kitterman  writes:
> 
> > Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in 
> > so 
> > that the sdist is complete when I point out the issue.
> 
> I have had some people who do argue that sdist is an installation
> format, not a source format - if you want the source use github.

Same here.

> I think most of them eventually do change, however it makes me a bit
> uneasy that maybe they might be right.

Until this thread was started, I had never questioned the
"canonicalness" of PyPI releases either. 

> As a result, some of my recent packages I have used github, rather
> then try to "fix" things on PyPI. That way I can be sure that the
> build will always be consistant, and not suddenly and unexpectedly
> drop files.

Do you get rid of the useless dotfiles (gitignore, ci settings, tox...)
or leave them alone then?

Ghis 

Re: PyPI source or github source?

[Brian May]

Scott Kitterman  writes:

> Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in 
> so 
> that the sdist is complete when I point out the issue.

I have had some people who do argue that sdist is an installation
format, not a source format - if you want the source use github. I think
most of them eventually do change, however it makes me a bit uneasy that
maybe they might be right. As a result, some of my recent packages I
have used github, rather then try to "fix" things on PyPI. That way I
can be sure that the build will always be consistant, and not suddenly
and unexpectedly drop files.
-- 
Brian May 

Re: PyPI source or github source?

[Ole Streicher]

Brian May  writes:
> Should we be using PyPI as our source of packages? Or github?

I already had packages that use a helper to generate the version
information from the git repository, and therefore need the git metadata
to build from source. I could solve this with a "fake" helper that
uses debian/changelog instead, but this looks more like a hack than a
clean solution. Unfortunately, they did't provide pypi source; otherwise
I would have used them.

Use case: 

https://github.com/spacetelescope/pysynphot

The originally "relic" package which extracts the info from git:

https://github.com/jhunkeler/relic

The "fake" relic package is in a Debian patch:

https://sources.debian.net/src/pysynphot/0.9.8.5%2Bdfsg-1/debian/patches/Don-t-use-relic-package-for-version-info.patch

Best

Ole

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 14, 2017, at 08:16 AM, Brian May wrote:

>I just found another one: python-ledger. Unfortunately in this case,
>Python packages are also bundled with the C code (libraries and client).

Not entirely relevant, but pyparted also contained C code, but its mainline
does support Python 3.

-Barry


pgpJS7pdIq9jm.pgp
Description: OpenPGP digital signature

Re: PyPI source or github source?

[Scott Kitterman]

On Monday, March 13, 2017 05:55:32 PM Thomas Goirand wrote:
> On 03/12/2017 11:34 AM, Ghislain Vaillant wrote:
> > On the other hand, I have seen very few pieces of software which had a
> > *comprehensive* MANIFEST.in for generating a tarball suitable for
> > packaging. The file is often either absent, or missing inclusion of the
> > docs, tests, change log or license files. Upstream is usually receptive
> > in providing "better" source tarballs, but I have had some developers
> > taking an aggressive stance towards keeping the PyPI tarball as minimal
> > as possible in the past.
> 
> IMO, upstream are right that the PyPi releases should be minimal. They
> are, from my view point, a binary release, not a source release.
> 
> It makes a lot of sense to therefore use the git repository, which is
> what I've been doing as much as possible.

I think they are a binary release if they release a wheel, but if they put an 
sdist on pypi (and I certainly always do), it's source.  That's what the 's' 
stands for.

Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in so 
that the sdist is complete when I point out the issue.

Scott K

Re: PyPI source or github source?

[Brian May]

Barry Warsaw  writes:

> I've so far only encountered one package that releases on GitHub and not PyPI:
> pyparted.  And I've filed an upstream bug about that.

I just found another one: python-ledger. Unfortunately in this case,
Python packages are also bundled with the C code (libraries and client).
This will make it hard to fix #857335 (python3 support) as upstream only
provide python3 support at the moment in an experimental branch, and
don't appear to be in any hurry to merge the experimental branch.

So I don't expect it to appear in PyPI anytime soon.
-- 
Brian May 

Re: PyPI source or github source?

[Thomas Goirand]

On 03/13/2017 06:50 PM, Barry Warsaw wrote:
> Wheels should be considered a binary release, but tarballs should still be
> considered the canonical source release.

For the case of PyPi releases, they are to be used using "pip install",
which is what makes me think they are designed for this use case, and
therefore are binary releases.

> So, what do you do about upstreams that don't use git?  (I'm guessing that if
> they use git but not GitHub, your workflow doesn't break.)

Correct, any git repo is ok (and in fact, git or mecurial, thanks to the
git remote plugin). First of all, upstream not using git are the very
few minority these days, so it's very rare cases to handle, and most of
the time, it's upstream who aren't very active, so it's not really a big
problem: I just use pristine-tar.

Cheers,

Thomas Goirand (zigo)

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 13, 2017, at 05:55 PM, Thomas Goirand wrote:

>IMO, upstream are right that the PyPi releases should be minimal. They
>are, from my view point, a binary release, not a source release.

Wheels should be considered a binary release, but tarballs should still be
considered the canonical source release.  OTOH, most non-geeks I know
exclusively top-post so eventually pragmatism will probably win out over
historical conventions.

>It makes a lot of sense to therefore use the git repository, which is
>what I've been doing as much as possible.

So, what do you do about upstreams that don't use git?  (I'm guessing that if
they use git but not GitHub, your workflow doesn't break.)

Cheers,
-Barry

Re: PyPI source or github source?

[Jeremy Stanley]

On 2017-03-13 17:55:32 +0100 (+0100), Thomas Goirand wrote:
[...]
> IMO, upstream are right that the PyPi releases should be minimal. They
> are, from my view point, a binary release, not a source release.
> 
> It makes a lot of sense to therefore use the git repository, which is
> what I've been doing as much as possible.

Yes, as much as the name "sdist" indicates it's a source
distribution, in many cases it's not exactly pristine source and may
be missing files deemed unimportant for end users or could include
some autogenerated files the upstream authors would rather not check
into their revision control systems. So sdists, while a tarball
under the hood (and by filename extension), are still really an
installable packaging format more than they are a source
distribution format.
-- 
Jeremy Stanley

Re: PyPI source or github source?

[Thomas Goirand]

On 03/12/2017 11:34 AM, Ghislain Vaillant wrote:
> On the other hand, I have seen very few pieces of software which had a
> *comprehensive* MANIFEST.in for generating a tarball suitable for
> packaging. The file is often either absent, or missing inclusion of the
> docs, tests, change log or license files. Upstream is usually receptive
> in providing "better" source tarballs, but I have had some developers
> taking an aggressive stance towards keeping the PyPI tarball as minimal
> as possible in the past.

IMO, upstream are right that the PyPi releases should be minimal. They
are, from my view point, a binary release, not a source release.

It makes a lot of sense to therefore use the git repository, which is
what I've been doing as much as possible.

Cheers,

Thomas Goirand (zigo)

Re: PyPI source or github source?

[Barry Warsaw]

On Mar 12, 2017, at 01:19 PM, Brian May wrote:

>Should we be using PyPI as our source of packages? Or github?

PyPI for two reasons: not every developer uses git, and even those that do may
not use GitHub.

I've so far only encountered one package that releases on GitHub and not PyPI:
pyparted.  And I've filed an upstream bug about that.

>However, often packages in PyPI are not the complete source package you
>would get from github. They may be sufficient for installations, but
>often not for Debian packaging - which really should have the complete
>source package. e.g. they can be missing tests, license files,
>documentation that doesn't get installed, etc.

Yep, and I always file a bug when that happens.  Sometimes I include the file
in debian/ and add rules to move them in place.  Usually the missing file is
just caused by a MANIFEST.in that's out of date or some such (heck, I've done
that myself).  But FWIW, I've yet to encounter an upstream that hasn't been
very helpful in fixing their releases to help downstreams like Debian.

Cheers,
-Barry

Re: PyPI source or github source?

[Ghislain Vaillant]

On Sun, 2017-03-12 at 10:53 +0800, Paul Wise wrote:
> On Sun, Mar 12, 2017 at 10:19 AM, Brian May wrote:
> 
> > Sure, you could argue that PyPI source packages should contain
> > everything the github package does. In fact there is a PyPI tool to help
> > get the MANIFEST.in correct for such purposes -
> > https://pypi.python.org/pypi/check-manifest
> 
> Anyone interested in packaging this?

There is an RFP filed for it [1]. I could have a look at it. I recently
found out that this tool is listed in the PyPA sample project [2].

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734117
[2] https://github.com/pypa/sampleproject/blob/master/setup.py

> > Unfortunately, github releases cannot (AFAIK) easily be signed, unless
> > you retrieve signed git tags directly from git (which is not supported
> > by uscan AFAIK). Would be good if gbp buildpackage supported signing git
> > tags, I don't think it does either.
> 
> uscan does support git but doesn't check OpenPGP signatures on tags.
> It would probably be easy to add that, please file a bug about it.
> 
> > * Do we consider signed git tags / commits secure, considering they are
> >   based on SHA1?
> 
> Better than having unsigned tags/commits.
> 
> > * Is there any point having signed PyPI releases when (very likely) the
> >   underlying upstream git repository has no signatures?
> 
> Yes, presumably the PyPI releases are built from the author's copy of
> the git repository, rather than directly from the online repository,
> hopefully they have verified all commits they pulled into it.
> 
> > * Is there any point having signed PyPI releases when (very likely) the
> >   public key is stored in an insecure DPMT respository on
> >   git.debian.org?
> 
> Yes, it is also stored in immutable places like the archive and snapshot.d.o.

My personal gripe with GitHub releases is that they are often full of
unwanted stuff, such as various CI config files (travis, circle,
appveyor...), test config files (pytest, nose, tox...), conda scripts,
GitHub files (.github/ directory) and whatnot. They are not harmful but
it is clutter the distributed sources could definitely live without.

On the other hand, I have seen very few pieces of software which had a
*comprehensive* MANIFEST.in for generating a tarball suitable for
packaging. The file is often either absent, or missing inclusion of the
docs, tests, change log or license files. Upstream is usually receptive
in providing "better" source tarballs, but I have had some developers
taking an aggressive stance towards keeping the PyPI tarball as minimal
as possible in the past.

Ghis

Re: PyPI source or github source?

[Paul Wise]

On Sun, Mar 12, 2017 at 10:19 AM, Brian May wrote:

> Sure, you could argue that PyPI source packages should contain
> everything the github package does. In fact there is a PyPI tool to help
> get the MANIFEST.in correct for such purposes -
> https://pypi.python.org/pypi/check-manifest

Anyone interested in packaging this?

> Unfortunately, github releases cannot (AFAIK) easily be signed, unless
> you retrieve signed git tags directly from git (which is not supported
> by uscan AFAIK). Would be good if gbp buildpackage supported signing git
> tags, I don't think it does either.

uscan does support git but doesn't check OpenPGP signatures on tags.
It would probably be easy to add that, please file a bug about it.

> * Do we consider signed git tags / commits secure, considering they are
>   based on SHA1?

Better than having unsigned tags/commits.

> * Is there any point having signed PyPI releases when (very likely) the
>   underlying upstream git repository has no signatures?

Yes, presumably the PyPI releases are built from the author's copy of
the git repository, rather than directly from the online repository,
hopefully they have verified all commits they pulled into it.

> * Is there any point having signed PyPI releases when (very likely) the
>   public key is stored in an insecure DPMT respository on
>   git.debian.org?

Yes, it is also stored in immutable places like the archive and snapshot.d.o.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

PyPI source or github source?

[Brian May]

Ben Finney  writes:

> Debian's UScan has the ability to find, download, and verify the GnuPG
> signature for a package source release. Lintian will remind the
> maintainer if a Debian source package is not taking advantage of this.

A related issue is:

Should we be using PyPI as our source of packages? Or github?

You mention a very good reason why PyPI should be the preferred form,
packages *can* be signed by the author.

However, often packages in PyPI are not the complete source package you
would get from github. They may be sufficient for installations, but
often not for Debian packaging - which really should have the complete
source package. e.g. they can be missing tests, license files,
documentation that doesn't get installed, etc.

Sure, you could argue that PyPI source packages should contain
everything the github package does. In fact there is a PyPI tool to help
get the MANIFEST.in correct for such purposes -
https://pypi.python.org/pypi/check-manifest; however a counter argument
could also be made that upstrems generally don't care about such issues
- probably won't notice any regressions - if it installs from PyPI that
is all that really matters - and the only way we can be sure we are
getting the complete source iis from github.

Unfortunately, github releases cannot (AFAIK) easily be signed, unless
you retrieve signed git tags directly from git (which is not supported
by uscan AFAIK). Would be good if gbp buildpackage supported signing git
tags, I don't think it does either.

I started off thinking github was the best source, then slowly swayed
towards PyPI, now I am thinking maybe github again.

Comments?

Some related issues:

* Do we consider signed git tags / commits secure, considering they are
  based on SHA1?

* Is there any point having signed PyPI releases when (very likely) the
  underlying upstream git repository has no signatures?

* Is there any point having signed PyPI releases when (very likely) the
  public key is stored in an insecure DPMT respository on
  git.debian.org?
-- 
Brian May