Re: Updating Celery, Kombu, python-amqp

2017-03-13 Thread Brian May
On 2017-03-14 14:48, Christopher Hoskin wrote:

> For reasons of my own, I need to create a Celery 4.0.2 Debian package. This
> means also updating the Kombu and AMQP packages. As I'm doing this work 
> anyway,
> my preference would be to share it with the World through DPMT.
> 
> However, I notice that python-amqp has a lot of other reverse dependancies,
> including OpenStack, and that we're currently in a release freeze. I've also
> seen there's been some discussion about using the DEP14 branch/tag convention
> and switching to gbp pq.
> 
> Would people be happy for me to start updating Celery and its dependancies,
> uploading the results to experimental, or should I keep my work to myself for
> the time being?

As an uploader for celery, kombu, and python-amqp, I see no problem
myself. I can't speak for other packages, and definitely I can't speak
for packages not under DPMT. 

For now, I would suggest creating a debian/experimental branch,
switching to gbp pq (as using non-standard branch names is easier with
gbp pq), and then continuing. I have done this already for the
python-mkdocs package. 

If you need any help, let me know. 

  

Updating Celery, Kombu, python-amqp

2017-03-13 Thread Christopher Hoskin
For reasons of my own, I need to create a Celery 4.0.2 Debian package. This
means also updating the Kombu and AMQP packages. As I'm doing this work anyway,
my preference would be to share it with the World through DPMT.

However, I notice that python-amqp has a lot of other reverse dependancies,
including OpenStack, and that we're currently in a release freeze. I've also
seen there's been some discussion about using the DEP14 branch/tag convention
and switching to gbp pq.

Would people be happy for me to start updating Celery and its dependancies,
uploading the results to experimental, or should I keep my work to myself for
the time being?

Thanks.

Christopher



Re: PyPI source or github source?

2017-03-13 Thread Ghislain Vaillant
On Tue, 2017-03-14 at 08:32 +1100, Brian May wrote:
> Scott Kitterman  writes:
> 
> > Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in 
> > so 
> > that the sdist is complete when I point out the issue.
> 
> I have had some people who do argue that sdist is an installation
> format, not a source format - if you want the source use github.

Same here.

> I think most of them eventually do change, however it makes me a bit
> uneasy that maybe they might be right.

Until this thread was started, I had never questioned the
"canonicalness" of PyPI releases either. 

> As a result, some of my recent packages I have used github, rather
> then try to "fix" things on PyPI. That way I can be sure that the
> build will always be consistant, and not suddenly and unexpectedly
> drop files.

Do you get rid of the useless dotfiles (gitignore, ci settings, tox...)
or leave them alone then?

Ghis 



Re: PyPI source or github source?

2017-03-13 Thread Brian May
Scott Kitterman  writes:

> Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in 
> so 
> that the sdist is complete when I point out the issue.

I have had some people who do argue that sdist is an installation
format, not a source format - if you want the source use github. I think
most of them eventually do change, however it makes me a bit uneasy that
maybe they might be right. As a result, some of my recent packages I
have used github, rather then try to "fix" things on PyPI. That way I
can be sure that the build will always be consistant, and not suddenly
and unexpectedly drop files.
-- 
Brian May 



Re: PyPI source or github source?

2017-03-13 Thread Ole Streicher
Brian May  writes:
> Should we be using PyPI as our source of packages? Or github?

I already had packages that use a helper to generate the version
information from the git repository, and therefore need the git metadata
to build from source. I could solve this with a "fake" helper that
uses debian/changelog instead, but this looks more like a hack than a
clean solution. Unfortunately, they did't provide pypi source; otherwise
I would have used them.

Use case: 

https://github.com/spacetelescope/pysynphot

The originally "relic" package which extracts the info from git:

https://github.com/jhunkeler/relic

The "fake" relic package is in a Debian patch:

https://sources.debian.net/src/pysynphot/0.9.8.5%2Bdfsg-1/debian/patches/Don-t-use-relic-package-for-version-info.patch

Best

Ole



Re: PyPI source or github source?

2017-03-13 Thread Barry Warsaw
On Mar 14, 2017, at 08:16 AM, Brian May wrote:

>I just found another one: python-ledger. Unfortunately in this case,
>Python packages are also bundled with the C code (libraries and client).

Not entirely relevant, but pyparted also contained C code, but its mainline
does support Python 3.

-Barry


pgpJS7pdIq9jm.pgp
Description: OpenPGP digital signature


Re: PyPI source or github source?

2017-03-13 Thread Scott Kitterman
On Monday, March 13, 2017 05:55:32 PM Thomas Goirand wrote:
> On 03/12/2017 11:34 AM, Ghislain Vaillant wrote:
> > On the other hand, I have seen very few pieces of software which had a
> > *comprehensive* MANIFEST.in for generating a tarball suitable for
> > packaging. The file is often either absent, or missing inclusion of the
> > docs, tests, change log or license files. Upstream is usually receptive
> > in providing "better" source tarballs, but I have had some developers
> > taking an aggressive stance towards keeping the PyPI tarball as minimal
> > as possible in the past.
> 
> IMO, upstream are right that the PyPi releases should be minimal. They
> are, from my view point, a binary release, not a source release.
> 
> It makes a lot of sense to therefore use the git repository, which is
> what I've been doing as much as possible.

I think they are a binary release if they release a wheel, but if they put an 
sdist on pypi (and I certainly always do), it's source.  That's what the 's' 
stands for.

Like Barry, I've never had an issue with upstreams fixing their MANIFEST.in so 
that the sdist is complete when I point out the issue.

Scott K



Re: PyPI source or github source?

2017-03-13 Thread Brian May
Barry Warsaw  writes:

> I've so far only encountered one package that releases on GitHub and not PyPI:
> pyparted.  And I've filed an upstream bug about that.

I just found another one: python-ledger. Unfortunately in this case,
Python packages are also bundled with the C code (libraries and client).
This will make it hard to fix #857335 (python3 support) as upstream only
provide python3 support at the moment in an experimental branch, and
don't appear to be in any hurry to merge the experimental branch.

So I don't expect it to appear in PyPI anytime soon.
-- 
Brian May 



Re: I want to maintain my current packages within the team

2017-03-13 Thread Piotr Ożarowski
[Thomas Güttler, 2017-03-10]
> [1]http://python-modules.alioth.debian.org/policy.html
> 
> Yes, I accept the above policy.

welcome in the team :)
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645



Re: PyPI source or github source?

2017-03-13 Thread Thomas Goirand
On 03/13/2017 06:50 PM, Barry Warsaw wrote:
> Wheels should be considered a binary release, but tarballs should still be
> considered the canonical source release.

For the case of PyPi releases, they are to be used using "pip install",
which is what makes me think they are designed for this use case, and
therefore are binary releases.

> So, what do you do about upstreams that don't use git?  (I'm guessing that if
> they use git but not GitHub, your workflow doesn't break.)

Correct, any git repo is ok (and in fact, git or mecurial, thanks to the
git remote plugin). First of all, upstream not using git are the very
few minority these days, so it's very rare cases to handle, and most of
the time, it's upstream who aren't very active, so it's not really a big
problem: I just use pristine-tar.

Cheers,

Thomas Goirand (zigo)



Re: PyPI source or github source?

2017-03-13 Thread Barry Warsaw
On Mar 13, 2017, at 05:55 PM, Thomas Goirand wrote:

>IMO, upstream are right that the PyPi releases should be minimal. They
>are, from my view point, a binary release, not a source release.

Wheels should be considered a binary release, but tarballs should still be
considered the canonical source release.  OTOH, most non-geeks I know
exclusively top-post so eventually pragmatism will probably win out over
historical conventions.

>It makes a lot of sense to therefore use the git repository, which is
>what I've been doing as much as possible.

So, what do you do about upstreams that don't use git?  (I'm guessing that if
they use git but not GitHub, your workflow doesn't break.)

Cheers,
-Barry



Re: PyPI source or github source?

2017-03-13 Thread Jeremy Stanley
On 2017-03-13 17:55:32 +0100 (+0100), Thomas Goirand wrote:
[...]
> IMO, upstream are right that the PyPi releases should be minimal. They
> are, from my view point, a binary release, not a source release.
> 
> It makes a lot of sense to therefore use the git repository, which is
> what I've been doing as much as possible.

Yes, as much as the name "sdist" indicates it's a source
distribution, in many cases it's not exactly pristine source and may
be missing files deemed unimportant for end users or could include
some autogenerated files the upstream authors would rather not check
into their revision control systems. So sdists, while a tarball
under the hood (and by filename extension), are still really an
installable packaging format more than they are a source
distribution format.
-- 
Jeremy Stanley



Re: PyPI source or github source?

2017-03-13 Thread Thomas Goirand
On 03/12/2017 11:34 AM, Ghislain Vaillant wrote:
> On the other hand, I have seen very few pieces of software which had a
> *comprehensive* MANIFEST.in for generating a tarball suitable for
> packaging. The file is often either absent, or missing inclusion of the
> docs, tests, change log or license files. Upstream is usually receptive
> in providing "better" source tarballs, but I have had some developers
> taking an aggressive stance towards keeping the PyPI tarball as minimal
> as possible in the past.

IMO, upstream are right that the PyPi releases should be minimal. They
are, from my view point, a binary release, not a source release.

It makes a lot of sense to therefore use the git repository, which is
what I've been doing as much as possible.

Cheers,

Thomas Goirand (zigo)



Re: PyPI source or github source?

2017-03-13 Thread Barry Warsaw
On Mar 12, 2017, at 01:19 PM, Brian May wrote:

>Should we be using PyPI as our source of packages? Or github?

PyPI for two reasons: not every developer uses git, and even those that do may
not use GitHub.

I've so far only encountered one package that releases on GitHub and not PyPI:
pyparted.  And I've filed an upstream bug about that.

>However, often packages in PyPI are not the complete source package you
>would get from github. They may be sufficient for installations, but
>often not for Debian packaging - which really should have the complete
>source package. e.g. they can be missing tests, license files,
>documentation that doesn't get installed, etc.

Yep, and I always file a bug when that happens.  Sometimes I include the file
in debian/ and add rules to move them in place.  Usually the missing file is
just caused by a MANIFEST.in that's out of date or some such (heck, I've done
that myself).  But FWIW, I've yet to encounter an upstream that hasn't been
very helpful in fixing their releases to help downstreams like Debian.

Cheers,
-Barry



Re: GnuPG signatures on PyPI: why so few?

2017-03-13 Thread Barry Warsaw
On Mar 12, 2017, at 11:46 AM, Ben Finney wrote:

>What prospect is there in the Python community to get signed upstream
>releases become the obvious norm?

I don't know.  Digital security seems to be mostly an afterthought
unfortunately.  I always use `twine upload --sign` so all my projects have
signatures, and for those where I'm also the Debian maintainer or primary
uploader, I try to enable signatures for uscan, but it seems oddly
self-serving. ;)

-Barry



Re: GnuPG signatures on PyPI: why so few?

2017-03-13 Thread Hans-Christoph Steiner


Donald Stufft:
> 
>> On Mar 12, 2017, at 10:35 PM, Paul Wise  wrote:
>>
>> On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:
>>
>>> upload them to PyPI since the authors of the coming Warehouse
>>> replacement for the current CheeseShop PyPI have already indicated
>>> that they intend to drop support for signatures entirely.
>>
>> Did they give any reasoning for this decision?
>>
> 
> 
> https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 
> 
> 
> —
> Donald Stufft

Your analysis makes sense: the hardest part of this problem is managing
_who_ is the trusted signer of a given package.  TOFU
(Trust-On-First-Use) is the easy case here: the first person to sign the
release is the one to trust.  Android's APK signature verification for
updates is entirely based on that principal.

It seems that so many Python and Java library developers (mavenCentral
is in a similar boat to PyPi) just want to ignore the signing key
management because its annoying.  Yes, it is annoying, I do a lot of it.
 Android forces all devs to manage a signing key: unsigned APKs are not
installable, and Google Play even rejects APKs signed by debug keys.

I think the Android/Google Play model would work quite well for PyPI,
but the social problem of getting python devs to take responsibility
would be the hard part.  I manages the transition of F-Droid app repos
to only signed, which is a small example.  There were some complaints,
but nothing so bad when we pointed them to the fdroid tool that
automatically generated the signing key for them.  pip can easily
generate and use a signing key (GPG, TUF, whatever) automatically.  Then
devs just need to learn out to safely manage them.  Android Studio's key
generation is pretty new, and has kind of shitty security
recommendations.  Android devs are still learning, but I think overall,
its working out well there.

As for TUF (The Update Framework), I know the concepts well.  I've
looked into it while implementing the F-Droid security model
(https://f-droid.org/wiki/page/Security_Model).  It does not deal with
the issue of managing identity of package uploaders at all, it assumes
you already have all the software in place, and just deals with
distributing updates.  Also, with TUF, you'll be in the exact same boat
here: signing keys are required per package stream, so each developer
will need to managing signing keys.

.hc



Re: I want to maintain my current packages within the team

2017-03-13 Thread Thomas Güttler

What is the next step now?

Regards,
  Thomas Güttler

Am 10.03.2017 um 09:27 schrieb Thomas Güttler:

Hi,

I am following this instruction: 
http://python-modules.alioth.debian.org/policy.html

Yes, I accept the above policy.

Yes, collaborative maintenance is preferred.

Here is what I want to package: https://github.com/guettli/reprec

Up to now reprec contains these tools:

  * reprec: Replace strings in text files. Can work recursive in a 
directory tree
  * setops: Set operations (union, intersection, ...) for line based files.


Acoording to the policy I should include my alioth login. Up to now I have non.

What is the next stop now? If possible the username should be "guettli".

Thank you,
 Thomas Güttler

--
Thomas Guettler http://www.thomas-guettler.de/



--
Thomas Guettler http://www.thomas-guettler.de/