Re: GnuPG signatures on PyPI: why so few?

[Barry Warsaw]

On Mar 12, 2017, at 11:46 AM, Ben Finney wrote:

>What prospect is there in the Python community to get signed upstream
>releases become the obvious norm?

I don't know.  Digital security seems to be mostly an afterthought
unfortunately.  I always use `twine upload --sign` so all my projects have
signatures, and for those where I'm also the Debian maintainer or primary
uploader, I try to enable signatures for uscan, but it seems oddly
self-serving. ;)

-Barry

Re: GnuPG signatures on PyPI: why so few?

[Hans-Christoph Steiner]

Donald Stufft:
> 
>> On Mar 12, 2017, at 10:35 PM, Paul Wise  wrote:
>>
>> On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:
>>
>>> upload them to PyPI since the authors of the coming Warehouse
>>> replacement for the current CheeseShop PyPI have already indicated
>>> that they intend to drop support for signatures entirely.
>>
>> Did they give any reasoning for this decision?
>>
> 
> 
> https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 
> 
> 
> —
> Donald Stufft

Your analysis makes sense: the hardest part of this problem is managing
_who_ is the trusted signer of a given package.  TOFU
(Trust-On-First-Use) is the easy case here: the first person to sign the
release is the one to trust.  Android's APK signature verification for
updates is entirely based on that principal.

It seems that so many Python and Java library developers (mavenCentral
is in a similar boat to PyPi) just want to ignore the signing key
management because its annoying.  Yes, it is annoying, I do a lot of it.
 Android forces all devs to manage a signing key: unsigned APKs are not
installable, and Google Play even rejects APKs signed by debug keys.

I think the Android/Google Play model would work quite well for PyPI,
but the social problem of getting python devs to take responsibility
would be the hard part.  I manages the transition of F-Droid app repos
to only signed, which is a small example.  There were some complaints,
but nothing so bad when we pointed them to the fdroid tool that
automatically generated the signing key for them.  pip can easily
generate and use a signing key (GPG, TUF, whatever) automatically.  Then
devs just need to learn out to safely manage them.  Android Studio's key
generation is pretty new, and has kind of shitty security
recommendations.  Android devs are still learning, but I think overall,
its working out well there.

As for TUF (The Update Framework), I know the concepts well.  I've
looked into it while implementing the F-Droid security model
(https://f-droid.org/wiki/page/Security_Model).  It does not deal with
the issue of managing identity of package uploaders at all, it assumes
you already have all the software in place, and just deals with
distributing updates.  Also, with TUF, you'll be in the exact same boat
here: signing keys are required per package stream, so each developer
will need to managing signing keys.

.hc

Re: GnuPG signatures on PyPI: why so few?

[Brian May]

Donald Stufft  writes:

> https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 
> 

  "I am aware of a single tool anywhere that actively supports verifying the
  signatures that people upload to PyPI, and that is Debian's uscan program. 
Even
  in that case the people writing the Debian watch file have to hardcode in a
  signing key into it and in my experience, when faced with a validation error
  it's not unusual for Debian to simply disable signature checking for that
  project and/or just blindly update the key to whatever the new key is."

I would never just blindly disable signature checking or update the key
without carefully checking that this is legitimate first (and/or
carefully checking the diff). For example, if releases were signed by
person A, but now signed by person B, there should be some sort of
public record of this fact. If not, ask on a public forum.

If you remove signatures (or don't supply them in the first place), then
we - as Debian packagers - have no way of validating the upload. So you
only need to compromise the package the maintainer downloads, and
subsequently everyone who uses the (signed) Debian packaging is
affected.

If however PyPI were to remove signatures, this would make the decision
whether to use PyPI or github as the source somewhat easier.
-- 
Brian May 

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 12, 2017, at 10:35 PM, Paul Wise  wrote:
> 
> On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:
> 
>> upload them to PyPI since the authors of the coming Warehouse
>> replacement for the current CheeseShop PyPI have already indicated
>> that they intend to drop support for signatures entirely.
> 
> Did they give any reasoning for this decision?
> 


https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html 


—
Donald Stufft

Re: GnuPG signatures on PyPI: why so few?

[Paul Wise]

On Mon, Mar 13, 2017 at 4:28 AM, Jeremy Stanley wrote:

> upload them to PyPI since the authors of the coming Warehouse
> replacement for the current CheeseShop PyPI have already indicated
> that they intend to drop support for signatures entirely.

Did they give any reasoning for this decision?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: GnuPG signatures on PyPI: why so few?

[Jeremy Stanley]

On 2017-03-12 11:46:31 +1100 (+1100), Ben Finney wrote:
[...]
> In response to polite requests for signed releases, some upstream
> maintainers are now pointing to that thread and closing bug reports as
> “won't fix”.
> 
> What prospect is there in the Python community to get signed upstream
> releases become the obvious norm?

Speaking for OpenStack's tarballs at least, our sdists are built by
release automation which also generates detached OpenPGP
signatures so as to provide proof of provenance... but we don't
upload them to PyPI since the authors of the coming Warehouse
replacement for the current CheeseShop PyPI have already indicated
that they intend to drop support for signatures entirely. We
consider https://releases.openstack.org/ the official source of
information for our release information and host our signatures
there instead (well, really on https://tarballs.openstack.org/ with
direct links from the former).

The same key used to sign our tarballs (and wheels) also signs our
Git tags, for added consistency.
https://releases.openstack.org/#cryptographic-signatures
Of possible further interest: we modeled a fair amount of our key
management after what's employed for Debian's archive keys.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature

Re: GnuPG signatures on PyPI: why so few?

[Ben Finney]

Brian May  writes:

> Maybe there is some way of turning signatures on by default, so I don't
> have to remember for every upload, if so, I haven't been able to work it
> out yet however.

I don't know how it should be done using the currently-recommended
Twine tool.


For Distutils, the ‘upload’ command has a ‘--sign’ option
 to specify
the distribution should be GnuPG signed, and the ‘--identity’ option
specifies which GnuPG identity from your keyring should sign the
distribution.

To set an option default for Setuptools commands, you put it in the
‘setup.cfg’ file.

So, in the code base's ‘setup.cfg’:

[upload]

# Sign distributions, and upload the signing public key?
sign = true

# Which GnuPG identity to use for signing?
identity = christ...@example.org

-- 
 \  “Probably the earliest flyswatters were nothing more than some |
  `\sort of striking surface attached to the end of a long stick.” |
_o__) —Jack Handey |
Ben Finney

Re: GnuPG signatures on PyPI: why so few?

[Donald Stufft]

> On Mar 11, 2017, at 9:23 PM, Brian May  wrote:
> 
> Ben Finney  writes:
> 
>> However, this only works if upstream releases are actually accompanied
>> by a valid GnuPG signature each time. The PyPI infrastructure supports
>> this; why isn't it more widely encouraged?
> 
> One reason I have found for myself: I can forget to sign the package
> when uploading to PyPI, and PyPI doesn't let you make any changes after
> the package is uploaded without changing the version (including adding
> signature file). There is a long running bug report on this, it is not
> going to get fixed (TLDR it is not a bug, it is a design feature to
> allow for caching).

I think there is no reason we *couldn’t* allow uploads of signatures after the 
fact, although it sort of makes an already tenuous security claim even more 
tenuous I think. Since distutils/setuptools don’t have reproducible “builds” of 
sdists if you wanted to after the fact upload a signature your choices are:

* Hope you still have the original files you uploaded still laying around, sign 
*them* and then upload just the signature.
* Download the files, open them up and inspect all of the files to ensure they 
match the correct state.
* Download the files, blindly sign them.

Of those options, the first one is no more or less “secure” than uploading 
alongside the original file, but the problem is that it’s impossible to 
differentiate that from the second two, and that could genuinely be worse (for 
example, compromised PyPI gives author a bad file, they sign it and upload the 
signature thus making everyone feel it is a “safe” file). Both the second 
options are kind of bad because it is incredibly easy to smuggle something bad 
into a tarball that a human being would miss.

The larger reason why I personally haven’t implemented it is because I believe 
the GPG signatures on PyPI are largely useless from a practical standpoint and 
I’m not really interested in doing anything to improve them and I would 
personally much rather just disable them completely.


> 
> Maybe there is some way of turning signatures on by default, so I don't
> have to remember for every upload, if so, I haven't been able to work it
> out yet however.

Um, I think you can set something in setup.cfg for ``setup.py upload``. I don’t 
think there is anything like that for Twine. I can’t speak for Ian but I don’t 
personally see anything inherently wrong with adding a environment variable or 
config option to twine that allows it to always sign by default. Ian may feel 
differently though!

—
Donald Stufft

Re: GnuPG signatures on PyPI: why so few?

[Brian May]

Ben Finney  writes:

> However, this only works if upstream releases are actually accompanied
> by a valid GnuPG signature each time. The PyPI infrastructure supports
> this; why isn't it more widely encouraged?

One reason I have found for myself: I can forget to sign the package
when uploading to PyPI, and PyPI doesn't let you make any changes after
the package is uploaded without changing the version (including adding
signature file). There is a long running bug report on this, it is not
going to get fixed (TLDR it is not a bug, it is a design feature to
allow for caching).

Maybe there is some way of turning signatures on by default, so I don't
have to remember for every upload, if so, I haven't been able to work it
out yet however.
-- 
Brian May 

GnuPG signatures on PyPI: why so few?

[Ben Finney]

Howdy all,

What prospects are there for PyPI to have GnuPG-signed packages by default?

Debian's UScan has the ability to find, download, and verify the GnuPG
signature for a package source release. Lintian will remind the
maintainer if a Debian source package is not taking advantage of this.

However, this only works if upstream releases are actually accompanied
by a valid GnuPG signature each time. The PyPI infrastructure supports
this; why isn't it more widely encouraged?

This thread from 2016 has a possible answer:

while you can use GPG as is to verify that yes, "Donald Stufft"
signed a particular package, you cannot use it to determine if
"Donald Stufft" is *allowed* to sign for that package, a valid
signature from me on the requests project should be just as invalid
as an invalid signature from anyone on the requests project. The
only namespacing provided by GPG itself is "trusted key" vs "not
trusted key".

[…] I am aware of a single tool anywhere that actively supports
verifying the signatures that people upload to PyPI, and that is
Debian's uscan program. […]

All in all, I think that there is not a whole lot of point to having
this feature in PyPI, it is predicated a bunch of invalid
assumptions (as detailed above) and I do not believe end users are
actually even using the keys that are being uploaded.

[…] Thus, I would like to remove this feature from PyPI […].



The thread has some discussion, and Barry Warsaw makes the case for
Debian's use for signed releases. The last (?) post in the thread has a
kind of interim conclusion:

My main concern when implementing this is how to communicate it to
users […]. [A move that gives the impression] "we're getting rid of
this thing that only kinda works now in favor of something amazing
that doesn't exist yet" is just not a popular move.



In response to polite requests for signed releases, some upstream
maintainers are now pointing to that thread and closing bug reports as
“won't fix”.

What prospect is there in the Python community to get signed upstream
releases become the obvious norm?

-- 
 \ “It is the fundamental duty of the citizen to resist and to |
  `\  restrain the violence of the state.” —Noam Chomsky, 1971 |
_o__)  |
Ben Finney