Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) __**_ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/**mailman/listinfo/catalog-sighttp://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
Yes, Nick's suggestions are good ones. I'd agree that getting an initial deployment together that doesn't include things like custom metadata is probably for the best. We can certainly add things incrementally. Thanks, Justin On Thu, Mar 14, 2013 at 3:21 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 3:03 AM, Nick Coghlan wrote: I think what you currently propose (signing the metadata pip already understands) is a good first step, especially if we can have PyPI signing *all* the target metadata in the initial deployment, and defer the delegation to package developers until the next phase of the rollout (we obviously want to do that eventually, but it's easier if we can get a preliminary version working without needing to change the upload tools). While such an approach doesn't immediately give us the end-to-end security we ultimately want to set up, it means a few things become possible: 1. Rather than requiring every developer to start signing end-to-end metadata immediately, we can ask a few major projects (e.g. Django, Zope, NumPy) if they're willing to serve as guinea pigs for the developer target signing delegations. Once we're happy the signing process is usable, we can make it generally available as an option to projects (while also allowing them to continue with PyPI's existing upload mechanisms and only offer PyPI-user integrity checks rather than developer-user) 2. Gives the PSF infrastructure team and the PyPI maintainers a chance to work with the installation tool developers to get the PyPI-user link sorted out, before needing to work on the developer-PyPI link 3. Considering alternate mirroring solutions based on replicating the TUF metadata rather than PEP 381 Eventually I would also like to tunnel a subset of the PEP 426 metadata through TUF's custom fields, but again, I think we're better off skipping that for the first iteration. Incremental enhancements are a good thing :) This sounds good to me --- I like the idea of incremental enhancements. Justin, what are your thoughts from a security perspective? ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
We may have something unclear in the doc. We definitely don't just worry about package names. (In between meetings, will send a longer response in a bit.) Thanks, Justin On Wed, Mar 13, 2013 at 2:15 PM, Daniel Holth dho...@gmail.com wrote: On Wed, Mar 13, 2013 at 5:13 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: Hello Nick, On 3/13/13 4:09 AM, Nick Coghlan wrote: - the PSF board generally stays out of the technical details of running the python.org infrastructure, so it's likely that any root keys would be handled by the PSF infrastructure committee. A (2, 4) or (3, 5) trust configuration would likely be manageable at this level. Understood. We think a higher (t, n) [where t out of n signatures are needed to trust the metadata for a role] is better for the root role simply because its crucial metadata (the authorized keys for top-level roles) should change very rarely. - at the target delegation level, PyPI supports the registration of new projects through the web service (see http://docs.python.org/2/distutils/packageindex.html). If my understanding of target delegation is correct, this means the simple and packages/source/letter delegations will need to be (1, 1) and online. - higher levels of the target delegation hierarchy could conceivably be kept offline, but there seems little value in doing so if they're trusting on online (1, 1) key Fortunately, the targets/simple and targets/packages/(version)/(letter)/ roles should not require (1, 1) online keys, as their metadata (simply target delegations and no actual target files) should also fluctuate fairly rarely. I should make this clearer in our design document. - many PyPI packages are maintained by single developers, so (1, 1) or (1, n) is likely to be the only generally feasible level of signing at the project level. Yes, the package developers themselves could choose any (t, n) they like. In our design, we propose that PyPI could eventually delegate to stable packages which need little change (and use more security with more offline keys) and to unstable packages which need frequent change (and use less security with more online keys). With the current focus being on getting an improvement from the status quo that we can successfully deploy in a reasonable period of time, the target delegation side of things probably needs to be substantially simpler in the initial iteration. Yes, it leaves us open to certain vulnerabilities we would like to remove in the long run, but we need to be very cautious in the additional demands we place on the users uploading to PyPI. It may even mean the initial iteration allows projects to rely on a PyPI provided signing key for their TUF metadata, using the existing upload mechanisms to add the files to PyPI. I agree that there is a delicate problem of balancing security with usability here, especially in the beginning. You raised a very good issue there: on first migration, how would PyPI accommodate packages which have not had their target files delegated to their developers? We imagine that in this case, PyPI could assume initial responsibility for these packages, and later PyPI would delegate those packages to their respective developers. Thanks for your input, Trishank With all the different kinds of metadata, It's interesting to note that currently TUF seems to only be concerned with the available file names and their integrity. (Some of us will think of PEP 426 PKG-INFO first when we hear the word metadata.) It looks like the D metadata lists all the filenames for Django, and then Django lists them again with hashes and signatures. Why all the lists? Does every Django release re-assert all the versions of Django that are available on the index? How might I deal with producing the official source distribution myself and having a friend produce the official Windows build of a package? As an aside PyPI has been doubling in size every 1.5 - 2 years. Thanks Daniel Holth ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
We use the simple directory and filenames because that is what pip uses. You have a nice suggestion to include other metadata in the TUF metadata. We certainly could do this if desirable. This required a redesign of the PyPI API and we weren't sure if this was wanted. Our current doc / prototype is trying to minimize the changes needed all around. Thanks, Justin On Wed, Mar 13, 2013 at 2:15 PM, Daniel Holth dho...@gmail.com wrote: On Wed, Mar 13, 2013 at 5:13 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: Hello Nick, On 3/13/13 4:09 AM, Nick Coghlan wrote: - the PSF board generally stays out of the technical details of running the python.org infrastructure, so it's likely that any root keys would be handled by the PSF infrastructure committee. A (2, 4) or (3, 5) trust configuration would likely be manageable at this level. Understood. We think a higher (t, n) [where t out of n signatures are needed to trust the metadata for a role] is better for the root role simply because its crucial metadata (the authorized keys for top-level roles) should change very rarely. - at the target delegation level, PyPI supports the registration of new projects through the web service (see http://docs.python.org/2/distutils/packageindex.html). If my understanding of target delegation is correct, this means the simple and packages/source/letter delegations will need to be (1, 1) and online. - higher levels of the target delegation hierarchy could conceivably be kept offline, but there seems little value in doing so if they're trusting on online (1, 1) key Fortunately, the targets/simple and targets/packages/(version)/(letter)/ roles should not require (1, 1) online keys, as their metadata (simply target delegations and no actual target files) should also fluctuate fairly rarely. I should make this clearer in our design document. - many PyPI packages are maintained by single developers, so (1, 1) or (1, n) is likely to be the only generally feasible level of signing at the project level. Yes, the package developers themselves could choose any (t, n) they like. In our design, we propose that PyPI could eventually delegate to stable packages which need little change (and use more security with more offline keys) and to unstable packages which need frequent change (and use less security with more online keys). With the current focus being on getting an improvement from the status quo that we can successfully deploy in a reasonable period of time, the target delegation side of things probably needs to be substantially simpler in the initial iteration. Yes, it leaves us open to certain vulnerabilities we would like to remove in the long run, but we need to be very cautious in the additional demands we place on the users uploading to PyPI. It may even mean the initial iteration allows projects to rely on a PyPI provided signing key for their TUF metadata, using the existing upload mechanisms to add the files to PyPI. I agree that there is a delicate problem of balancing security with usability here, especially in the beginning. You raised a very good issue there: on first migration, how would PyPI accommodate packages which have not had their target files delegated to their developers? We imagine that in this case, PyPI could assume initial responsibility for these packages, and later PyPI would delegate those packages to their respective developers. Thanks for your input, Trishank With all the different kinds of metadata, It's interesting to note that currently TUF seems to only be concerned with the available file names and their integrity. (Some of us will think of PEP 426 PKG-INFO first when we hear the word metadata.) It looks like the D metadata lists all the filenames for Django, and then Django lists them again with hashes and signatures. Why all the lists? Does every Django release re-assert all the versions of Django that are available on the index? How might I deal with producing the official source distribution myself and having a friend produce the official Windows build of a package? As an aside PyPI has been doubling in size every 1.5 - 2 years. Thanks Daniel Holth ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
Speaking of which, it may be the case that our design document for integrating PyPI with TUF may not be terribly easy to understand. (After all, you do need to understand TUF first, but TUF is fairly easy once you understand its main ideas.) I plan to publish a friendlier document which introduce TUF at a very high-level and instead discuss more pragmatic issues (such as workflows). Feel free to chime in if you'd rather see something else or want us to focus on clarifying a specific topic. Thanks, Justin ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI/pip security: waiting for input
Yes, we're finishing this up now. We have a working demo with TUF signing PyPI metadata and pip (integrated with TUF) correctly checking signatures, etc. Trishank: when do you plan to share this? Does Kon still have some integration tests to write to show we meet the use cases from Giovanni's document? Thanks, Justin On Mon, Mar 11, 2013 at 9:34 AM, Giovanni Bajo ra...@develer.com wrote: Hi Justin, just a quick reminder that we are still waiting for you guys to move over and start actually doing something. Are you going to draft a document on how exactly we can use TUF within the context of pip + PyPI, with all the different concerns and thread models handled in my document? Thanks! -- Giovanni Bajo :: ra...@develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Deprecate External Links
Having different sources for package metadata does pose security concerns, for example version mismatch attacks by a MITM. Unless we co-locate all package metadata at a single source that is trusted for protecting against these issues, this will be an issue.(However, possibly not the biggest threat right now.) I do believe that if you do centralize metadata, you could outsource mirroring the data if desired without losing the other security goals you have. Thanks, Justin On Wed, Feb 27, 2013 at 10:39 AM, M.-A. Lemburg m...@egenix.com wrote: On 27.02.2013 16:26, Donald Stufft wrote: PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However there is still another massive area of insecurity within the packaging tool chain. For those who don't know, when you attempt to install a particular package a number of urls are visited. The steps look roughly something like this: 1. Visit http://pypi.python.org/simple/Package/ and attempt to collect any links that look like it's installable (tarballs, #egg=, etc). Note: /simple/Package/ contains download_url, home_page, and any link that is contained in the long_description). 2. Visit any link referenced as home_page and attempt to collect any links that look like it's installable. 3. Visit any link referenced in a dependency_links and attempt to collect any links that look like it's installable. 4. Take all of the collected links and determine which one best matches the requirement spec given and download it. 5. Rinse and repeat for every dependency in the requirement set. I propose we deprecate the external links that PyPI has published on the /simple/ indexes which exist because of the history of PyPI. Ideally in some number of months (1? 2?) we would turn off adding these links from new releases, leaving the existing ones intact and then a few months later the existing links be removed completely. -1. There are many reasons for not hosting packages and distributions on PyPI itself. If you use and trust a package, you also have to know and trust its dependencies, no matter where they are hosted, so you're not gaining any security by disabling links to other download locations: if you don't trust the way a package is hosted, you don't use it; if you do, then removing the link breaks the package and all its dependencies. Instead of suggesting to removing support for externally hosted packages, why not propose a mechanism to provide a more direct/secure way to reference them ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 26 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
Okay, I took a quick look and posted a bunch of comments in the document. I took a more thorough look at the early sections than the later. You've done a nice job with the design overall and clearly thought through a lot of security issues. I did point several places where I either don't understand something or there might be a potential to improve the security. After reading the doc, I'm not clear on how mirrors / CDNs / separate file servers will be used in the system and what sort of trust you are placing in them. In particular, much of the text about PyPI may or may not apply to mirrors. These are a major headache from a security standpoint and something we've really tried to minimize in TUF. I've also thought more about how TUF would address the issues you've mentioned. I believe TUF addressed the concerns mentioned in the doc (except of course things like password storage which are PyPI website changes). We also all of the proposed future enhancements mentioned at the end of the document. I must confess I'm still digging out after my deadline, so my responses may be delayed. Thanks, Justin On Sat, Feb 9, 2013 at 4:23 PM, Giovanni Bajo ra...@develer.com wrote: Hello, my proposal for fixing PyPI and pip security is here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. Comments are welcome! -- Giovanni Bajo :: ra...@develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
Thanks. I've replied to your comments. Sure, thanks a lot for clearing up some of my confusion. After reading the doc, I'm not clear on how mirrors / CDNs / separate file servers will be used in the system and what sort of trust you are placing in them. In particular, much of the text about PyPI may or may not apply to mirrors. These are a major headache from a security standpoint and something we've really tried to minimize in TUF. I think the current PyPI mirror can be highly simplified once we introduce end-to-end authenticity with GPG. My suggestion would be to make them simple file servers, or even drop them and switch to commercial CDN, that would simplify lots of management. What we should drop is the concept of a full mirror, as it creates lots of security headaches as you say. I think the PSF board is open to a proposal to set up a budget for this. I definitely agree that simple file servers are by far the easiest thing to secure. I think you can have mirrors, even ones run by untrusted parties, and still have strong security. It's really the metadata that you need to be most worried about the correctness of (except for a few oddball attacks). I'll talk more about this when providing more details. I've also thought more about how TUF would address the issues you've mentioned. I believe TUF addressed the concerns mentioned in the doc (except of course things like password storage which are PyPI website changes). We also all of the proposed future enhancements mentioned at the end of the document. I think TUF is a large superset of what I proposed, that means that it is also a large superset of what it is (likely) needed. I'm still worried of how we can simplify TUF from an UX and IT perspective. I think that I need some inputs from you. Can you please write down something that describes: Sure, we will get a document together and follow up with you. However, in the meantime, I'll give you some basic answers. 1) What is exactly expected from a package maintainer to do to: 1a) register themselves as package maintainers on PyPI So the normal process would still apply. We may encourage them to generate and upload a key, but otherwise it is the same. The PyPI maintainer needs to delegate trust to their target package. This could be an online action (for ease of operation) or an offline one (for security). 1b) sign/publish a new package Run a single command to do the signature in the normal case. If they choose to use a threshold of keys, it may require multiple devels to do so. They need to upload the signature and the new package. 1c) hide/show a package version I need to look into this more. There are several ways this can be set up and I need to understand more to know how to respond. Offhand, I would say that having the developer sign and upload metadata indicating hidden vs. visible is the most secure. From a usability perspective, PyPI could sign something stating this instead, but this requires trusting PyPI more than may be wise. Were it my system, I'd prefer the former (and can talk more about risks with the latter), but either choice seems reasonable. 2) What modifications are required on the PyPI server? As for PyPI mods, we'll look into this and get back to you. TUF was designed to slot into a repo that is on a static webserver (or similar). I don't know if PyPI will cause any problems. How many GPG keys the server would need to handle? Would they be online or offline? What processes do we need to setup? Parts of this are also up for debate. TUF isn't meant to be a rigid set of rules w/ keys one must follow. It's meant to be flexible enough that it can be used for a variety of environments. For example, you likely want to have a threshold of keys for the root signing keys. These would be rarely used (only for key revocation) and should be kept offline. So you may collectively decide to have one for every board member of the PSF (for example). Alternatively, you may decide to give them to the 5 people who are most involved with PyPI. I don't know enough to know what makes sense politically or for your workload. What I will do is come up with something based upon my understanding of what might work. Feel free to push back if something seems to onerous and let us know if you don't understand why we said you should have a role for X, etc. I also might need some PyPI stats to make sane recommendations. I'll let you know... I would expect such document to describe also required changes to distutils and PyPI protocols, if required. Sure. This is an important concern and one we've been looking into. I must confess I'm still digging out after my deadline, so my responses may be delayed. There is no specific hurry, though I would like these issues to be sorted out. I'm happy to integrate TUF if it's the best solution, but we need to discuss how. Okay, this sounds good.
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
Yes, that is what I meant. Sorry for any confusion about this. Thanks, Justin On Tue, Feb 12, 2013 at 3:40 AM, Giovanni Bajo ra...@develer.com wrote: Il giorno 11/feb/2013, alle ore 20:33, Justin Cappos jcap...@poly.edu ha scritto: Once again, apologies for being mostly out of this discussion for the next 10 days or so, but I did want to jump in and clarify a point. TUF can be used exactly with a one-key-per-devel model. (If fact, see our CCS 10 paper on this for details.) It's possible to revoke keys and have split keys, etc. but a simple developer setup is just as simple as what you propose. Sorry I can't find this in the CCS10 document, but maybe it's just that I don't understand what you mean. The document talks about 1 key per role (§8.2), but there are still 4 roles that need to be implemented, as far as I can tell. Are you suggesting that a single developer only handles the target role, while the others are centrally handled by PyPI? -- Giovanni Bajo :: ra...@develer.com Develer S.r.l. :: http://www.develer.com My Blog: http://giovanni.bajo.it ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
Once again, apologies for being mostly out of this discussion for the next 10 days or so, but I did want to jump in and clarify a point. TUF can be used exactly with a one-key-per-devel model. (If fact, see our CCS 10 paper on this for details.) It's possible to revoke keys and have split keys, etc. but a simple developer setup is just as simple as what you propose. Thanks, Justin On Mon, Feb 11, 2013 at 2:26 PM, Giovanni Bajo ra...@develer.com wrote: Il giorno 10/feb/2013, alle ore 23:20, Jesse Noller jnol...@gmail.com ha scritto: On Feb 10, 2013, at 7:54 AM, Nick Coghlan ncogh...@gmail.com wrote: On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel jan...@leidel.info wrote: On 10.02.2013, at 05:44, Nick Coghlan ncogh...@gmail.com wrote: On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo ra...@develer.com wrote: Hello, my proposal for fixing PyPI and pip security is here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. I think the parts related to improving the HTTPS/SSL based security are solid, but for the other aspects of secure updates, integrating TUF (https://www.updateframework.com/) into the PyPI based distribution infrastructure sounds like the best available option for enhancing the end-to-end integrity checking. TUF has a comparatively well-developed threat model, and systematically covers many of the attack vectors discussed in the past few day (including provision of old, known vulnerable, versions). Would you mind explaining why TUF is good? The main benefit in my mind is that it isn't a from-scratch design of a secure update infrastructure. Instead, it's a project that was started in order to resolve some security holes found in Tor's already robust automatic update mechanism, then proceeded from there into updates to yum, yast, apt, etc (i.e. the distro update mechanisms that are vetted by the security teams of the various Linux vendors). The fact Geremy Condra is involved in TUF also counts for a lot with me (as I suspect it would for many people that have heard Geremy talk about security issues in Python). However, the design itself also seems sensible, and is able to provide its security guarantees even if you're *not* using SSL certs to protect the in-flight traffic (thus meaning that the SSL infrastructure in the near term will become a matter of providing defence-in-depth, rather than being a required part of the security scheme). I trust our collective ability to make TUF sufficiently easy to use as part of Python's packaging infrastructure a *lot* more than I trust our collective ability to come up with a from-scratch distribution scheme that is both usable *and* secure. The site doesn't seem to work for me right now. D'oh, looks like their domain wasn't set to auto-renew :( Cheers, Nick. Feedback from Geremy below: OK, so, I think there's a lot of stuff conflated here. It'll probably help to simplify things if we decouple them. First, the point about serving metadata over a secure channel and data over a cheap one is right on. Given the size of your metadata versus actual data, maintaining a central metadata service but not caring about where/how data is hosted is the right way to go. Note that that channel doesn't have to be SSL- a verifying cert on device would still give you everything you needed. Second, decouple the transport-level problem from verifying code. SSL is good, but it doesn't provide end to end security, which is what you care about here. A good alternative is the Android model, with per developer keys- it keeps attribution with code and clients can verify that the key is correct based on the current and possibly previous signed metadata bundle. The Android model is the self-signed model (aka SSH model), where an user is presented with a self-signed certificate and just needs to press Yes without verifying. In fact, Android doesn't even ask the question to the user, and it assumes that everything you download from the app store is correct. So in a way, the main protection it offers is that it's not possible for another user to publish an application with the same name on the Android market, but AFAIK if I download an application, insert a malware, sign it myself, and directly send to a victim, the victim will have no way to realize the application has a wrong signature (unless it's been installed before). The Android model works well for their use cases, but we can do more than that. In our case, users install packages with their unique package name, so we are
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
So we obviously have egg on our face about the domain expiry. Our sys
admin should be fixing this now.
I'm a bit crunched for time with other deadlines right now, but one quick
thing I'd like to correct is that TUF works with package managers /
metadata, etc. TUF certainly handles package managers and appropriately
protects their metadata, which in turn makes sure that dependency
resolution is secure, etc.
TUF does also prevent against replay attacks on old package versions (or at
least those that were not valid within the lifetime of the timestamp.txt
file and aren't earlier than the last time the user fetched a package.)
Note that TUF does not preclude you from asking for foo-1.0 when foo-1.1
was released if there is a valid signed version of foo-1.0. It just makes
sure that if you ask for foo you get foo-1.1.
I'll be happy to give a detailed look at the other proposal before PyCon
and send some notes. (Unfortunately I have a conflicting venue I need to
attend so will miss PyCon.)
Thanks,
Justin
On Sun, Feb 10, 2013 at 9:18 AM, Giovanni Bajo ra...@develer.com wrote:
Il giorno 10/feb/2013, alle ore 14:43, Nick Coghlan ncogh...@gmail.com
ha scritto:
On Sun, Feb 10, 2013 at 11:30 PM, Giovanni Bajo ra...@develer.com
wrote:
This is by far the biggest problem to be solved, and my document brings
a proposal here. It would be great if the TUF guys reviewed it.
Ensuring we fully address the problems that are addressed by TUF is
more important than the question of whether or not we use the TUF
software itself. However, the concern I have with your proposal is
that I saw zero information regarding how it deals with attackers
supplying old versions of software,
It's true that it's something that my document doesn't cover, but there is
a reason.
The way TUF handles it, is through an automatic signing of a file called
timestamp.txt, using a fast expiring signature. This files basically
contains (simplifying) a reference to the latest version of the software.
Since the signature expires, an attacker cannot replay it.
On PyPI, a maintainer is able to decide which is the current/available set
of releases of a packages through the web interface of PyPI. This means
that the maintainer can click a few buttons on the web UI, and the
timestamp.txt should change to its desire, including reviving an old
version of the software. So in a way, we're offering a feature that already
*breaks* the feature that TUF is trying to protect against. An attacker
with the user's PyPI password can revive any old version of the software.
In the TUF complete model, this is less of an issue because all commands
release this, unrelease this are made through GPG-validated signatures.
So to fully implement this, we would also need to modify setup.py to offer
any package release management feature of PyPI.
TL;DR: an integration of TUF doens't magically fix the replay attack of
older versions, unless we also specifically modify distutils to include
package management administrative commands to it, with GPG
validation/signing, and we expose the corresponding webserver API from
PyPI. We still need a design here.
NOTE: within my proposal, an attacker with write access to the PyPI file
storage area can't do a replay attack; in fact, even if it overwrites the
contents of django-1.1.2.tar.gz{,.sig} with django-1.1.1.tar.gz{,.sig}, and
even if the signature would be verified, the package metadata will show the
1.1.1 version, and thus pip would be able to detect the attack. I've
clarified this in my document.
or, indeed, any description of the
threat model at all.
Can you please elaborate on what you expect to find as thread model?
Every task in my document that is supposed to give an enhanced security
explicitly lists the possible attacks that are prevented by implementing
the task. This is my definition of threat model, but maybe you are
thinking of something different.
Or maybe you would prefer to see all those sections together at the
beginning of the document?
The parts of your proposal that I believe need to
be closely reviewed are:
- GPG vs PKCS#1
Do you have any specific open-source multi-platform implementation of
PCKS#1 in mind? I think we don't want to write our own crypto here,
especially since PCKS#1 is *really* tricky.
- your custom trust model vs TUF target delegation
- any threats that TUF covers and your proposal does not
To do this, we first need a proposal from TUF authors on how they plan to
integrate it with pip/PyPI. On their website, there is (was) a preliminary
document on this, but it just scratched the surface.
Let me stress agan that TUF is just a component of the solution, because
it's not a design for a package manager, just for a single-software update.
There are important differences in how trust is handled (eg: for a specific
software, you can embed the trust in the first version, and upgrade from
there), while in a
Re: [Catalog-sig] PyPI and setuptools
We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... Thanks, Justin On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller jnol...@gmail.com wrote: On Feb 9, 2013, at 6:13 PM, Stephen Thorne step...@thorne.id.au wrote: Hello, One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). This leaves us at the point where they can not be sunset unless the other tools grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. Just patching them for security fixes seems like an easy task; the bigger question is how to do that only without further feature addition and getting a release out the door? Jesse ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI and setuptools
No, it's not. We're working on something separate that is based on some of the security architecture changes we got pushed into apt, YUM, YaST, Pacman, etc. It's designed with help from the Tor folks after we disclosed some security issues in their software updater. I'd need to look at Giovanni's proposal in detail (and I don't have time until late this month at the earliest), but there are a lot of really subtle bugs and issues one can get into. That's why we're trying to push for a universal framework instead of having every project roll their own... Thanks, Justin On Sat, Feb 9, 2013 at 6:43 PM, Jesse Noller jnol...@gmail.com wrote: Is what you just said part of Giovanni's proposal he sent for review? On Feb 9, 2013, at 6:40 PM, Justin Cappos jcap...@poly.edu wrote: We're hoping to be able to fix this by interposing on network communications by these tools. The basic idea is that we'll have a replacement for urllib, urllib2, etc. that adds and validates security cleanly. (Note the replacements will only be used in python package managers.) TUF ( https://www.updateframework.com/ ) will correctly validate security metadata and only pass validly signed information to the package manager for installation. So the hope is that other than a few lines of code that import the alternative for urllib, urllib2, etc. there won't be any changes. We will be maintaining the security code as a separate project (TUF is used by things other than Python package managers) and will be constantly improving it. Anyways, I won't be able to attend, but I will try to get a student to show a demo in the hallways at PyCon to show what we mean... Thanks, Justin On Sat, Feb 9, 2013 at 6:28 PM, Jesse Noller jnol...@gmail.com wrote: On Feb 9, 2013, at 6:13 PM, Stephen Thorne step...@thorne.id.au wrote: Hello, One of my concerns with the recent pip dramas that have seen some excellent and timely action from catalog-sig and others, is that 'setuptools' is still widely distributed and used instead of distribute/pip. Well, lets back up: these aren't pip specific problems: just about every client side tool for installing from pypi suffers from lax security. Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). This leaves us at the point where they can not be sunset unless the other tools grow the features of setuptools/easy_install or we (the collective we) take on the burden of updating that tool chain to support secure installations. Just patching them for security fixes seems like an easy task; the bigger question is how to do that only without further feature addition and getting a release out the door? Jesse ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Add link to secure connection to the PyPI front page
It depends on the threat model which is worse. If you're worried about the Chinese govt inserting malicious packages to track dissidents then using an universally accepted SSL cert is a bad idea. It's easy for a powerful and motivated attacker to get arbitrary certs signed. If you think that the risk of having the certificate stolen, loss of administrative control, etc. is a bigger threat, then an universally accepted SSL cert seems the wiser outcome. Of course, if distutils and other tools don't check certs, etc. this is all academic... Thanks, Justin On Sat, Jun 4, 2011 at 1:30 PM, M.-A. Lemburg m...@egenix.com wrote: Martin v. Löwis wrote: Which makes me wonder, why is it that PyPI doesn't use a universally accepted SSL cert instead of the CAcert one? Note: I'm a CAcert assurer myself but would prefer using a cert by one of the commercial CAs for the sake of the users. Any opinions? Primarily because of lack of volunteer time. Buying a certificate is a big effort, issuing a cacert one is simple. And before anybody says no, it's not difficult, or no, it shouldn't be difficult, please consider volunteering for the next ten years to manage the PSF server certificates (as one of the key problems that makes it difficult is that responsibilities change so often with volunteers). Perhaps we could get Pat, the PSF secretary and administrator to deal with the paperwork that's needed to get a certificate. Installing it is not really such a major task, once you have the paperwork done. Should we take this to the PSF board for discussion ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 04 2011) Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 16 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
On Sat, Jun 19, 2010 at 8:58 AM, Martin v. Löwis mar...@v.loewis.de wrote: A simple way to protect against just the issue you mentioned is to have the clients retrieve the key over HTTPS or distribute the key with the client. Ok. I have now enabled https for PyPI (https://pypi.python.org/pypi) Great. Assuming cert checking is implemented properly for the clients who retrieve your server's key, this will protect against many simple attacks. I don't think adding another dependency to the clients is really acceptable. Instead, it must all be self-contained. Okay, sounds good. We'll look elsewhere! Thanks, Justin ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
On Tue, Jun 15, 2010 at 11:09 PM, Martin v. Löwis mar...@v.loewis.de wrote: I'm not clear on this and the document is a little vague, so perhaps I should be perusing the source, but if you don't protect against a serverkey MITM and you are supposed to update the serverkey any time a signature doesn't match up, couldn't an attacker just MITM you, produce a known bad signature, and then wait for you to request a serverkey from them? That's true; transmission of the serverkey is not currently protected against MITM. How would you suggest to fix that? A simple way to protect against just the issue you mentioned is to have the clients retrieve the key over HTTPS or distribute the key with the client. In general, the problems are much, much trickier than just this. I won't bore you with all of the details (unless you'd like to know more), but we found and fixed a lot of problems with the security of linux package managers. A quick pointer to some of the technical details can be found here: http://www.cs.arizona.edu/stork/packagemanagersecurity/papers.html As for perusing the source: the client behavior is not implemented yet, so there isn't really any source to check, yet. Okay. We'd be happy to work with you to get an easy solution put in place. As I was shamelessly plugging before, we've been working on a library called TUF that is supposed to make this as simple as possible for whomever maintains the repository and be completely transparent for the clients. TUF is fairly early stage (our first major deployment is on going), but might be worth consideration. I think we could probably put together a quick demo so that you and others could see how it might work with one of the existing client updaters. Thanks, Justin ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability
On Tue, Jun 15, 2010 at 2:55 PM, Jesus Cea j...@jcea.es wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 15/06/10 20:52, Tarek Ziadé wrote:
Do you trust the package you are installing more than an official
mirror ? if so, why ?
If a package is signed by the author, I only need to trust the author.
I think it might not be this simple. You're still trusting PYPI to
provide you with the latest version of a package. Absent other
mechanisms, you don't have a way to tell if the file you're being
served is actually a version that is obsolete (possibly due to
security flaws).
Also, in practice many package managers perform dependency resolution
based upon on metadata that isn't signed with the author's GPG key.
http://www.cs.arizona.edu/stork/packagemanagersecurity/otherattacks.html#extradep
Is the plan to use what is proposed in
http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html in
practice? Is more information available about this? Does this
protect against man-in-the-middle attacks?
If a package is not signed in PYPI, I must trust the author, PYPI
admins and pypi machines security.
If I download from a mirror, with no digital signature, I must trust the
author, PYPI admins, pypi machines security, mirror admins, mirror
machine security and mirror replication protocol. And all network
connections and harddisks in between.
It is just me, call me paranoid, but I pay close attention to where the
package being installed by easy_install is pulled from. I have
documented where each package used to live and I check carefully when I
see an unexpected URL. And I freak out when I package upgrade includes
new dependencies I haven't seen before.
Anyone can upload a package at PyPI with
os.system('rm -rf /')
in its setup.py...
True. And SCARY. Fortunatelly I only install packages I am interested
in, check signatures, etc. Of course, I can be hacked if the original
autor put a trojan in the package, or he/she was hacked before. But my
exposure is smaller that if I must trust too every link in a LONG chain
of mirrors.
Just check his link, for a recent example:
http://it.slashdot.org/firehose.pl?op=viewtype=storysid=10/06/13/0046256
The trojan was not in the original sourcecode, but in an altered mirror
version.
Asking for pypi central node to add signatures is a trivial way of
avoiding this issue. The question is not to trust or not to trust
mirrors, but that we have technology to be safe even if the mirrors are
not trusted. I don't NEED to trust you to be safe. I am happy!.
I think there are other subtle issues here dealing with key
revocation, mismatching of package versions, etc.
A lot of these issues are pretty subtle and I'd be happy to talk in
more detail about how one might address them. In fact, we have a
project that is trying to do so:
https://www.updateframework.com/
Geremy do you want to chime in?
Thanks,
Justin
- --
Jesus Cea Avion _/_/ _/_/_/ _/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/
jabber / xmpp:j...@jabber.org _/_/ _/_/ _/_/_/_/_/
. _/_/ _/_/ _/_/ _/_/ _/_/
Things are not so easy _/_/ _/_/ _/_/ _/_/ _/_/ _/_/
My name is Dump, Core Dump _/_/_/ _/_/_/ _/_/ _/_/
El amor es poner tu felicidad en la felicidad de otro - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQCVAwUBTBf21Jlgi5GaxT1NAQLPngP+NfLf7js3ni9FvoDjkrzOB0AmRIyfmDJm
tm0wNEVIlTY+d3st76Gd62ET+VxtgNHfWyNQ82Zp0iAISoWlpDyflJlZ1r5oVjAR
sWOSntdXXZAaaxOkumggi1cHKVCbWAe+62fGctTLWt4QtP4557yJDHZO1LKp1nWe
qtHX5LyUD5k=
=yGPk
-END PGP SIGNATURE-
___
Catalog-SIG mailing list
Catalog-SIG@python.org
http://mail.python.org/mailman/listinfo/catalog-sig
___
Catalog-SIG mailing list
Catalog-SIG@python.org
http://mail.python.org/mailman/listinfo/catalog-sig
