Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
In addition, maintainers of installation tools are asked to release two updates. The first one shall provide clear warnings [...] The second update for installation tools should change the default mode to allow only installation of package files hosted at the index domain, sounds good to me. It is expected that tools in this release may choose to change the default index url to ``https://pypi.python.org/simple/-with-ext``https://pypi.python.org/simple/-with-extin so, *eventually*, the /simple interface (that has been transitioned to only serve pypi links) could be deprecated? (because new tools would be smart enough to responsibly navigate /simple/-with-ext) but slightly ironic that we'd be left with an interface called simple/-with-ext, given the goal of all this, but it makes sense. Marcus ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
Hi Marcus, On 03/15/2013 01:32 AM, Marcus Smith wrote: In addition, maintainers of installation tools are asked to release two updates. The first one shall provide clear warnings [...] The second update for installation tools should change the default mode to allow only installation of package files hosted at the index domain, sounds good to me. Excellent, having the installer-tool maintainers on-board is obviously important here :-) It is expected that tools in this release may choose to change the default index url to ``https://pypi.python.org/simple/-with-ext`` https://pypi.python.org/simple/-with-ext in so, *eventually*, the /simple interface (that has been transitioned to only serve pypi links) could be deprecated? (because new tools would be smart enough to responsibly navigate /simple/-with-ext) but slightly ironic that we'd be left with an interface called simple/-with-ext, given the goal of all this, but it makes sense. Right, it was precisely this awkwardness (the likelihood that tools would want to default to -with-ext and use host-comparison to distinguish internal/external, so as to provide info about external links with a single request-response) that led us to eliminate the separate indexes in our latest V4 draft and use rel attributes to distinguish link types. Carl signature.asc Description: OpenPGP digital signature ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 5:16 PM, Carl Meyer c...@oddbird.net wrote: There is no instead of. There are parallel proposals (see the TUF thread) to improve the security of the ecosystem, and those proposals are not mutually exclusive with this one. If you search the PEP text, note that you don't find the words secure or security anywhere within it, or any claims of security achieved by this proposal alone. There is a brief mention of MITM attacks, which is relevant to the PEP because avoiding external link-crawling does reduce that attack surface, even if other proposals will also help with that (even more). Right, the changes to provide end-to-end security require more extensive changes and need to be given appropriate consideration before we proceed to implementation and deployment. This PEP, especially with the additional changes you propose here is an excellent approach to *near term* improvement, as a parallel effort to the more complex proposals. The /simple/ index will also be around for a long time for backwards compatibility reasons, regardless of any other changes that happen in the overall distribution ecosystem. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 23:43 -0700, Nick Coghlan wrote: On Wed, Mar 13, 2013 at 5:16 PM, Carl Meyer c...@oddbird.net wrote: There is no instead of. There are parallel proposals (see the TUF thread) to improve the security of the ecosystem, and those proposals are not mutually exclusive with this one. If you search the PEP text, note that you don't find the words secure or security anywhere within it, or any claims of security achieved by this proposal alone. There is a brief mention of MITM attacks, which is relevant to the PEP because avoiding external link-crawling does reduce that attack surface, even if other proposals will also help with that (even more). Right, the changes to provide end-to-end security require more extensive changes and need to be given appropriate consideration before we proceed to implementation and deployment. This PEP, especially with the additional changes you propose here is an excellent approach to *near term* improvement, as a parallel effort to the more complex proposals. The /simple/ index will also be around for a long time for backwards compatibility reasons, regardless of any other changes that happen in the overall distribution ecosystem. I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. best, holger Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) __**_ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/**mailman/listinfo/catalog-sighttp://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Thu, Mar 14, 2013 at 7:13 AM, Justin Cappos jcap...@poly.edu wrote: Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. I think what you're proposing now is a pretty good place to state (although I'm suggesting making it even simpler in the near term by starting by focusing on the PyPI-end user link, and then moving to delegating signing of the per-project metadata to the individual projects as a later step) If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. I'd like to do a distribution 2.0 at some point where we make the simple index redundant by including that info (and more) directly in the TUF metadata, but I think that's a later project - securing what we have now is a better place to start. Cheers, Nick. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 7:21 AM, holger krekel hol...@merlinux.eu wrote: Hi all, after some more discussions and hours spend by Carl Meyer (who is now co-authoring the PEP) and me, here is a new V3 pre-submit draft. It is now more ambitious than the previous draft as should be obvious from the modified abstract (and Carl Meyers and Philip's earlier interactions on this list). There also are more details of how the current link-scraping works among other improvements and incorporations of feedback from discussions here. We intend to submit this draft tonight to the PEP editors. Feedback now and later remains welcome. I am sure there are issues to be sorted and clarified, among them the versioning-API suggestion by Marc-Andre. Thanks for everybody's support and feedback so far, holger Looks good to me! Setuptools' two releases will probably look like this: 1. Default to externals index, warn when fetching URLs that are not the same host as the index 2. Default to externals index, reject URLs that are not the same host as the index unless --allow-hosts is configured (IOW, default allow-hosts to equal index-url host) That way, external URLs can still be discovered by the user, but the default configuration is still secure. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Mar 13, 2013, at 10:26 AM, PJ Eby p...@telecommunity.com wrote: On Wed, Mar 13, 2013 at 7:21 AM, holger krekel hol...@merlinux.eu wrote: Hi all, after some more discussions and hours spend by Carl Meyer (who is now co-authoring the PEP) and me, here is a new V3 pre-submit draft. It is now more ambitious than the previous draft as should be obvious from the modified abstract (and Carl Meyers and Philip's earlier interactions on this list). There also are more details of how the current link-scraping works among other improvements and incorporations of feedback from discussions here. We intend to submit this draft tonight to the PEP editors. Feedback now and later remains welcome. I am sure there are issues to be sorted and clarified, among them the versioning-API suggestion by Marc-Andre. Thanks for everybody's support and feedback so far, holger Looks good to me! Setuptools' two releases will probably look like this: 1. Default to externals index, warn when fetching URLs that are not the same host as the index 2. Default to externals index, reject URLs that are not the same host as the index unless --allow-hosts is configured (IOW, default allow-hosts to equal index-url host) That way, external URLs can still be discovered by the user, but the default configuration is still secure. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig For the record I support the PEP and these 2 steps sound ok to me. My only suggestion is an additional rel attribute for indexes to indicate this is index hosted file incase the index domain and the package host domain differ (as is the case with Crate). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On 13.03.2013 12:21, holger krekel wrote: Hi all, after some more discussions and hours spend by Carl Meyer (who is now co-authoring the PEP) and me, here is a new V3 pre-submit draft. It is now more ambitious than the previous draft as should be obvious from the modified abstract (and Carl Meyers and Philip's earlier interactions on this list). There also are more details of how the current link-scraping works among other improvements and incorporations of feedback from discussions here. We intend to submit this draft tonight to the PEP editors. Feedback now and later remains welcome. I am sure there are issues to be sorted and clarified, among them the versioning-API suggestion by Marc-Andre. Thanks for everybody's support and feedback so far, holger PEP: XXX Title: Transitioning to release-file hosting on PyPI Version: $Revision$ Last-Modified: $Date$ Author: Holger Krekel hol...@merlinux.eu, Carl Meyer c...@oddbird.net Discussions-To: catalog-sig@python.org Status: Draft (PRE-submit V3) Type: Process Content-Type: text/x-rst Created: 10-Mar-2013 Post-History: Abstract This PEP proposes a backward-compatible two-phase transition process to speed up, simplify and robustify installing from the pypi.python.org (PyPI) package index. To ease the transition and minimize client-side friction, **no changes to distutils or existing installation tools are required in order to benefit from the transition phases, which is to result in faster, more reliable installs for most existing packages**. The first transition phase implements easy and explicit means for a package maintainter to control which release file links are served to present-day installation tools. The first phase also includes the implementation of analysis tools for present-day packages, to support communication with package maintainers and the automated setting of default modes for controling release file links. The second transition phase will result in the current PYPI index to only serve PYPI-hosted files by default. Externally hosted files will still be automatically discoverable through a second index. Present-day installation tools will be able to continue working by specifying this second index. New versions of installation tools shall default to only install packages from PYPI unless the user explicitely wishes to include non-PYPI sites. I must say, don't like this change in motivation compared to V1 and V2. The original of the discussion was to make PyPI more secure and the installation process faster and more reliable by moving away from crawling arbitrary external web pages. Both can be had by: * limiting the crawling to package author defined specific URLs, with added hashes to make sure that the URLs and their target content is not modified (this is the securing external downloads part - see here for an example: https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5), and * adding a way for the package authors to say PyPI, please go ahead and cache/copy my distributions files (this is the increase download reliability part - can be had by doing opt-in CDN caching/proxying of external links via PyPI) Now, with V3 of the proposal, you are moving towards a system that basically says do it this way, or stay out of our eco system, which, in my book, is not what the Python eco system is all about. Your V2 was much more inviting in this respect. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 13 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Mar 13, 2013, at 2:57 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 12:21, holger krekel wrote: Hi all, after some more discussions and hours spend by Carl Meyer (who is now co-authoring the PEP) and me, here is a new V3 pre-submit draft. It is now more ambitious than the previous draft as should be obvious from the modified abstract (and Carl Meyers and Philip's earlier interactions on this list). There also are more details of how the current link-scraping works among other improvements and incorporations of feedback from discussions here. We intend to submit this draft tonight to the PEP editors. Feedback now and later remains welcome. I am sure there are issues to be sorted and clarified, among them the versioning-API suggestion by Marc-Andre. Thanks for everybody's support and feedback so far, holger PEP: XXX Title: Transitioning to release-file hosting on PyPI Version: $Revision$ Last-Modified: $Date$ Author: Holger Krekel hol...@merlinux.eu, Carl Meyer c...@oddbird.net Discussions-To: catalog-sig@python.org Status: Draft (PRE-submit V3) Type: Process Content-Type: text/x-rst Created: 10-Mar-2013 Post-History: Abstract This PEP proposes a backward-compatible two-phase transition process to speed up, simplify and robustify installing from the pypi.python.org (PyPI) package index. To ease the transition and minimize client-side friction, **no changes to distutils or existing installation tools are required in order to benefit from the transition phases, which is to result in faster, more reliable installs for most existing packages**. The first transition phase implements easy and explicit means for a package maintainter to control which release file links are served to present-day installation tools. The first phase also includes the implementation of analysis tools for present-day packages, to support communication with package maintainers and the automated setting of default modes for controling release file links. The second transition phase will result in the current PYPI index to only serve PYPI-hosted files by default. Externally hosted files will still be automatically discoverable through a second index. Present-day installation tools will be able to continue working by specifying this second index. New versions of installation tools shall default to only install packages from PYPI unless the user explicitely wishes to include non-PYPI sites. I must say, don't like this change in motivation compared to V1 and V2. The original of the discussion was to make PyPI more secure and the installation process faster and more reliable by moving away from crawling arbitrary external web pages. Both can be had by: * limiting the crawling to package author defined specific URLs, with added hashes to make sure that the URLs and their target content is not modified (this is the securing external downloads part - see here for an example: https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5), and * adding a way for the package authors to say PyPI, please go ahead and cache/copy my distributions files (this is the increase download reliability part - can be had by doing opt-in CDN caching/proxying of external links via PyPI) Now, with V3 of the proposal, you are moving towards a system that basically says do it this way, or stay out of our eco system, which, in my book, is not what the Python eco system is all about. I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site. This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option. Maybe i'm missing it but in what way does this force authors to do it this way or stay out of our eco system since all the same options are available as there are today? Your V2 was much more inviting in this respect. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 13 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On 13.03.2013 20:08, Donald Stufft wrote: On Mar 13, 2013, at 2:57 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 12:21, holger krekel wrote: [V3 proposal] I must say, don't like this change in motivation compared to V1 and V2. The original of the discussion was to make PyPI more secure and the installation process faster and more reliable by moving away from crawling arbitrary external web pages. Both can be had by: * limiting the crawling to package author defined specific URLs, with added hashes to make sure that the URLs and their target content is not modified (this is the securing external downloads part - see here for an example: https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5), and * adding a way for the package authors to say PyPI, please go ahead and cache/copy my distributions files (this is the increase download reliability part - can be had by doing opt-in CDN caching/proxying of external links via PyPI) Now, with V3 of the proposal, you are moving towards a system that basically says do it this way, or stay out of our eco system, which, in my book, is not what the Python eco system is all about. I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site. This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option. Maybe i'm missing it but in what way does this force authors to do it this way or stay out of our eco system since all the same options are available as there are today? The proposal marks all external links as evil, and instead of making external links more secure, the user is left with the option to either not enable external links at all, or to let the devil in :-) That's not nice. It's also security theater. The real problem is unreviewed code getting executed by users, or worse, automated build systems. Yet, we let users believe that everything is secured on PyPI. Taking an extreme position, it would probably be better just leave everything as it is and instead educate users about the risk they are taking with a pip install AngryBirds, signed with keys issued by the PSF on the official PyPI server, delivered straight to your drive via the latest in crypto technology, only to wipe your notebook... But then, I don't like extreme positions, so would rather like to incrementally improve the situation both from the server and the client side, both addressing user and author concerns, and keeping the Python eco system a friendly place to be. Your V2 was much more inviting in this respect. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 13 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 3:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 20:08, Donald Stufft wrote: On Mar 13, 2013, at 2:57 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 12:21, holger krekel wrote: [V3 proposal] I must say, don't like this change in motivation compared to V1 and V2. The original of the discussion was to make PyPI more secure and the installation process faster and more reliable by moving away from crawling arbitrary external web pages. Both can be had by: * limiting the crawling to package author defined specific URLs, with added hashes to make sure that the URLs and their target content is not modified (this is the securing external downloads part - see here for an example: https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5), and * adding a way for the package authors to say PyPI, please go ahead and cache/copy my distributions files (this is the increase download reliability part - can be had by doing opt-in CDN caching/proxying of external links via PyPI) Now, with V3 of the proposal, you are moving towards a system that basically says do it this way, or stay out of our eco system, which, in my book, is not what the Python eco system is all about. I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site. This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option. Maybe i'm missing it but in what way does this force authors to do it this way or stay out of our eco system since all the same options are available as there are today? The proposal marks all external links as evil, and instead of making external links more secure, the user is left with the option to either not enable external links at all, or to let the devil in :-) That's not nice. It's also security theater. The real problem is unreviewed code getting executed by users, or worse, automated build systems. Yet, we let users believe that everything is secured on PyPI. Taking an extreme position, it would probably be better just leave everything as it is and instead educate users about the risk they are taking with a pip install AngryBirds, signed with keys issued by the PSF on the official PyPI server, delivered straight to your drive via the latest in crypto technology, only to wipe your notebook... But then, I don't like extreme positions, so would rather like to incrementally improve the situation both from the server and the client side, both addressing user and author concerns, and keeping the Python eco system a friendly place to be. Your V2 was much more inviting in this respect. Perhaps it would be better to decide whether it is reliability theater and concentrate on consistency rather than whether the code actually does what you want. It is nice to have a system that at least prevents targeted third party bad-package attacks. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Mar 13, 2013, at 3:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 20:08, Donald Stufft wrote: On Mar 13, 2013, at 2:57 PM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 12:21, holger krekel wrote: [V3 proposal] I must say, don't like this change in motivation compared to V1 and V2. The original of the discussion was to make PyPI more secure and the installation process faster and more reliable by moving away from crawling arbitrary external web pages. Both can be had by: * limiting the crawling to package author defined specific URLs, with added hashes to make sure that the URLs and their target content is not modified (this is the securing external downloads part - see here for an example: https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5), and * adding a way for the package authors to say PyPI, please go ahead and cache/copy my distributions files (this is the increase download reliability part - can be had by doing opt-in CDN caching/proxying of external links via PyPI) Now, with V3 of the proposal, you are moving towards a system that basically says do it this way, or stay out of our eco system, which, in my book, is not what the Python eco system is all about. I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site. This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option. Maybe i'm missing it but in what way does this force authors to do it this way or stay out of our eco system since all the same options are available as there are today? The proposal marks all external links as evil, and instead of making external links more secure, the user is left with the option to either not enable external links at all, or to let the devil in :-) It doesn't mark them as evil, it marks them as requiring users to opt into them. Authors are free to not publish their packages directly to PyPI and users are free to opt in to installing the external urls that the authors haven chosen to publish. Further more it gives package authors complete control over what urls appear on their simple index page. ISTM that this is even friendlier than before because now both sides have explicitly decided to use those urls, instead of it being completely implicit on one said, and partially implicit on the other. That's not nice. It's also security theater. It's not security theater, it moves the defaults to more secure. Further work can (and will be) to ensure that for those users and authors who opt into the external urls it's still secure while again requiring both sides to explicitly opt into it. The real problem is unreviewed code getting executed by users, or worse, automated build systems. Yet, we let users believe that everything is secured on PyPI. We? I' don't think anyones ever said that *everything is secured on pypi*. The best the PyPI infrastructure and tooling can do (security wise) is to try and make as sure as possible then when you ask for foo==X.Y PyPI currently can't make that claim for external links. On top of that many users (and i'd wager most users) are not aware that when they install something it reaches outwardly to other hosts. This proposal makes it so they *are* aware so they opt into potentially lowering their downtime and they opt into exposing details to external hosts (which may or may not be SSL secured). Taking an extreme position, it would probably be better just leave everything as it is and instead educate users about the risk they are taking with a pip install AngryBirds, signed with keys issued by the PSF on the official PyPI server, delivered straight to your drive via the latest in crypto technology, only to wipe your notebook... But then, I don't like extreme positions, so would rather like to incrementally improve the situation both from the server and the client side, both addressing user and author concerns, and keeping the Python eco system a friendly place to be. Your V2 was much more inviting in this respect. This gives _all_ the abilities of the current system (besides spidering random urls) with *more* control given to the authors as to what exists on their various index pages. This is a net win for everyone involved. The only loss is that projects that choose to host externally to PyPI will have people trying to install it told to explicitly allow it (as mentioned by PJ Elby). -- Marc-Andre
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On 03/13/2013 01:33 PM, M.-A. Lemburg wrote: The proposal marks all external links as evil, I'm sorry the text of the PEP gave you that impression. I can see how you'd have gotten it from some of the comments here on catalog-sig, but we went to some lengths to avoid it in the PEP text, and plan to further revise the text to try harder to avoid that implication. In the proposed PEP, we are attempting to balance two things that I believe to be true: 1) There are good and valid reasons for some package owners to prefer external hosting, and it is good for automated installers to easily be able to install such packages (on user request). 2) Installing non-PyPI-hosted packages should not be the *default* behavior of installer tools, for many reasons, among them because that is unusual and surprising behavior to many newcomers to the Python ecosystem, and often leads to concerns on their part about the stability of the ecosystem. These are the axioms, if you will, of this proposal, and while I'd guess many people in this discussion are at least slightly uncomfortable with one or the other of them, I think accepting both is the most likely path to a compromise everyone can live with. I think we can find a solution that embraces both these axioms and maintains good backwards-compatibility and usability. Holger and I had a long talk this evening about that, and here are some of our thoughts: A) You mentioned opt-in PyPI caching of externally-hosted files as a means to improve reliability. We basically agree, but implementing this on the PyPI side adds complexity to the PyPI implementation that we are hesitant to propose. Rather, we propose that this is better handled by a client-side tool that you point at a PyPI release with externally-hosted files, and it simply copies those release files onto PyPI. This has essentially the same effect. We envision this being a simple enough tool that it could reasonably be run for every release of a project in an ongoing way, not just as a one-time project-wide migration. We plan to change the line in the PEP that says the existence of this tool is NOT REQUIRED to begin the phase 2 transition to instead say that the existence of this tool IS REQUIRED before the phase 2 transition begins. (Holger already has a partial implementation of this tool.) B) We also plan to change the PEP to say even more strongly that installer tools should provide an easy option for installing externally-hosted projects, and that our definition of easy includes the ability for an installer to automatically tell a user what options they can use to install a specific externally-hosted package that the tool is refusing to install by default. C) To make that latter part of (B) easier, we also propose that the basic simple index include a link with a distinct rel attribute that points to the -with-externals index page for that project, only for a package that has external links. This way even tools using the no-externals index by default can notify users of the existence of external links for a project when they try to install it. There's also another possible change, a bit more significant, that we discussed that I'd be curious to hear your thoughts on. The initial motivation for separating external links from the main simple/ index was twofold: 1) Allow future tools to distinguish between internal and external links without every tool needing to implement host-comparison algorithms (which may break indexes that host internal files on a CDN), and 2) Allow today's installers, without upgrade, to automatically migrate eventually to no-external-installs-by-default. Some things have caused us to re-evaluate these points: - PyPI can automatically tag internal/external links in the simple index with rel=internal and rel=external, which gives future tools a more reliable marker than host-comparison. So this takes care of #1. - It may be that giving up #2 is acceptable in the interest of better backward-compatibility. Old tools will still gain most of the benefits of this PEP due to the eventual elimination of automatic link-scraping (both from metadata and external pages) and the move to explicit submission of external links, only for those projects that want them. And old tools will not be able to provide a useful error message to users trying to install an externally-hosted package that is no longer listed in the main simple/ index, which is a bad usability breakage. Given that, we are thinking of perhaps simplifying the PEP to eliminate the separate -with-externals index, and list external links in the main simple/ index, clearly marked with rel=external. The PEP would still recommend that future installer tools not follow rel=external links without specific user authorization. Old tools still get many of the benefits, without the breakage. and instead of making external links more secure, the user is left with the option to either not enable external links at all, or to let the devil in
