Re: [Catalog-sig] How to determine if archive is an sdist or bdist
On Mon, Apr 1, 2013 at 8:13 AM, James Carpenter nawk...@gmail.com wrote: Do you have a module/function/line number in easy_install I should use? I'm sure I can dig it out myself but it sounds like you might just be able to put your finger on it in only a minute or two. Same question for a pre-existing utility function for reading a requirements file. I'm guessing there is one burried down in the PIP code, but I haven't looked yet. A third more interesting question, is whether there is any way to determine the origin of an installed package. I am building up a build-info to push into Artifactory and it takes a type if available when listing dependencies of a build. In Java/Maven land a typical type would be jar, war, pom, etc. My current understanding is this is a non-sense question, since once a Python module is installed it looks the same regardless of whether it came from an sdist, wheel, etc. The closest equivalent is PEP 376's INSTALLER file, but that only tells you which installer was used to add the distribution, rather than where that installer retrieved it from. An ORIGIN entry in the installation database could be an interesting future addition. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Distutils] Merge catalog-sig and distutils-sig
On Fri, Mar 29, 2013 at 7:47 PM, Richard Jones rich...@python.org wrote: On 29 March 2013 14:45, Tres Seaver tsea...@palladion.com wrote: If we leave the main list the 'distutils-sig', and just announce that 'catalog-sig' is retired, folks who want to follow the new list just switch over. All the archives (mailman / gmane / etc.) stay valid, but the list goes into moderated mode. Whoever has the power to do this, do it please. +1 distutils-sig it is. We're expanding the charter to the distutils standard library module, the Python Package Index and associated interoperabilty standards, but that's a lot easier than forcing everyone to rewrite their mail filters. Besides, it's gonna be a *long* time before the default build system in the standard library is anything other than distutils. Coupling the build system to the language release cycle has proven to be a *bad idea*, because the addition of new platform support needs to happen in a more timely fashion than language releases. The incorporation of pip bootstrapping into 3.4 will also make it a lot easier to recommend more readily upgraded alternatives. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
On Wed, Mar 13, 2013 at 1:23 AM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 07:28, Nick Coghlan wrote: On Tue, Mar 12, 2013 at 12:59 PM, M.-A. Lemburg m...@egenix.com wrote: I think we should establish a versioned API like that for PyPI to make progress easier. All major web APIs use versioning for this reason. Why set up versioning for something we want to phase out? There will never be a simple-v3, so this is really overengineering the proposed change. Who says that we want to phase out the /simple/ index ? I want to render it redundant, because it's a crazy way to distribute completely inadequate metadata. Cheers, Nick. FWIW, I don't think that two or three small changes to the PyPI (see my email to Holger) server warrants calling this over-engineering. This is about moving forward in a backwards compatible and future proof way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 13 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
On Wed, Mar 13, 2013 at 11:19 PM, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Mar 13, 2013 at 1:23 AM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 07:28, Nick Coghlan wrote: On Tue, Mar 12, 2013 at 12:59 PM, M.-A. Lemburg m...@egenix.com wrote: I think we should establish a versioned API like that for PyPI to make progress easier. All major web APIs use versioning for this reason. Why set up versioning for something we want to phase out? There will never be a simple-v3, so this is really overengineering the proposed change. Who says that we want to phase out the /simple/ index ? I want to render it redundant, because it's a crazy way to distribute completely inadequate metadata. Specifically, once we have the infrastructure in place to publish metadata v2.0 (or a suitable subset) to installation tools, the relatively impoverished contents of the simple index will be a legacy interface retained only to preserve the correct operation of existing tools. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 5:16 PM, Carl Meyer c...@oddbird.net wrote: There is no instead of. There are parallel proposals (see the TUF thread) to improve the security of the ecosystem, and those proposals are not mutually exclusive with this one. If you search the PEP text, note that you don't find the words secure or security anywhere within it, or any claims of security achieved by this proposal alone. There is a brief mention of MITM attacks, which is relevant to the PEP because avoiding external link-crawling does reduce that attack surface, even if other proposals will also help with that (even more). Right, the changes to provide end-to-end security require more extensive changes and need to be given appropriate consideration before we proceed to implementation and deployment. This PEP, especially with the additional changes you propose here is an excellent approach to *near term* improvement, as a parallel effort to the more complex proposals. The /simple/ index will also be around for a long time for backwards compatibility reasons, regardless of any other changes that happen in the overall distribution ecosystem. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Thu, Mar 14, 2013 at 7:13 AM, Justin Cappos jcap...@poly.edu wrote: Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. I think what you're proposing now is a pretty good place to state (although I'm suggesting making it even simpler in the near term by starting by focusing on the PyPI-end user link, and then moving to delegating signing of the per-project metadata to the individual projects as a later step) If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. I'd like to do a distribution 2.0 at some point where we make the simple index redundant by including that info (and more) directly in the TUF metadata, but I think that's a later project - securing what we have now is a better place to start. Cheers, Nick. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Publishing metadata (was: V2 pre-PEP: transitioning to release file hosting on PYPI)
On Thu, Mar 14, 2013 at 12:54 AM, M.-A. Lemburg m...@egenix.com wrote: The index itself is just a bag of things and, as such, one that's very well suited to publish data, since it can easily be exposed in form of static files, which can be put on a CDNs or mirrored using rsync. The TUF metadata is also just a collection of static files which can be put on CDNs and mirrored using rsync. That's one of the reasons TUF is an interesting approach :) It's easy to add the metadata file to that index for tools to pick up - in addition to the other data exposed on the index pages and perfectly backwards compatible. As mentioned before, I think we should start publishing the existing metadata stored in the PyPI database on those index pages as PKG-INFO files, so that tools can easily access the data without having to go through XML-RPC. Yes, I think that's a good near term approach. However, there's still a lot of duplication of functionality between the TUF metadata and the simple index, so if we get TUF-based security up and running, my long term aim will be to make it so that once you have downloaded the TUF metadata, you shouldn't *need* anything from the simple index, and would be able to go directly to downloading the release files. That's a longer term idea, though and we may even decide it isn't worth the hassle if PKG-INFO is made available through /simple. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
On Tue, Mar 12, 2013 at 12:59 PM, M.-A. Lemburg m...@egenix.com wrote: I think we should establish a versioned API like that for PyPI to make progress easier. All major web APIs use versioning for this reason. Why set up versioning for something we want to phase out? There will never be a simple-v3, so this is really overengineering the proposed change. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
On Tue, Mar 12, 2013 at 11:41 PM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: Hello everyone, I am pleased to announce our demonstration of PyPI and pip with TUF. Firstly, we solicit your thoughts and comments on our design document for integrating PyPI with TUF: https://docs.google.com/document/d/1sHMhgrGXNCvBZdmjVJzuoN5uMaUAUDWBmn3jo7vxjjw/edit?usp=sharing Thanks for putting this together! Just a few notes regarding key management: - the PSF board generally stays out of the technical details of running the python.org infrastructure, so it's likely that any root keys would be handled by the PSF infrastructure committee. A (2, 4) or (3, 5) trust configuration would likely be manageable at this level. - at the target delegation level, PyPI supports the registration of new projects through the web service (see http://docs.python.org/2/distutils/packageindex.html). If my understanding of target delegation is correct, this means the simple and packages/source/letter delegations will need to be (1, 1) and online. - higher levels of the target delegation hierarchy could conceivably be kept offline, but there seems little value in doing so if they're trusting on online (1, 1) key - many PyPI packages are maintained by single developers, so (1, 1) or (1, n) is likely to be the only generally feasible level of signing at the project level. With the current focus being on getting an improvement from the status quo that we can successfully deploy in a reasonable period of time, the target delegation side of things probably needs to be substantially simpler in the initial iteration. Yes, it leaves us open to certain vulnerabilities we would like to remove in the long run, but we need to be very cautious in the additional demands we place on the users uploading to PyPI. It may even mean the initial iteration allows projects to rely on a PyPI provided signing key for their TUF metadata, using the existing upload mechanisms to add the files to PyPI. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
That looks pretty good to me. My only comment is that qualifiers like new don't age well in an API. The explicit nocrawlhomepage and nocrawldownload might be a better choice. Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site
On Mon, Mar 11, 2013 at 9:32 PM, Donald Stufft don...@stufft.io wrote: I know your joking but if this is an actual limiting factor my next proposal will be to change the name :]. PyPR would not only be more accurate, it would actually get rid of the confusion with PyPy. We'd get a new pronunciation argument (Pie-pee-arr vs Pie-per) to corresponding with the existing one, though (Pie-pee-eye vs Pie-pie) Hell, the next generation of PyPI is going to have a different enough architecture for metadata distribution that a name change may be entirely appropriate :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Deprecate External Links
On Thu, Feb 28, 2013 at 5:01 PM, Donald Stufft donald.stu...@gmail.com wrote: I'm glad the next set of Metadata won't have external links, however even if it showed up tomorrow it's going to be a long time until people are completely migrated to it. Furthermore you estimate months but the first phase will have positive benefits right away, namely that it will prompt people to start uploading their packages better increasing the security and reliability of the current system. And finally while I'm glad to see forward movement It's been said before not to bother making a fix to the existing system because X was going to happen soon, in the past i was distutils2/packaging, now it's PEP426/packaging. While I have every hope and I believe it will happen this time, the past has made me worry about holding off on good incremental improvements to the current infra. Pissing off the maintainers off packages that currently rely on external hosting by telling them they have to change their release processes if they want to keep releasing software on PyPI and have their users actually be able to download it is *not* a good idea, especially when we're about to ask them to upgrade their build chains for other reasons (including both security and reliability). Working on the installation tools and getting them to complain about external links is a *fine* idea. It doesn't break anything, but maintainers will start fielding questions from their users asking Hey, why am I getting this warning when I install your project?. Working on the upload tools and having them *warn* distributors that self-hosting is problematic is also a good idea, as is exploring PJE's suggestions about refining the set of URLs that PyPI currently publishes However, getting PyPI to effectively *break uploads* of projects that rely on external links at this point in time is *not* a good idea: we should NOT mess with people's existing build and upload processes lightly, as any such changes burn up a *lot* of community good will, and that's not something in great supply when it comes to Python's software distribution infrastructure. All current generation infrastructure should continue to work without modification on both the upload side *and* the download side (although, as noted above, it's highly desirable for both the upload side and the download side to be updated to warn users about any reliance on insecure legacy behaviour). I expect a similar rollout in the transition to the next generation metadata format and distribution infrastructure - initially download tools will support both formats, emitting a warning when falling back to the legacy distribution infrastructure, then they will start requiring an option to enable fallback to legacy mode, and eventually there will be released installation tools that don't support the legacy distribution infrastructure at all (such as any default installer included in the standard library). For *next* generation infrastructure, it's our job as system designers to sell it to potential users (primarily everyone writing software which they publish on PyPI, or at least the authors of the toolchains used for that publication, but also to consumers of that software). The distutils2 team failed, in large part because their proposal required radical changes to the way people published Python software. I have deliberately moderated that in the way I have approached PEP 426 - if people can't generate the new metadata with only minor changes to their current processes, it isn't going to fly, and any trade-offs (such as the loss of external hosting support), need to be bought with corresponding benefits (such as guaranteed correct pre-release handling, solid Python version declarations, a clean post-install hook design, and, hopefully, a vastly improved rich metadata publication system for PyPI, probably based on TUF). Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Deprecate External Links
On Thu, Feb 28, 2013 at 6:12 PM, M.-A. Lemburg m...@egenix.com wrote: On 28.02.2013 07:39, Nick Coghlan wrote: 1. The next generation metadata infrastructure will NOT support external hosting of files indexed on PyPI - if you don't upload the archive files to PyPI, they won't be included in the next generation metadata. If you want external hosting, you will need to run a separate index (this is similar to the yum model - you can host files wherever you want, but you need to run yum createrepo yourself to generate the metadata, and instruct users on how to get their installers to retrieve your metadata. The big difference between PyPI and the yum model is that the default index still won't be curated at all, so there's no review process to get through if you want to use it, thus less need for external hosting). Could you elaborate on this ? AFAIK, the metadata only works on package names, regardless of where an installer finds them. Caveat: this is NOT a final design, and people that aren't me will be working out the exact details. It is, however, how I want it to work. The next generation metadata publication infrastructure is likely to be based on TUF, and thus will consist of pregenerated, signed metadata served as static files. Installers will just download metadata files, sdists and wheels (and probably eggs and tarballs), and never need to contact an active web service. The only active web service technically required will be one to regularly refresh the signed timestamp file that prevents certain kinds of attacks based on providing old, insecure versions of software (a cron job running on the server hosting the metadata would suffice for this task). PyPI itself will have another active service to automatically regenerate the metadata when files are uploaded by maintainers. The delegation of trust within the framework will be defined only for files hosted by PyPI - it will not be extended to allow the declaration of external URLs as a source for the target files. Publishers will still be able to publish on external sites, but they will need to generate their own metadata, and distributions published that way won't be indexed in the next generation metadata on PyPI. This is the same way yum repos work - the metadata for each repo only covers SRPMs and RPMs hosted in that repo. If you want to download software from somewhere else, you have to add another repo definition in the client so it knows where to look for the metadata. APT works in a similar fashion. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Deprecate External Links
On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft donald.stu...@gmail.com wrote: Sometimes you need to break things. The goal is to do it with ample warning and migration time so that people have a chance to move to the new way of doing things. Again, I am not suggesting we delete all external links immediately, just disable new ones. Removing old ones will come later. This thread is long enough that I'm not sure on where to weigh in. Here seems appropriate enough. 1. The next generation metadata infrastructure will NOT support external hosting of files indexed on PyPI - if you don't upload the archive files to PyPI, they won't be included in the next generation metadata. If you want external hosting, you will need to run a separate index (this is similar to the yum model - you can host files wherever you want, but you need to run yum createrepo yourself to generate the metadata, and instruct users on how to get their installers to retrieve your metadata. The big difference between PyPI and the yum model is that the default index still won't be curated at all, so there's no review process to get through if you want to use it, thus less need for external hosting). 2. Near term, with the current generation infrastructure, I think it's better to approach the problem *very* gently. Our political capital with users is low at this point, and we need to prioritise what things we want to make people angry about (whether or not we consider their anger justified is completely irrelevant). This proposal is for a transition that would take months. Since I want to have the next generation metadata up and running within months *anyway*, that means this strikes me as primarily a distraction from fixing the problem properly. 3. Various other problems raised in this thread will only be fixed with next generation metadata that the automated tools can *rely* on rather than having to guess the intended semantics. That's why PEP 426 is now explicit about pre-release handling, and why it makes version specifiers like (for example), Requires-Python: 2.6 exclude Python 3 by default. (although the thread does raise an interesting question of whether or not you can cleanly specify dual Python 2 3 support given the current state of PEP 426) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] User profile: PGP Key ID
On 21 Feb 2013 06:57, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 20, 2013 at 3:50 PM, Daniel Holth wrote: Bikeshed detected. Basically. We basically can't use any of the properties of the various signing techs besides their ability to sign documents so the choice of them doesn't particularly matter. Not *quite* true - GPG comes with more mature client side tech for managing signing keys at the developer end, and that's independent of the PyPI trust model. Since it's a coin flip otherwise, that's probably going to be enough for us to favour GPG as the signing tech. In the spirit of status quo wins a stalemate, GPG should currently be considered the default choice, with alternatives needing to offer genuinely compelling advantages to displace it. (note that isolating the signature generation and verification to a separate non-Python process isn't a major issue from my point of view) Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] New PyPI stats available
On Tue, Feb 19, 2013 at 10:18 AM, Richard Jones rich...@python.org wrote: On 19 February 2013 10:03, Éric Araujo mer...@netwok.org wrote: Le 18/02/2013 17:58, Richard Jones a écrit : Thanks, I'm aware of vanity; it reports on one package. The latest-totals is all packages. I can’t say if you’re talking about a project, a release or a distribution here. package is a name registered on PyPI. release is a version of a package. A distribution isn't referred to as far as I'm aware, but could be a label applied to what PyPI calls package file - a single file related to a release. We've been trying to move to distribution as the thing projects register on PyPI to better distinguish them from the kind of package you can import directly. The general taxonomy as I understand it: - projects are an overall activity. They have policies, bug trackers, source control systems, mailing lists, developers, etc and may control multiple distributions. Hence Project-URL - distributions are what you register on PyPI: you intend to distribute Python software using that name. Hence Requires-Dist, etc. - packages and modules are the things you can actually import at runtime - most, but not all, distributions will ship exactly one module or package with the same name as the distribution - a version is a [pre-|post-]release as described in the metadata spec - sdists are source archives for a particular version of a distribution - wheels are binary archives for a particular version of a distribution - eggs are binary archives in an alternative (discouraged) format We can get away with PyPI continuing to be the Python *Package* Index (rather than the Python *Distribution* Index), because most distributions contain packages, and it isn't worth the hassle of trying to change it. It would be good to have PyPI calling distributions by that name in the UI, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] New PyPI stats available
On Tue, Feb 19, 2013 at 1:35 PM, Daniel Holth dho...@gmail.com wrote: Who will remember the distinction without a glossary? Creating and publishing a glossary is on the list... (actually pretty high on the list now that PEP 426 is in mostly done status) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] HTTPS now promoted on PyPI
On Tue, Feb 19, 2013 at 3:13 PM, Richard Jones r1chardj0...@gmail.com wrote: Hi all, I've just altered the nginx configuration to promote (ie. redirect to) HTTPS for all GET/HEAD requests. This includes HSTS, but I've set the lifetime to 1 day just in case there's some HTTPS compatibility issues. Once it's bedded down I'll bump it to a year. I looked into distutils, but since it uses urllib and urllib just raises an error on 307 redirects we're a little stymied as to what we can actually do for POSTs for it... We really need to fix distutils to replace the HTTP URL with HTTPS and handle .pypirc issues. At this point I believe our options are: 1. live with it, 2. incorporate some monkey-patching into distribute and setuptools and promote those, 3. write a stand-alone uploader (or add such functionality to pip) which can monkey-patch distutils, 4. fix distutils (and accept a long lead time to actual impact), or I suggesting getting in touch with Benjamin Petersen and Georg Brandl ASAP (e.g. through a release blocker for 2.7 and 3.3 on the issue tracker), as Python 2.7.4 and Python 3.3.1 are planned for this month. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Proposal for the bootstrap API
On Fri, Feb 15, 2013 at 7:28 PM, Tarek Ziadé ta...@ziade.org wrote: Looks completely legit to me, unfortunately... So until we catch that fish, damage can already be done. When you're already in a (security) hole, the first thing you need to do is *stop digging*. We have a handful of projects which need to trusted way to distribute a Python script in order to bootstrap installation tools on current versions of Python. That's a real problem, and this proposal is a good solution for that. Generalising that to grant the ability to upload arbitrary bootstrap scripts to every project for no good reason is making a bad situation worse, for zero payoff. So let's not do that. For projects other than distribute or pip, the bootstrap process should be: 1. Bootstrap pip 2. pip install project Or, if the project needs egg support: 1. Bootstrap distribute 2. easy_install project Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] RubyGems Threat Model and Requirements
On Thu, Feb 14, 2013 at 6:46 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 13 Feb, 2013, at 15:21, Nick Coghlan ncogh...@gmail.com wrote: For now, though, we would probably start off with release/target/timestamp roles sharing a key, all threshold values set to 1, and just doing simple project based target delegation to user keys. Given the existing GPG infrastructure, I'm also inclined to stick with GPG based keys and work with the TUF folks to define that format in their spec. We may also need to leave the protection against replay attacks off by default, do to the problem with incorrect clocks noted at the end of the TUF spec. On the other hand, AFAIK the GPG infrastructure of PyPI isn't used a lot and OpenSSL exposes PCKS#1 which means those can be used without adding new dependencies to CPython, and likely without installing new software on all unix-like systems (the openssl command-line tool is available on both osx and all linux boxes I checked). For 3.4 the PCKS#1 support in openssl could be exposed through a new extension module. I don't have preferences either way, I believe the main challenge in either case is Windows. For GPG, the suggestion is for the pip folks to figure out a way to bundle GPG (which has its own problems), but there's no current suggested Windows approach for PCKS#1. I wonder if a way could be found to retrieve it from CPython's bundled OpenSSL on Windows? Currently, I expect we're going to have to rely on pip or distribute to handle the signed upload side of things, since legacy distutils is unlikely to be fully compatible with whatever new scheme we come up with. *If* we can come up with an approach that integrates with the legacy GPG signing capability in distutils, that would be a massive point in favour of using GPG. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Allowing the upload of .py files at PyPI
On 15 Feb 2013 05:50, Tarek Ziadé ta...@ziade.org wrote: On 2/14/13 8:37 PM, Donald Stufft wrote: On Thursday, February 14, 2013 at 2:28 PM, Tarek Ziadé wrote: Hello Some tools (setuptools, distribute, zope, pip) use bootstrap files to get installed, In order to have a more secured installation process, we'd like to be able to push those files on PyPI so people can download them through https using the PSF certificate. As Phillip Eby noticed, that requires changing this method https://bitbucket.org/loewis/pypi/src/f18ce0fbe947c1ce778761ea81d6704572cebb24/webui.py?at=default#cl-2233 by: - allowing .py extensions, - allowing arbitrary file names when they have the .py extension Arbitrary file names is a bad idea imo. What's to stop me from uploading setup_distribute.py and linking to it as if it was distribute_setup.py and installing a malware'd distribute. If you can upload in that location, it means you are a legit owner/maintainer of the project AFAIK I'm more concerned about phishing style attacks. I don't want the PyPI admins to have to start scanning for hostile names like distirbute. So how often do the bootstrap files change? If relatively frequently, I would prefer this to be a project-specific privilege granted by the PyPI admins (at least for now). If rarely, then I'd be happy enough if the update process required PyPI admin involvement (the project whitelist is probably a better idea, though). Cheers, Nick. -- Tarek Ziadé · http://ziade.org · @tarek_ziade ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Allowing the upload of .py files at PyPI
On 15 Feb 2013 08:38, Donald Stufft donald.stu...@gmail.com wrote: On Thursday, February 14, 2013 at 5:34 PM, M.-A. Lemburg wrote: I don't follow the reasoning here. What's the difference between uploading a .py file and a .tar.gz file ? AFAIK, the only reason why the file extensions are restricted is to prevent people from uploading MP3s, movies or other material that doesn't belong on PyPI - not because there are security concerns. Personally (might by different for Nick) it's less a problem with uploading .py files and more a problem with allowing arbitrary names. The sensible security mindset is to only open yourself up to attack vectors when you have no other choice. Since phishing attacks on the bootstrap scripts can be prevented categorically with a whitelist (even a hardcoded one at this point), the onus should be on others to explain why we should leave the bootstrap scripts open to such attacks. The difference relative to releases is that those *have* to be open access for PyPI to work. The same is not true for the bootstrap scripts - any other package can automate its installation by bootstrapping pip, and then installing itself. There's no need to declare open season on Python file uploads, therefore we shouldn't do so. Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] RubyGems Threat Model and Requirements
On Wed, Feb 13, 2013 at 7:58 PM, Giovanni Bajo ra...@develer.com wrote: Il giorno 13/feb/2013, alle ore 04:31, Nick Coghlan ncogh...@gmail.com ha scritto: TUF's target delegation is thus in direct competition to the trusted keys file in your design. TUF specifically aims to take care of the online key needed problem, by confining the online key role to the generation of the timestamp file, with offline keys used to sign the regenerated metadata when a target delegation changes. Does this mean that adding a package to PyPI, adding a maintainer to a package, removing a maintainer from a package, etc. all require an offline operation in this model? If I'm reading the spec correctly, you can decide what level of consequence you want if an attacker compromises the master server. What TUF appears to allow (but does not require), is for the access levels to be split as follows: Distribution server: can update timestamp file, but not upload new releases or add/remove project keys. Only needs the timestamp role key Upload server: can add/remove files within target subtrees that have been delegated appropriately to user keys. Needs the release role key in order to publish the resulting metadata updates. Management server: can add/remove target delegations (e.g. to create new projects, or add new maintainers). Needs the target role key in order to update the target delegations and the release role key in order to publish them. Offline: the root key is the one that the clients actually trust, and it's the one that is used to identify the target, release and timestamp keys on the distribution server. If *this* gets compromised, you're pretty much screwed until the clients are updated to blacklist it. The only time you should need this key is if you need to change the server keys (e.g. following a server compromise) Since PyPI currently uses a single server for management, upload and distribution, it probably doesn't make sense to use all four roles (at least initially). An offline root key, and a single master key used to sign the online metadata is probably the best we can do within the current PyPI architecture (and perhaps ever, given it's nature as an almost completely open server). By contrast, if you were only publishing releases from a private build service, then the *only* internet facing machine would be your distribution server, and thus the only directly exposed key would be the timestamp key. You could then lock down your upload and management functions as much as you wished. (As near as I can tell, the advice in section 6.1 of the TUF spec to keep all keys other than the timestamp key offline is just plain wrong for a system with PyPI's current architecture, where the same public web service that distributes the software also needs to let people create new projects and update their keys. If I'm wrong, maybe Justin will be able to explain how at some point...). I wouldn't oppose it. In fact, I was just scared that such a model would not be accepted as it would break too much flexibility (within my design, the equivalent would be having the trust file signed by a master PyPI GPG key, whose fingerprint is hardcoded in pip, and it is kept offline; or it might even be online, but on a separate signing server, that eg. logs everything it does by email to a public mailing list, like Adding this fingerprint to django). Yep, I think you've nailed the equivalence there. The reasons I still favour the TUF model at this stage is that it lets us start with a simple single-online-key, and target delegation direct to projects model (i.e. something functionally equivalent to your proposal, with TUF's releases.txt and the target delegation files together fulfilling the role of your trust file), while still leaving the door open to various future improvements like: 1. separating out PyPI's management/upload server from the distribution server (with the latter only having access to a timestamp key) 2. permitting delegation to umbrella projects/organisations rather than directly to individual projects 3. migrate to using the custom fields in the TUF file formats for metadata publication 4. use TUF's threshold mechanism to allow projects to opt in to a higher security configuration where multiple signatures are needed for a release to be trusted We may never do any of those things, but they're all good options to have open to us (in particular, metadata publication through TUF is something you can scale with a CDN, and inherently comes with markers to indicate what has changed since you last looked at the index. You can't easily do that with XML-RPC calls or the current PyPI simple project listing). For now, though, we would probably start off with release/target/timestamp roles sharing a key, all threshold values set to 1, and just doing simple project based target delegation to user keys. Given the existing GPG infrastructure, I'm also inclined to stick with GPG based keys and work
Re: [Catalog-sig] RubyGems Threat Model and Requirements
On 14 Feb 2013 03:59, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 13, 2013 at 5:29 AM, Robert Collins wrote: On 13 February 2013 15:12, Giovanni Bajo ra...@develer.com wrote: Yes, that's correct. GPG chain-of-trust concept is not used in my proposal, because I don't think it would be a good fit for this problem given its requirements. Specifically, I believe pip users should not be bothered with useless click-through questions for each new package they install, which is what you would get far too often in case chain-of-trust were used. But this means someone that gets access to the PyPI server can just mark their own key as trusted and compromise any package they want. -Rob I used to have the same idealistic idea that we should be able to *not* trust PyPI for the average user. However PyPI *is* the final authority on who has the right to publish to what name. It would be a bit like trying to determine if the PSF owns python.org without involving the company running the .org TLD. I see it as similar to the SSL CA system - it has plenty of known flaws, but still closes a whole lot of attack vectors, and thus is worth doing. Particularly security conscious users will still be able to do their own verification, or pay a redistributor to do additional verification on their behalf. (For example, I expect you would fail all the meaningful Common Criteria EAL certification levels if you blindly trusted PyPI). Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] RubyGems Threat Model and Requirements
On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo ra...@develer.com wrote: Hello Nick, I've added the initial Requirements and Thread Model section to my document. I've also added a section Future scenarios at the end of the document. I hope they complete what you were feeling was missing from the proposal. Thanks, that helps me a lot in understanding the overall goals of your approach - in particular, it more clearly puts several things out of scope :) Your Task #6/#7 (related to PyPI generating the trust file, and pip verifying it) are the ones where I think the input of the TUF team will be most valuable, as well as potentially the folks responding to the rubygems.org attack. The rubygems.org will also be looking at server side incident response - I suspect a lot of that side of things will end up running through the PSF infrastructure team moreso than catalog-sig (although it may end up here if it involves PyPI code changes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] RubyGems Threat Model and Requirements
On Wed, Feb 13, 2013 at 2:27 AM, Giovanni Bajo ra...@develer.com wrote: Il giorno 12/feb/2013, alle ore 14:12, Nick Coghlan ncogh...@gmail.com ha scritto: On Tue, Feb 12, 2013 at 10:09 PM, Giovanni Bajo ra...@develer.com wrote: Hello Nick, I've added the initial Requirements and Thread Model section to my document. I've also added a section Future scenarios at the end of the document. I hope they complete what you were feeling was missing from the proposal. Thanks, that helps me a lot in understanding the overall goals of your approach - in particular, it more clearly puts several things out of scope :) Your Task #6/#7 (related to PyPI generating the trust file, and pip verifying it) are the ones where I think the input of the TUF team will be most valuable, as well as potentially the folks responding to the rubygems.org attack. My undestanding is that #6/#7 are not currently covered by TUF. It's not explained very clearly in the spec, but #6/#7 are covered by TUF's target delegation concept (see https://www.updateframework.com/browser/specs/tuf-spec.txt#L241 and https://www.updateframework.com/browser/specs/tuf-spec.txt#L382) The PyPI target role key can delegate authority to individual package developer keys by delegating authority for the relevant paths within PyPI (i.e. the locations of the sdists and other files). This is discussed briefly at https://www.updateframework.com/wiki/SecuringPythonPackageManagement#Notes (where they note they have added an extra level of delegation to separate out the package specific delegations by first letter rather than dumping them all in one directory). TUF's target delegation is thus in direct competition to the trusted keys file in your design. TUF specifically aims to take care of the online key needed problem, by confining the online key role to the generation of the timestamp file, with offline keys used to sign the regenerated metadata when a target delegation changes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Distutils] imp.find_modules and namespaces
On 12 Feb 2013 07:56, Alessandro Dentella san...@e-den.it wrote:
On Mon, Feb 11, 2013 at 04:11:38PM -0500, PJ Eby wrote:
On Mon, Feb 11, 2013 at 11:40 AM, Alessandro Dentella san...@e-den.it
wrote:
I believe that this issue belongs to this list, please let me know if
I'm
wrong.
Suppose I have 2 packages:
jmb.foo
jmb.bar
distributed separately. Each has in jmb's __init__ a standard:
__import__('pkg_resources').declare_namespace(__name__)
or
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
I just realized that imp.find_module() will return fake values
imp.find_module('jmb', None)
may return (a tuple with) the path from the first package or from the
second. Many framework will fail to discover commands in the inner
module:
one is detailed here [1] another is Django way of getting
application's
commands.
I find it misleading to return a value that is not thorohly correct.
Is there a workaround? Is the current behaviour considered correct for
reasons I don't yet understand?
Since Python 2.5, the right way to do this is with
pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages()
(for a subpackage tree).
For your example, if I wanted to find just the subpackages of 'jmb', I
would do:
import jmb, pkgutil
for (module_loader, name, ispkg) in
pkgutil.iter_modules(jmb.__path__, 'jmb.'):
# 'name' will be 'jmb.foo', 'jmb.bar', etc.
# 'ispkg' will be true if 'jmb.foo' is a package, false if it's
a module
thanks for the answer but this way I need to really import jmb while
imp.find_module doesn't really import it. I *think* that django people
wanted to test if some modules are present w/o importing. If an import
occurs, since I already know the name it should have, it bettere to try
that
one directly.
In that case, not without upgrading to Python 3.3 and using the native
namespace package support. In earlier versions you *have* to import the
parent packages in order to run the code that populates the full path for
the namespace packages.
Cheers,
Nick.
sandro
*:-)
--
Sandro Dentella *:-)
http://www.reteisi.org Soluzioni libere per le scuole
http://sqlkit.argolinux.orgSQLkit home page -
PyGTK/python/sqlalchemy
___
Catalog-SIG mailing list
Catalog-SIG@python.org
http://mail.python.org/mailman/listinfo/catalog-sig
___
Catalog-SIG mailing list
Catalog-SIG@python.org
http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Sun, Feb 10, 2013 at 6:56 PM, Marcus Smith qwc...@gmail.com wrote: For many users, virtualenvs are their user install, and sudo global installs are pretty rare. So, putting in a lot of work to fix what to many seems like a rare behavior makes me a little hesitant. But many users isn't all I guess, and maybe I'm off on the many part, not sure. I guess it's still worthwhile to prevent *any* unnecessary root installs. Maybe user education is enough? I was going to add a section to pip's new cookbook on --user installs, and with all the focus on security now, it could be emphasized really strongly and explained why it's a good thing. This, along with adding informative messages when users fail writing to the global site might go a long way. Btw, the user site is visible in virtualenvs that have global access (and is lower in precedence than the virtualenv site-packages), so I'm pretty sure it can serve as a place for common packages, that the global site is often used for. The main use cases I've found for user site packages are I am not working on anything in particular, I am just using the Python interactive shell on my computer and I am using Python to script things on my computer (see dotfiles and checkoutmanager in http://blog.aclark.net//2013/02/08/i-love-checkoutmanager-and-dotfiles/ for good examples of the latter). If you're not in the habit of working on multiple projects, or those projects are low level ones with few or no dependencies (i.e. not web applications), then per-user installations are also a lot easier to manage than a special default virtualenv. As you note, such installations also have the benefit of showing up in all your virtualenvs that are set to allow the use of system packages. While virtualenv isolation is wonderful when you're working on cross-platform web applications, it's not an available option if you want access to packages that are distro specific and unavailable on PyPI (such as some of the infrastructure on Fedora/RHEL systems, including mockbuild, yum, etc). The main problem I see at the moment is the similarity in workflow between: $ yum install pkg # Error due to insufficient privileges $ sudo yum install pkg # The right way to run yum and: $ pip install pkg # Error due to insufficient privileges $ sudo pip install pkg # The wrong way to run pip 'pip install --user pkg' or virtualenv are much better alternatives than sudo in the latter case, but the muscle memory induced by working with the system package manager means I still reach for sudo when I shouldn't. Having pip print out a Consider using virtualenv or the --user option *could* work (it would probably be enough of a prompt to disrupt my own distro package manager conditioned reflexes, for example), but it seems more sensible and user friendly to just start down the path of making --user the default, and requiring an explicit --system flag to install globally. Cheers, Nick. Marcus Inside a virtual environment: pip install pkg: works as now pip uninstall pkg: works as now Ordinary user (no write-access to system site packages): pip install pkg: installs to per-user site packages pip uninstall pkg: uninstalls from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: fails (likely with a permissions error) pip uninstall --system pkg: fails, even if the package is present (likely with a permissions error) Administrator/root (write-access to system site packages): pip install pkg: asks for confirmation before installing to per-user site packages pip uninstall pkg: asks for confirmation before uninstalling from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: install to system site packages pip uninstall --system pkg: uninstalls from site packages ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
On Sun, Feb 10, 2013 at 10:36 PM, Jannis Leidel jan...@leidel.info wrote: On 10.02.2013, at 05:44, Nick Coghlan ncogh...@gmail.com wrote: On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo ra...@develer.com wrote: Hello, my proposal for fixing PyPI and pip security is here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. I think the parts related to improving the HTTPS/SSL based security are solid, but for the other aspects of secure updates, integrating TUF (https://www.updateframework.com/) into the PyPI based distribution infrastructure sounds like the best available option for enhancing the end-to-end integrity checking. TUF has a comparatively well-developed threat model, and systematically covers many of the attack vectors discussed in the past few day (including provision of old, known vulnerable, versions). Would you mind explaining why TUF is good? The main benefit in my mind is that it isn't a from-scratch design of a secure update infrastructure. Instead, it's a project that was started in order to resolve some security holes found in Tor's already robust automatic update mechanism, then proceeded from there into updates to yum, yast, apt, etc (i.e. the distro update mechanisms that are vetted by the security teams of the various Linux vendors). The fact Geremy Condra is involved in TUF also counts for a lot with me (as I suspect it would for many people that have heard Geremy talk about security issues in Python). However, the design itself also seems sensible, and is able to provide its security guarantees even if you're *not* using SSL certs to protect the in-flight traffic (thus meaning that the SSL infrastructure in the near term will become a matter of providing defence-in-depth, rather than being a required part of the security scheme). I trust our collective ability to make TUF sufficiently easy to use as part of Python's packaging infrastructure a *lot* more than I trust our collective ability to come up with a from-scratch distribution scheme that is both usable *and* secure. The site doesn't seem to work for me right now. D'oh, looks like their domain wasn't set to auto-renew :( Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
On Sun, Feb 10, 2013 at 11:30 PM, Giovanni Bajo ra...@develer.com wrote: This is by far the biggest problem to be solved, and my document brings a proposal here. It would be great if the TUF guys reviewed it. Ensuring we fully address the problems that are addressed by TUF is more important than the question of whether or not we use the TUF software itself. However, the concern I have with your proposal is that I saw zero information regarding how it deals with attackers supplying old versions of software, or, indeed, any description of the threat model at all. The parts of your proposal that I believe need to be closely reviewed are: - GPG vs PKCS#1 - your custom trust model vs TUF target delegation - any threats that TUF covers and your proposal does not As far as the involvement TUF has had with other projects goes, I suspect this paper is at the heart of it: http://freehaven.net/~arma/tuf-ccs2010.pdf You may be right that those other projects addressed their issues by fixing the schemes they already had, rather than adopting TUF directly. We're in a somewhat different situation to those projects though, since we don't currently have an end-to-end integrity checking scheme at all. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] PyPI and setuptools
On Sun, Feb 10, 2013 at 9:37 AM, Stephen Thorne step...@thorne.id.au wrote: On Sat, Feb 9, 2013 at 11:28 PM, Jesse Noller jnol...@gmail.com wrote: On Feb 9, 2013, at 6:13 PM, Stephen Thorne step...@thorne.id.au wrote: Setuptools either needs to be sunset, notices put on pypi, warnings given to its users, out of linux distros, or it has to upgraded to be feature compatible with the security updates. That's a strong statement I've made, but I feel strongly that something has to be done. I would like to solicit opinions here before an action plan is composed. This is a bit of a question mark to me: the reality is that easy_install/setup tools usage is probably still dramatically higher than that of more modern tooling. One thing to keep in mind is that at least Fedora, and I believe other distros, actually ship distribute rather than vanilla setuptools. Migrating from insecure infrastructure is going to be a slow process, the immediate task is to make such a migration possible by: 1. Getting the server side in order 2. Offering at least one tool that better handles the security side of things That, and AFAIK, there are still features of them that the alternatives do not support (binary eggs, which are a must for windows). Yikes. This is something I didn't fully understand until now. Our windows users prefer to use setuptools and eggs? That make sense I guess. Many of the changes in PEP 426, and Daniel Holth's wheel PEPs arise directly from asking the question Why are some people still using setuptools rather than the alternatives?. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [DRAFT] Proposal for fixing PyPI/pip security
On Sun, Feb 10, 2013 at 7:23 AM, Giovanni Bajo ra...@develer.com wrote: Hello, my proposal for fixing PyPI and pip security is here: https://docs.google.com/a/develer.com/document/d/1DgQdDCZY5LiTY5mvfxVVE4MTWiaqIGccK3QCUI8np4k/edit# I tried to sum up the discussions we had here last week, elaborating on Heimes' proposal by simplifying it where I thought the additional steps wouldn't guarantee additional security. At this point, the proposal does not include a central, uber-master online GPG signing key to be stored on PyPI, which is IMO quite hard to handle correctly. I think the parts related to improving the HTTPS/SSL based security are solid, but for the other aspects of secure updates, integrating TUF (https://www.updateframework.com/) into the PyPI based distribution infrastructure sounds like the best available option for enhancing the end-to-end integrity checking. TUF has a comparatively well-developed threat model, and systematically covers many of the attack vectors discussed in the past few day (including provision of old, known vulnerable, versions). I have more faith in our collective ability to build a usable *and* secure cross-platform distribution infrastructure on TUF (which already has many of the more difficult security aspects covered), along with devising a migration path from our existing distribution infrastructure, than I do in our ability to come up with something completely new. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Draft] Package signing and verification process
On 8 Feb 2013 02:43, Giovanni Bajo ra...@develer.com wrote: Il giorno 07/feb/2013, alle ore 17:21, Donald Stufft donald.stu...@gmail.com ha scritto: On Thursday, February 7, 2013 at 10:50 AM, Giovanni Bajo wrote: 1. If we're going to implicitly trust PyPI when it says that key X is valid for package Y, do we really gain much here? If we're trusting PyPI then we only really need secure ingress and egress neither of which need packaging signing. Adding GPG signature on top of SSL helps mitigating (at least) the following concerns: 1) If a PyPI account password is compromised (stolen, bruteforced, etc.), an attacker cannot upload a package that will be installed by package managers. This also requires making sure that a GPG fingerprint cannot be added to the account without a second factor authentication (can be anything from a link to a security email address, to a SMS). Notice that PyPI passwords are currently saved in the filesystem in clear ($HOME/.pypirc). Which reminds me, that system *really* should be replaced/supplemented with a time limited server generated auth token, the way Bugzilla and various other services do it. If need be, I can bug a couple of GPL RH projects to contribute their existing solutions to that problem, but there should be non-GPL examples kicking around the web already. Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
[Catalog-sig] Packaging Distribution Mini-Summit at PyCon US
As folks may be aware, I am moderating a panel called Directions in Packaging on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a Packaging Distribution Mini-Summit as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ (that page should be editable by anyone that has registered for PyCon US). We're certainly not going to resolve all our disagreements and come up with a grand plan to fix Python packaging in a couple of hours on a Friday night. My hopes are far more modest: that we can get an idea of the different things people are worried about and trying to resolve, and start pondering ways we can work towards an cooperative ecosystem of interoperable tools, rather than the one tool to rule them all model of distutils. I fully expect that discussions on the Friday night will continue as hallway track discussions during the conference, development efforts at the sprints, and, of course, requests for feedback on the mailing lists (since there will likely be quite a few interested people that won't be present at PyCon US). This is not a new conversation - it is one that has been going on for years, and while some improvements have been made, we still have a long way to go. For those that are able to make it, I look forward to meeting you in person in March :) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Fwd: readthedocs.org or packages.python.org?
On Thu, Feb 7, 2013 at 10:43 AM, Jacob Kaplan-Moss ja...@jacobian.org wrote: OK I just bought both. If the PSF wants them, I'll transfer them. Otherwise I'll sit on 'em for a year and then let 'em expire. Heh, I just did the same thing for pythonhosted.org. (borrowing the name from Fedora, where the main site is all under fedoraproject.org, while fedorahosted.org is their project hosting service. We're only talking about docs hosting, rather than full project hosting, but it's the same general idea). Personally, what I would love to see happen is: Near term: packages.python.org becomes a CNAME to another top-level site (obviously, my suggestion is pythonhosted.org, since I just bought that for a year in order to be to mention it without worrying about whether or not it would remain available) Slightly longer term: the pythonhosted.org (or whatever) naming scheme includes subdomains (e.g. six.pythonhosted.org, in addition to pythonhosted.org/six) Even longer term: PyPI offers the option to set up a project's pythonhosted subdomain as a ReadTheDocs reference (using the existing subdomain delegation feature of RTFD) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Fwd: readthedocs.org or packages.python.org?
On Thu, Feb 7, 2013 at 12:29 PM, Jesse Noller jnol...@gmail.com wrote: Yes, thanks Nick (and everyone involved in this discussion!) I can chat to Nick offline about how to get this going - I have the ability to set up services to support it. Unless we need to transfer the domain to the PSF first... I'll probably transfer it later (it looks like my registrar/ICANN/some-rule-somewhere requires that I hang on to it for a couple of months before transferring it). In the meantime, it's probably easiest if Richard, Noah and I have an offline discussion to get the mechanics of the delegation worked out. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Fwd: readthedocs.org or packages.python.org?
On Thu, Feb 7, 2013 at 12:35 PM, Nick Coghlan ncogh...@gmail.com wrote: In the meantime, it's probably easiest if Richard, Noah and I have an offline discussion to get the mechanics of the delegation worked out. As a quick update - DNS authority for pythonhosted.org has now been delegated to the same name servers as are used for python.org itself, and Richard and Noah are in the process of working out the details of splitting out a separate virtual host for the user uploaded data. The published whois data for pythonhosted.org is still wrong, but we'll figure that out over the next few days. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Tue, Feb 5, 2013 at 7:57 PM, Giovanni Bajo ra...@develer.com wrote: One meta-question: does this mailing-list have any authority over pip? Are there any pip maintainers here? Because I see that pip development being done on different channels, so I was wondering what is the workflow to discuss such modifications. It's a handy place to get feedback before I post a suggestion to the pip issue tracker, plus catalog-sig has a nice audience of pip *users* as well as developers. As MAL rightly pointed out, the (when running as anyone other than root) part of my suggestion is seriously flawed, and I wasn't clear that I didn't want to alter pip's default behaviour when run inside a virtualenv. So, to clarify, the behaviour I would *like* to see pip exhibiting is for the default install location to *change*, rather than trying to install to the system packages directory and then implicitly falling back to the user directory if that fails. Instead, installing to the system site-packages would require an explicit --system flag. Desired final behaviour: Inside a virtual environment: pip install pkg: works as now pip uninstall pkg: works as now Ordinary user (no write-access to system site packages): pip install pkg: installs to per-user site packages pip uninstall pkg: uninstalls from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: fails (likely with a permissions error) pip uninstall --system pkg: fails, even if the package is present (likely with a permissions error) Administrator/root (write-access to system site packages): pip install pkg: asks for confirmation before installing to per-user site packages pip uninstall pkg: asks for confirmation before uninstalling from per-user site packages pip install --user pkg: installs to per-user site packages pip uninstall --user pkg: uninstalls from per-user site packages pip install --system pkg: install to system site packages pip uninstall --system pkg: uninstalls from site packages Confirmation message: Warning: the current user has write access to the system site-packages directory, but '--system' was not specified. Proceed with installation to/uninstallation from the user package directory at 'path/to/user/dir'? (y/n) Transition: For ordinary users, the transitional release would print out a warning before proceeding with the installation to the per-user site packages For admin users, the transitional release would print out a warning to start passing --system, as the behaviour of *not* passing that flag is going to change in the next release Consequences: - the harmful Cannot write to blah - Hit it with the sudo hammer behaviour is eliminated - user packages are hidden from scripts executed as root, even if the execution of that script neglected the -SE flags - users may encounter the situation where a server process (e.g. mod_wsgi in a local Apache instance) won't be able to see packages in their user directory. This provides an opportunity to nudge them towards virtualenv I see this as very similar to the install for everyone, or just for me model used by modern Windows installers, and the default should be just for me, with install for everyone needing to be explicitly requested. It is by no means a comprehensive security solution, but neither is it meant to be (that's what SELinux is for). It is merely an early line of defence that aims to avoid getting users into the habit of running pip with elevated privileges. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Tue, Feb 5, 2013 at 11:55 PM, Jeroen Dekkers jer...@dekkers.ch wrote: At Tue, 5 Feb 2013 11:36:46 +1000, Nick Coghlan wrote: Something that caught my attention in the recent security discussions is the observation that one of the most common insecure practices in the Python community is to run sudo pip with unsigned packages (sometimes on untrusted networks). To my mind, this is a natural reaction to the user experience of pip: you run pip install package, it complains it can't write to the system site packages directory, so you run sudo pip install package to give it the permissions it clearly wants. If pip used the user site packages by default (when running as anyone other than root), that dangerous UI flow wouldn't happen. Even when pip was run outside a virtualenv, it would just work from the users perspective. It also has the advantage of keeping systems cleaner by default, since there will be a clear separation between system packages and pip-installed packages. Thoughts? How this is going to improve anything with regards to security? There might be other good reasons for changing it, but I don't see the security benefit when installing untrusted packages. It limits the primary threat to data destruction and one shot retrieval of local data (with one obvious choice on *nix systems being to upload the contents of ~/.ssh). The default PATH on Linux doesn't include any user writable directions before the system directories, so you can't use those to shadow system utilities like sudo, and not running as root means more pernicious attacks (like setting up persistent keylogging, or zombie systems) won't work. A useful security barrier is one which prevents some threats, without opening up the overall system to additional dangers. pip and similar tools confining themselves to user writable directories by default, and always requiring an explicit flag in order to touch system directories with intent to modify them, would be one such barrier. (There is no additional complexity introduced, as the --user option means the capability is already present - it is merely a matter of switching the default behaviour). Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Wed, Feb 6, 2013 at 12:46 AM, Giovanni Bajo ra...@develer.com wrote: Il giorno 05/feb/2013, alle ore 15:06, Holger Krekel holger.kre...@gmail.com ha scritto: In the end, however, none of this prevents MITM attacks between a downloader and pypi.python.org. Or between the uploader and pypi.python.org (using basic auth over http often). Signing methods like https://wiki.archlinux.org/index.php/Pacman-key are key. If a signature is available (also at a download_url site), then we can exclude undetected tampering. And there might not be a need to break currently working package releases. A signature is not enough; if you don't have a secure channel, signatures can be replayed. Eg: if you install through an unsecure channel and you just verify GPG signatures on the package, I can MITM you and serve you an older, vulnerable package version (with its correct signature), and then go exploit that vulnerability. Don't let perfect become the enemy of better. There are a *truckload* of potential vulnerabilities in the way people currently use PyPI, and *all* of them need to be addressed over time. It's great that the problems with rubygems and the published MITM attack on PyPI have drawn attention to these issues, but it's important to remember that the reason most of them haven't been addressed before now is because they're *hard problems*, and because there's a tension between encouraging relative newcomers to Python (and open source in general) to share their work with the world, providing reasonable transition plans from existing insecure practices to more secure allternatives, and ultimately satisfying the dependency management needs of those that want to be able to obtain trusted versions of software directly from PyPI using the standard tools. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Wed, Feb 6, 2013 at 12:54 AM, Jim Fulton j...@zope.com wrote: pip will need to learn to prefer non-final releases. I was pressed to put buildout alpha and beta releases on a separate site because of the concern that they'd be installed inadvertently by pip. FWIW, PEP 426 aims to address this by expressly requiring that all pre-releases (dev, alpha, beta, release candidate) be excluded from version specifiers by default, unless a pre-release is explicitly mentioned as part of the specifier. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Wed, Feb 6, 2013 at 1:06 AM, Donald Stufft donald.stu...@gmail.com wrote: On Tuesday, February 5, 2013 at 9:53 AM, holger krekel wrote: Point taken. I guess unless someone sits down and writes a PEP-ish path for fortification, it's gonna be hard to assess viability and resilience against the several attack vectors which should be sorted/prioritized. Or is somebody on that already? (there were hints of some background discussions - not sure that's helping much as most attack vectors against the python packaging ecosystem are kind of well known or easy to guess after a bit of research and experimentation). There are easy wins to take care of before we go this route. It's a *hard* problem that on the surface appears easy. I've personally got some ideas and I'm sure others do as well, but focusing on the hard problems when there are several low hanging fruit is a red herring IMO. The background discussions Holger mentioned earlier are actually aimed at picking some of those low hanging front (a lot of it related to the general provision of the PSF infrastructure at OSU/OSL and making it easier to improve PyPI's handling of HTTPS). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Use user-specific site-packages by default?
On Tue, Feb 5, 2013 at 3:20 PM, Yuval Greenfield ubershme...@gmail.com wrote: Excellent idea. I've been using sudo pip install since forever for the exact reason you mention. I don't even know how to install anything with pip and no sudo. If you're not inside a virtualenv, then pip install --user whatever will install whatever to your user packages directory rather than the system site packages. I'm not sure what it does inside a virtualenv, as I've never tried it. The behaviour of pip uninstall might need a little thought, though, if pip install changes to automatically user the user site directory when it can't write to the system one. At the moment, I think it's the same as install - it complains it can't write to the system directory, even if the package is present in the user directory. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Python-Dev] accept the wheel PEPs 425, 426, 427
On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth dho...@gmail.com wrote: I think Metadata 1.3 is done. Who would like to czar? (Apologies for the belated reply, it's been a busy few weeks) I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated with some additional rationale based on Ronald's comments later in this thread, though. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] [Python-Dev] egg_info in PyPI
On Sat, Sep 18, 2010 at 10:24 PM, Michael Foord fuzzy...@voidspace.org.uk wrote: On 18/09/2010 12:25, Martin v. Löwis wrote: I think you are misunderstanding. The infrastructure will *not* depend on the old formats. Instead, packaging that have that information will provide it, packages that don't will not. The infrastructure is entirely agnostic on whether the data is available or not. In particular, it will not try to interpret the data in any way. No, not at all. If tools *use* the information (and if they don't then what is the point?) then packages that want to be compatible with those tools will have to provide this information. Not true. Tools could/should also support PEP 345 data, and then they can support either kind of package. But you are proposing that PyPI will provide an interface / API for exposing the setuptools information. So tools that get metadata from PyPI in this way (and depend on it - as they will if you expose it) will have functionality that only works for packages that provide this information in this form. Saying tools should work with other formats too is empty words. If the idea is solely to use legacy setuptools data as a fallback if the actual PEP 345 data is unavailable, it sounds like it would be far more robust to have *PyPI* do the fallback, implementing it correctly *once*, rather than simply exposing the raw setuptools data (which *will* lead to client applications that *only* understand the setuptools data and can't understand packages that only provide PEP 345 compliant metadata). Throwing the raw data at client applications and saying here, you figure it out when they can already do that by downloading and interrogating the packages directly seems likely to cause more confusion than anything else. So +1 to a allow_fallback_metadata flag in appropriate PyPI APIs, -1 on exposing the legacy data directly. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
