Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
On Wed, Mar 13, 2013 at 1:23 AM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 07:28, Nick Coghlan wrote: On Tue, Mar 12, 2013 at 12:59 PM, M.-A. Lemburg m...@egenix.com wrote: I think we should establish a versioned API like that for PyPI to make progress easier. All major web APIs use versioning for this reason. Why set up versioning for something we want to phase out? There will never be a simple-v3, so this is really overengineering the proposed change. Who says that we want to phase out the /simple/ index ? I want to render it redundant, because it's a crazy way to distribute completely inadequate metadata. Cheers, Nick. FWIW, I don't think that two or three small changes to the PyPI (see my email to Holger) server warrants calling this over-engineering. This is about moving forward in a backwards compatible and future proof way. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 13 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI
On Wed, Mar 13, 2013 at 11:19 PM, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Mar 13, 2013 at 1:23 AM, M.-A. Lemburg m...@egenix.com wrote: On 13.03.2013 07:28, Nick Coghlan wrote: On Tue, Mar 12, 2013 at 12:59 PM, M.-A. Lemburg m...@egenix.com wrote: I think we should establish a versioned API like that for PyPI to make progress easier. All major web APIs use versioning for this reason. Why set up versioning for something we want to phase out? There will never be a simple-v3, so this is really overengineering the proposed change. Who says that we want to phase out the /simple/ index ? I want to render it redundant, because it's a crazy way to distribute completely inadequate metadata. Specifically, once we have the infrastructure in place to publish metadata v2.0 (or a suitable subset) to installation tools, the relatively impoverished contents of the simple index will be a legacy interface retained only to preserve the correct operation of existing tools. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 5:16 PM, Carl Meyer c...@oddbird.net wrote: There is no instead of. There are parallel proposals (see the TUF thread) to improve the security of the ecosystem, and those proposals are not mutually exclusive with this one. If you search the PEP text, note that you don't find the words secure or security anywhere within it, or any claims of security achieved by this proposal alone. There is a brief mention of MITM attacks, which is relevant to the PEP because avoiding external link-crawling does reduce that attack surface, even if other proposals will also help with that (even more). Right, the changes to provide end-to-end security require more extensive changes and need to be given appropriate consideration before we proceed to implementation and deployment. This PEP, especially with the additional changes you propose here is an excellent approach to *near term* improvement, as a parallel effort to the more complex proposals. The /simple/ index will also be around for a long time for backwards compatibility reasons, regardless of any other changes that happen in the overall distribution ecosystem. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
On 3/14/13 3:03 AM, Nick Coghlan wrote: I think what you currently propose (signing the metadata pip already understands) is a good first step, especially if we can have PyPI signing *all* the target metadata in the initial deployment, and defer the delegation to package developers until the next phase of the rollout (we obviously want to do that eventually, but it's easier if we can get a preliminary version working without needing to change the upload tools). While such an approach doesn't immediately give us the end-to-end security we ultimately want to set up, it means a few things become possible: 1. Rather than requiring every developer to start signing end-to-end metadata immediately, we can ask a few major projects (e.g. Django, Zope, NumPy) if they're willing to serve as guinea pigs for the developer target signing delegations. Once we're happy the signing process is usable, we can make it generally available as an option to projects (while also allowing them to continue with PyPI's existing upload mechanisms and only offer PyPI-user integrity checks rather than developer-user) 2. Gives the PSF infrastructure team and the PyPI maintainers a chance to work with the installation tool developers to get the PyPI-user link sorted out, before needing to work on the developer-PyPI link 3. Considering alternate mirroring solutions based on replicating the TUF metadata rather than PEP 381 Eventually I would also like to tunnel a subset of the PEP 426 metadata through TUF's custom fields, but again, I think we're better off skipping that for the first iteration. Incremental enhancements are a good thing :) This sounds good to me --- I like the idea of incremental enhancements. Justin, what are your thoughts from a security perspective? ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Wed, Mar 13, 2013 at 23:43 -0700, Nick Coghlan wrote: On Wed, Mar 13, 2013 at 5:16 PM, Carl Meyer c...@oddbird.net wrote: There is no instead of. There are parallel proposals (see the TUF thread) to improve the security of the ecosystem, and those proposals are not mutually exclusive with this one. If you search the PEP text, note that you don't find the words secure or security anywhere within it, or any claims of security achieved by this proposal alone. There is a brief mention of MITM attacks, which is relevant to the PEP because avoiding external link-crawling does reduce that attack surface, even if other proposals will also help with that (even more). Right, the changes to provide end-to-end security require more extensive changes and need to be given appropriate consideration before we proceed to implementation and deployment. This PEP, especially with the additional changes you propose here is an excellent approach to *near term* improvement, as a parallel effort to the more complex proposals. The /simple/ index will also be around for a long time for backwards compatibility reasons, regardless of any other changes that happen in the overall distribution ecosystem. I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. best, holger Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm
On 12.03.2013 22:26, PJ Eby wrote: On Tue, Mar 12, 2013 at 3:59 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 19:15, M.-A. Lemburg wrote: I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip into the same directory and let easy_install running on Linux scan this, it considers the second file for Windows as best match. Is the algorithm used for determining the best match documented somewhere ? I've had a look at the implementation, but this left me rather clueless. I thought that setuptools would prefer the .egg file over the prebuilt .zip file - binary files being easier to install than source files. After some experiments, I found that the follow change in filename (swapping platform and python version, in addition to use '-' instead of '.) works: egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip OTOH, this one doesn't (notice the difference ?): egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip The logic behind all this looks rather fragile to me. easy_install only guarantees sane version parsing for distribution files built using setuptools' naming algorithms. If you use distutils, it can only make guesses, because the distutils does not have a completely unambiguous file naming scheme. And if you are naming the files by hand, God help you. ;-) The problem appears to be a bug in setuptools' package_index.py. The function interpret_distro_name() creates a set of possible separations of the found name into project name and version. It does find the right separation, but for some reason, the code using that function does not check the found project names against the project name the user is trying to install, but simply takes the last entry of the list returned by the above function. As a result, easy_install downloads and tries to install project files that don't match the project name in some cases. Here's another example where it fails (say you're on a x64 Linux box): # easy_install egenix-pyopenssl As example, say it finds these distribution files: 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-linux-x86_64-prebuilt.zip', 'egenix_pyopenssl-0.13.1.1.0.1.5-py2.7-linux-x86_64.egg', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-macosx-10.5-x86_64-prebuilt.zip', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs4-macosx-10.5-x86_64-prebuilt.zip', It then creates different interpretations of those names, puts them in a list and sorts them. Here's the end of that list: egenix-pyopenssl; 0.13.1.1.0.1.5 -- this would be the correct .egg file egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-linux-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-macosx-10.5-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs2-macosx; 10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx; 10.5-x86-64-prebuilt It picks the last entry, which would be for a project called egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx - not the one the user searched. I'm trying to find a way to get it to use the correct .egg file The .egg files does have precedence over the other files, since easy_install regards them as source files with lower precedence. This is important, because the /simple/ index page will have links not only to .egg files, but also to our prebuilt .zip files, which use a source file compatible setup.py interface. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 14 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Packaging Distribution Mini-Summit at PyCon US
On Thu, Feb 7, 2013 at 10:19 AM, Jim Fulton j...@zope.com wrote: On Wed, Feb 6, 2013 at 3:15 AM, Nick Coghlan ncogh...@gmail.com wrote: As folks may be aware, I am moderating a panel called Directions in Packaging on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a Packaging Distribution Mini-Summit as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). I wasn't going to be at PyCon, but I changed my plans specifically to participate in this. Thanks for setting this up. An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ (that page should be editable by anyone that has registered for PyCon US). Cool. A major difficulty in these sorts of discussions is that people have different problems they want to solve and argue about solutions without clearly stating their problems. If you don't mind, I'll try to find some time in the next few days to add a section to that page to list goals/problems. OK, well, hopefully better late than never. I took a stab at adding this to the end of: https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/ Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) __**_ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/**mailman/listinfo/catalog-sighttp://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files
On Thu, Mar 14, 2013 at 7:13 AM, Justin Cappos jcap...@poly.edu wrote: Maybe a different way to say it is that the current TUF integration doc assumes that it is desirable to make minimal change to PyPI's layout and pip, easy_install, etc. while adding security. We made several choices based upon this assumption, including using and retaining the /simple dir. I think what you're proposing now is a pretty good place to state (although I'm suggesting making it even simpler in the near term by starting by focusing on the PyPI-end user link, and then moving to delegating signing of the per-project metadata to the individual projects as a later step) If the community wants a more 'clean-slate' design, we could put that together also. This requires a lot of information specific to your setup and use cases so we'd appreciate collaboration with you guys to write that up. I'd like to do a distribution 2.0 at some point where we make the simple index redundant by including that info (and more) directly in the TUF metadata, but I think that's a later project - securing what we have now is a better place to start. Cheers, Nick. Thanks, Justin On Thu, Mar 14, 2013 at 8:14 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 4:58 AM, holger krekel wrote: I haven't followed the latest TUF discussions and related docs in depths yet but if those developments will regard simple/ as a deprecated interface, i think this PEP here should maybe not introduce simple/-with-externals as it will just make the situation more complicated for everyone to understand in a few months from now. I haven't yet followed your PEP in as much depth as I would like, but I wish to assure you that we do not regard /simple/ as a deprecated interface. In fact, we aim to preserve backwards-compatibility as much as possible! :) ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] Publishing metadata (was: V2 pre-PEP: transitioning to release file hosting on PYPI)
On Thu, Mar 14, 2013 at 12:54 AM, M.-A. Lemburg m...@egenix.com wrote: The index itself is just a bag of things and, as such, one that's very well suited to publish data, since it can easily be exposed in form of static files, which can be put on a CDNs or mirrored using rsync. The TUF metadata is also just a collection of static files which can be put on CDNs and mirrored using rsync. That's one of the reasons TUF is an interesting approach :) It's easy to add the metadata file to that index for tools to pick up - in addition to the other data exposed on the index pages and perfectly backwards compatible. As mentioned before, I think we should start publishing the existing metadata stored in the PyPI database on those index pages as PKG-INFO files, so that tools can easily access the data without having to go through XML-RPC. Yes, I think that's a good near term approach. However, there's still a lot of duplication of functionality between the TUF metadata and the simple index, so if we get TUF-based security up and running, my long term aim will be to make it so that once you have downloaded the TUF metadata, you shouldn't *need* anything from the simple index, and would be able to go directly to downloading the release files. That's a longer term idea, though and we may even decide it isn't worth the hassle if PKG-INFO is made available through /simple. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] A modest proposal for securing PyPI with TUF
Yes, Nick's suggestions are good ones. I'd agree that getting an initial deployment together that doesn't include things like custom metadata is probably for the best. We can certainly add things incrementally. Thanks, Justin On Thu, Mar 14, 2013 at 3:21 AM, Trishank Karthik Kuppusamy t...@students.poly.edu wrote: On 3/14/13 3:03 AM, Nick Coghlan wrote: I think what you currently propose (signing the metadata pip already understands) is a good first step, especially if we can have PyPI signing *all* the target metadata in the initial deployment, and defer the delegation to package developers until the next phase of the rollout (we obviously want to do that eventually, but it's easier if we can get a preliminary version working without needing to change the upload tools). While such an approach doesn't immediately give us the end-to-end security we ultimately want to set up, it means a few things become possible: 1. Rather than requiring every developer to start signing end-to-end metadata immediately, we can ask a few major projects (e.g. Django, Zope, NumPy) if they're willing to serve as guinea pigs for the developer target signing delegations. Once we're happy the signing process is usable, we can make it generally available as an option to projects (while also allowing them to continue with PyPI's existing upload mechanisms and only offer PyPI-user integrity checks rather than developer-user) 2. Gives the PSF infrastructure team and the PyPI maintainers a chance to work with the installation tool developers to get the PyPI-user link sorted out, before needing to work on the developer-PyPI link 3. Considering alternate mirroring solutions based on replicating the TUF metadata rather than PEP 381 Eventually I would also like to tunnel a subset of the PEP 426 metadata through TUF's custom fields, but again, I think we're better off skipping that for the first iteration. Incremental enhancements are a good thing :) This sounds good to me --- I like the idea of incremental enhancements. Justin, what are your thoughts from a security perspective? ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm
On Thu, Mar 14, 2013 at 6:07 AM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 22:26, PJ Eby wrote: On Tue, Mar 12, 2013 at 3:59 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 19:15, M.-A. Lemburg wrote: I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip into the same directory and let easy_install running on Linux scan this, it considers the second file for Windows as best match. Is the algorithm used for determining the best match documented somewhere ? I've had a look at the implementation, but this left me rather clueless. I thought that setuptools would prefer the .egg file over the prebuilt .zip file - binary files being easier to install than source files. After some experiments, I found that the follow change in filename (swapping platform and python version, in addition to use '-' instead of '.) works: egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip OTOH, this one doesn't (notice the difference ?): egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip The logic behind all this looks rather fragile to me. easy_install only guarantees sane version parsing for distribution files built using setuptools' naming algorithms. If you use distutils, it can only make guesses, because the distutils does not have a completely unambiguous file naming scheme. And if you are naming the files by hand, God help you. ;-) The problem appears to be a bug in setuptools' package_index.py. The function interpret_distro_name() creates a set of possible separations of the found name into project name and version. It does find the right separation, but for some reason, the code using that function does not check the found project names against the project name the user is trying to install, but simply takes the last entry of the list returned by the above function. As a result, easy_install downloads and tries to install project files that don't match the project name in some cases. Here's another example where it fails (say you're on a x64 Linux box): # easy_install egenix-pyopenssl As example, say it finds these distribution files: 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-linux-x86_64-prebuilt.zip', 'egenix_pyopenssl-0.13.1.1.0.1.5-py2.7-linux-x86_64.egg', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-macosx-10.5-x86_64-prebuilt.zip', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs4-macosx-10.5-x86_64-prebuilt.zip', It then creates different interpretations of those names, puts them in a list and sorts them. Here's the end of that list: egenix-pyopenssl; 0.13.1.1.0.1.5 -- this would be the correct .egg file egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-linux-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-macosx-10.5-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs2-macosx; 10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx; 10.5-x86-64-prebuilt It picks the last entry, which would be for a project called egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx - not the one the user searched. Actually, that's not quite true. It's picking: egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt Because it thinks that '0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt' is a higher version than 0.13.1.1.0.1.5. It does also record the possibility you mentioned, but it doesn't pick that one. The project names actually *do* have to match. If you open a ticket on the setuptools tracker, 'll try to see if I can get it to recognize that strings like py2.7, macosx, ucs, and the like are terminators for a version number. I don't know how successful I'll be, though. Basically, those zip files are (I assume) bdist_dumb distributions being taken for source distributions, and easy_install doesn't actually support bdist_dumb files at the moment. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm
On 14.03.2013 17:39, PJ Eby wrote: On Thu, Mar 14, 2013 at 6:07 AM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 22:26, PJ Eby wrote: On Tue, Mar 12, 2013 at 3:59 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 19:15, M.-A. Lemburg wrote: I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip into the same directory and let easy_install running on Linux scan this, it considers the second file for Windows as best match. Is the algorithm used for determining the best match documented somewhere ? I've had a look at the implementation, but this left me rather clueless. I thought that setuptools would prefer the .egg file over the prebuilt .zip file - binary files being easier to install than source files. After some experiments, I found that the follow change in filename (swapping platform and python version, in addition to use '-' instead of '.) works: egenix-mxodbc-connect-client-2.0.2-py2.6-win32.prebuilt.zip OTOH, this one doesn't (notice the difference ?): egenix-mxodbc-connect-client-2.0.2.py2.6-win32.prebuilt.zip The logic behind all this looks rather fragile to me. easy_install only guarantees sane version parsing for distribution files built using setuptools' naming algorithms. If you use distutils, it can only make guesses, because the distutils does not have a completely unambiguous file naming scheme. And if you are naming the files by hand, God help you. ;-) The problem appears to be a bug in setuptools' package_index.py. The function interpret_distro_name() creates a set of possible separations of the found name into project name and version. It does find the right separation, but for some reason, the code using that function does not check the found project names against the project name the user is trying to install, but simply takes the last entry of the list returned by the above function. As a result, easy_install downloads and tries to install project files that don't match the project name in some cases. Here's another example where it fails (say you're on a x64 Linux box): # easy_install egenix-pyopenssl As example, say it finds these distribution files: 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-linux-x86_64-prebuilt.zip', 'egenix_pyopenssl-0.13.1.1.0.1.5-py2.7-linux-x86_64.egg', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs2-macosx-10.5-x86_64-prebuilt.zip', 'egenix-pyopenssl-0.13.1.1.0.1.5-py2.7_ucs4-macosx-10.5-x86_64-prebuilt.zip', It then creates different interpretations of those names, puts them in a list and sorts them. Here's the end of that list: egenix-pyopenssl; 0.13.1.1.0.1.5 -- this would be the correct .egg file egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-linux-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs2-macosx-10.5-x86-64-prebuilt egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs2-macosx; 10.5-x86-64-prebuilt egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx; 10.5-x86-64-prebuilt It picks the last entry, which would be for a project called egenix-pyopenssl-0.13.1.1.0.1.5-py2.7-ucs4-macosx - not the one the user searched. Actually, that's not quite true. It's picking: egenix-pyopenssl; 0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt Because it thinks that '0.13.1.1.0.1.5-py2.7-ucs4-macosx-10.5-x86-64-prebuilt' is a higher version than 0.13.1.1.0.1.5. It does also record the possibility you mentioned, but it doesn't pick that one. The project names actually *do* have to match. Ah, ok, that makes sense then. Is there any way to have 0.13.1.1.0.1.5-something sort before 0.13.1.1.0.1.5 ? (e.g. like is done for release candidates) Ideally, I'd like to get this to work without any changes to setuptools, even though it would of course be better not to take stuff after a Python version marker into account when looking for a package version (since the Python marker is actually a new component in the file name). If you open a ticket on the setuptools tracker, 'll try to see if I can get it to recognize that strings like py2.7, macosx, ucs, and the like are terminators for a version number. I don't know how successful I'll be, though. Basically, those zip files are (I assume) bdist_dumb distributions being taken for source distributions, and easy_install doesn't actually support bdist_dumb files at the moment. If you could point me to that tracker, I'll open a ticket :-) Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 14 2013) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm
On Thu, Mar 14, 2013 at 2:11 PM, M.-A. Lemburg m...@egenix.com wrote: Is there any way to have 0.13.1.1.0.1.5-something sort before 0.13.1.1.0.1.5 ? (e.g. like is done for release candidates) Make it 0.13.1.1.0.1.5-devsomething, and it'll have lower precedence than both 0.13.1.1.0.1.5 and 0.13.1.1.0.1.5-something. If you could point me to that tracker, I'll open a ticket :-) http://bugs.python.org/setuptools/ ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig
