Re: [Distutils] How to make easy_install handle platlibs?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 David Cournapeau wrote: Buck wrote: I have no clue what you mean by 'sdists'. Is this a widely-known thing? sdist is the name of the distutils command - it just means distributing a source tarball. Nope, by sdist I mean the specific tarball generated by the 'sdist' command (PJE spelled out what is special about them). Manually-created source tarballs don't have the layout / metadata which makes automating their installation feasible. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJ5h3w+gerLs4ltQ4RAjMUAJ0bG4aCjLVeC7J2czCbrcMQshxg1QCfexDG u0/T9gwas+IJeqN+4GsFcsM= =l29V -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
On Apr 12, 12:46 pm, P.J. Eby p...@telecommunity.com wrote: On Sun, Apr 12, 2009 at 8:54 AM, Andrew Straw mailto:straw...@astraw.comstraw...@astraw.com wrote: On the other hand, sticking the egg into the place that distutils uses when not under easy_install would fix this much more simply, although from what I hear this would be a big change. For easy_install, yes. For pip, probably not so much. In fact, my guess would be that pip already supports this, as long as you can install from source, rather than from eggs. Are you saying I should dump easy_install in favor of pip? I hadn't heard of it till now. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
On Apr 12, 8:51 am, Tres Seaver tsea...@palladion.com wrote: zooko wrote: It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X. However, it (egg naming scheme on Linux) currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. ... Better: just don't distribute binary eggs for Linux at all: - Developers have the tools (or can install them) to build from 'sdists'. - Systems without tools are locked down, which means that they shouldn't be installing from public distributions anyway: the folks who run them should be able to build *private* eggs from 'sdists' which are known to work for their systems. Tres. I have no clue what you mean by 'sdists'. Is this a widely-known thing? A URL to an example would be sufficient. Thanks, --Buck ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
At 11:11 AM 4/13/2009 -0700, Buck wrote: On Apr 12, 12:46 pm, P.J. Eby p...@telecommunity.com wrote: On Sun, Apr 12, 2009 at 8:54 AM, Andrew Straw mailto:straw...@astraw.comstraw...@astraw.com wrote: On the other hand, sticking the egg into the place that distutils uses when not under easy_install would fix this much more simply, although from what I hear this would be a big change. For easy_install, yes. For pip, probably not so much. In fact, my guess would be that pip already supports this, as long as you can install from source, rather than from eggs. Are you saying I should dump easy_install in favor of pip? I hadn't heard of it till now. If it meets your other use cases, I would say yes. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
At 11:16 AM 4/13/2009 -0700, Buck wrote: On Apr 12, 8:51 am, Tres Seaver tsea...@palladion.com wrote: zooko wrote: It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X. However, it (egg naming scheme on Linux) currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. ... Better: just don't distribute binary eggs for Linux at all: - Developers have the tools (or can install them) to build from 'sdists'. - Systems without tools are locked down, which means that they shouldn't be installing from public distributions anyway: the folks who run them should be able to build *private* eggs from 'sdists' which are known to work for their systems. Tres. I have no clue what you mean by 'sdists'. Is this a widely-known thing? A URL to an example would be sufficient. An sdist is a source distribution, usually in .tgz or .zip form. Sdists contain certain well-defined directory layouts and files. In particular: * the entire contents are in a subdirectory named for the project and version * that subdirectory contains a setup.py and a PKG-INFO easy_install and pip can both find and retrieve sdists from PyPI and install them, they just go about the actual installation process a bit differently. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
Buck wrote: I have no clue what you mean by 'sdists'. Is this a widely-known thing? sdist is the name of the distutils command - it just means distributing a source tarball. cheers, David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
David Cournapeau da...@ar.media.kyoto-u.ac.jp writes: Buck wrote: I have no clue what you mean by 'sdists'. Is this a widely-known thing? sdist is the name of the distutils command - it just means distributing a source tarball. Again, please stop blurring the distinction. I can only assume at this point that it's deliberate on your part, but I have no idea why. A distutils ‘sdist’ is a combination of the source files *and* metadata for distutils. A mere source tarball is *not* an sdist. -- \ “I know you believe you understood what you think I said, but I | `\ am not sure you realize that what you heard is not what I | _o__) meant.” —Robert J. McCloskey | Ben Finney ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
On Fri, Apr 10, 2009 at 9:15 PM, P.J. Eby p...@telecommunity.com wrote: At 08:53 PM 4/10/2009 -0700, Buck wrote: I see the kernel version and architecture, but this is insufficient; RedHat 4 and RedHat 5 both use a 2.6 kernel, but the difference in provided libraries are sufficient to make many (most?) impure libraries stop working ( numpy, python-ldap, and hashlib for examples). easy_install apparently knows when packages are platform-dependant, and the necessary directory is sys.exec_prefix (or maybe better: distutils.sysconfig.get_python_lib(plat_specific=1) ), so this seems like an easy change. Would a patch be accepted? It would be a pretty major patch, since the distutils logic that's used for determining the installation directory is applied long before easy_install even knows what it's installing (and therefore, whether it's platform dependent). It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X. If the distutils logic is used, then why does numpy install to the platlib folder using distutils, but into the nonplatlib folder using easy_install? Your tone seems to suggest that such an effort might be accepted upstream. --Buck ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X. As PJE has mentioned, the intent is that the egg name should contain enough information to decide if that egg will work on your platform. For example, if it says py2.5-win32 then you know it will work on your Python 2.5 on 32-bit Windows, and if it says py2.5-macosx-10.3- ppc then you know it will work on your PowerPC-based Mac with Python 2.5. This can be used for example by easy_install to get a directory listing of eggs and choose which one will work on the local platform based solely on its filename. As PJE mentioned, it would be nice if this same technique worked on Linux. However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name? Regards, Zooko ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
zooko wrote: However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name? Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux. One example is anything that uses the numpy C API (e.g. matplotlib, pyopengl 2.x, and so on). Fortunately, the numpy C API is very stable, so there hasn't been any incompatibility introduced in the numpy 1.x series. Yet. -Andrew ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
zooko wrote: Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux. Wait a minute, an extension module built into the Python Standard Library, you mean? No, I'm writing about non stdlib extension modules. Because for separately packaged packages (distributions) such as the numpy that you mentioned, your package (distribution) would express its requirement on that other package (distribution) in its install_requires metadata, not in its name. There, in the install_requires metadata, it can also express which version it requires. Right? No, because the act of compiling your .egg fixes the specific version (e.g. numpy==1.3) to keep the ABI compatible with the version of numpy installed at extension module compile time. Whereas the install_requires is about API compatibility, and could thus be numpy=1.2, for example. (For pure Python modules, this isn't a problem because there is only an API version to worry about. But with compiled extensions, there is also the ABI version to worry about.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
At 12:36 AM 4/12/2009 -0700, Buck Golemon wrote: On Fri, Apr 10, 2009 at 9:15 PM, P.J. Eby mailto:p...@telecommunity.comp...@telecommunity.com wrote: At 08:53 PM 4/10/2009 -0700, Buck wrote: I see the kernel version and architecture, but this is insufficient; RedHat 4 and RedHat 5 both use a 2.6 kernel, but the difference in provided libraries are sufficient to make many (most?) impure libraries stop working ( numpy, python-ldap, and hashlib for examples). easy_install apparently knows when packages are platform-dependant, and the necessary directory is sys.exec_prefix (or maybe better: distutils.sysconfig.get_python_lib(plat_specific=1) ), so this seems like an easy change. Would a patch be accepted? It would be a pretty major patch, since the distutils logic that's used for determining the installation directory is applied long before easy_install even knows what it's installing (and therefore, whether it's platform dependent). It would probably be a lot easier to improve the platform string generation and comparison logic, as has been done for OS X. If the distutils logic is used, then why does numpy install to the platlib folder using distutils, but into the nonplatlib folder using easy_install? Please read what I wrote after the word distutils, i.e.: the distutils logic that's used for determining the installation directory is applied long before easy_install even knows what it's installing That is, the installation directory is determined *before* any packages are examined, so there is no way to know whether prefix or exec-prefix should be used. So prefix is used to determine the default installation location. Currently, there's only a way to specify one installation location. Your tone seems to suggest that such an effort might be accepted upstream. I'm suggesting that fixing the platform strings would be more beneficial (i.e. benefit other use cases besides this one), and would probably be a LOT easier to implement as a patch, because the changes would be limited to the internals of a few well-defined, isolated functions. Of course, if you happen to find some sane way to get easy_install to do what you want, I'll certainly take a look at it. I'm just hard pressed to imagine how you'd go about doing it without major refactoring that would be very difficult to verify was done correctly. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to make easy_install handle platlibs?
At 10:43 AM 4/12/2009 -0700, Buck Golemon wrote: On Sun, Apr 12, 2009 at 8:54 AM, Andrew Straw mailto:straw...@astraw.comstraw...@astraw.com wrote: zooko wrote: However, it currently doesn't. Eggs built on Linux are named something like py2.5-Linux-x86_64. To know whether such an egg would actually work on your Linux system, you would also need to know whether the Python was compiled with UCS-2 or UCS-4 internal unicode representation, as well as what version of glibc you have. Is there anything else that would need to be added into the egg name? Yes, if you used symbols from any shared library in an extension module, you'd need to know the version of that shared library. So it's not just libc. This is the same on any OS, not just linux. -Andrew Exactly. The platform name/version ( RedHat 3 / Ubuntu 7.07 / Gentoo X.X ) can serve as a much better proxy for the version of all shared libraries than just the kernel version alone (2.4 / 2.6). To add to Andrew's example, python-ldap created for Redhat 4 depends on ldap_r.so.1.0.4 but for RedHat 5 depends on ldap_r.so.1.0.6. Both have kernel 2.6, so are indistinguishable under current naming scheme. I notice that for Debian/Ubuntu the lsb_release command is implemented in Python, although it's mostly just reading out /etc/debian_version. RedHat's lsb_release uses bash, but similarly just reads in a file. Gentoo also has a lsb_release. On OS X, there's a similar thing done to get the platform name/version, and there's special version comparison code to determine compatibility; if this could be implemented for other platforms, that would be great. The completely correct way to do this would be to run ldd on all binaries, stick the result in a metadata file somewhere, then load the version of the egg where the most (hopefully all) libraries exist. On the other hand, sticking the egg into the place that distutils uses when not under easy_install would fix this much more simply, although from what I hear this would be a big change. For easy_install, yes. For pip, probably not so much. In fact, my guess would be that pip already supports this, as long as you can install from source, rather than from eggs. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
