Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Thu, 2013-11-14 at 10:36 -0800, Ned Deily wrote: In article 1384442536.3496.532.camel@pdsdesk, Paul Smith p...@mad-scientist.net wrote: [...] By relocatable I mean runnable from any location; i.e., not fixed. I have a wrapper around the Python executable that can compute the correct root directory and set any environment variables or add flags or whatever might be needed. In that case, the python.org installer may not be a good choice. You should be to accomplish what you want by building your own Python. You'll probably find you were getting tripped up by unnecessarily setting environment variables. Good luck! Thanks Ned. I got sidetracked for a while but I got back to this now, and I found my problem. The makefile I was using to control the build on Linux was stripping the python executable to make it smaller. However, stripping the python executable on MacOSX breaks it completely so it can't load its shared libraries and I get errors as in my original message. If I remove the strip operation, then everything starts to work as expected. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Wed, 2013-11-13 at 23:06 -0800, Ned Deily wrote: On Nov 13, 2013, at 17:24 , Paul Smith p...@mad-scientist.net wrote: I'm discovering that this is tricky. I don't want to bring OS wars into it, but this kind of thing is so simple and just works on GNU/Linux. I guess I've been spoiled :-). Well, if you are building from scratch, you would likely run into some of the same issues on Linux systems, too. There are many variations in configurations that are mutually incompatible. Possibly. However as long as you're using basic system libraries and not higher-level stuff like GTK etc., you can compile on an older version of Linux (say, Red Hat EL 5 or so) and the result will run without any problems on pretty much any distribution, even the newest ones: Red Hat, Ubuntu, Debian, Suse, etc. etc. We do this regularly, for production code we ship to customers. About the only basic libraries that cause problems are libssl/libcrypto... the openssl folks just don't seem to care much about this issue. I'm somewhat dreading my next effort after MacOS: the same thing, on Windows :-/. And another task, which seems like it will be fun: building GDB on MacOSX with Python support enabled... Keep in mind that on newer OS X releases, Apple no longer ships gcc or gdb. The standard compiler is clang and it has its own debuggers. Yes, I'm aware. Very frustrating. We have a large set of specialized macros, both in native GDB and in Python using GDB's extension, that are invaluable for debugging our code. I'm not thrilled about the idea of trying to get all that working in lldb as well. So we're going to stick with GDB, building it ourselves, until it no longer works at all. At that point we'll decide how to proceed. Hm, that's an idea. I don't HAVE to build Python myself, actually, I just need (a) it to be relocatable, and (b) to add these extra modules to it so I can use it across systems without installing them individually by hand. I'm not sure I know what you mean by relocatable. Like the third-party package manager solutions, the python.org installers place the Pythons they install in fixed locations (under /Library/Frameworks with links from /usr/local/bin) but one that does not conflict with any system files, including the Apple-supplied system Pythons. By relocatable I mean runnable from any location; i.e., not fixed. I have a wrapper around the Python executable that can compute the correct root directory and set any environment variables or add flags or whatever might be needed. Basically I have a large number of test systems and a set of test suites that are all written in Python, and I need to be able to distribute the same version of Python plus a specific set of additional modules across all those systems to be sure they all have the same environment. Plus the set of systems changes (new systems added/old ones removed) regularly. I'm handling this by checking in the Python distribution plus modules into a tools Git repository, then cloning it on each system. However, I have no special privileges on these systems (so I can't modify any system locations such as /usr/local) and I can't control what user account will be running the tests (so the user's home directory, where the tools repository is cloned, is not constant across the systems). I have this working on Linux with very little effort. Now I'm trying to get the same result on MacOSX. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
In article 1384442536.3496.532.camel@pdsdesk, Paul Smith p...@mad-scientist.net wrote: [...] By relocatable I mean runnable from any location; i.e., not fixed. I have a wrapper around the Python executable that can compute the correct root directory and set any environment variables or add flags or whatever might be needed. In that case, the python.org installer may not be a good choice. You should be to accomplish what you want by building your own Python. You'll probably find you were getting tripped up by unnecessarily setting environment variables. Good luck! -- Ned Deily, n...@acm.org -- https://mail.python.org/mailman/listinfo/python-list
python 2.7.x on MacOSX: failed dlopen() on .so's
Hi all. I need to build my own version of Python on a MacOSX system, and I can't seem to do it successfully. I need to build it with a particular location, etc. and so I can't use Homebrew or whatever: I need to compile it myself from the source tarball. I did look through the Homebrew recipe and I don't see anything that seems like it would help with my problem. I'm using MacOSX 10.7.5 with xcode 4.1, containing gcc 4.2.1 / clang 2.1 (configure seems to choose gcc). I've tried this with both Python 2.7.5 and 2.7.6. I get the tarball, unpack it, then: $ configure --prefix=/Users/build/python $ make $ make install $ export PATH=/Users/build/python/bin:$PATH $ export PYTHONHOME=/Users/build/python $ python2.7 --version Python 2.7.6 Simple scripts seem to work fine. But then I need to install various packages; for example pycrypto. When I try to build that it fails to build: $ cd src/pycrypto-2.6 python2.7 setup.py build Traceback (most recent call last): File setup.py, line 45, in module import struct File /Users/build/python/lib/python2.7/struct.py, line 1, in module from _struct import * ImportError: dlopen(/Users/build/python/lib/python2.7/lib-dynload/_struct.so, 2): Symbol not found: _PyObject_Free Referenced from: /Users/build/python/lib/python2.7/lib-dynload/_struct.so Expected in: dynamic lookup This happens with every module that is not pure-Python: boto, paramiko, etc. Looking at the various python2.7/lib-dynload/*.so files with nm I see that actually a number of them refer to an unresolved (U) symbol _PyObject_Free (and other _PyObject_* symbols). I can't find any shared library being built by the python build (.dylib or .so or whatever); there's just a libpython2.7.a file (I tried forcing --enable-shared on the configure line and it didn't change anything). I should say that I've used this same configuration and set of steps, etc. on my GNU/Linux system and it worked perfectly. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
In article 1384370183.3496.472.camel@pdsdesk, Paul Smith p...@mad-scientist.net wrote: I'm using MacOSX 10.7.5 with xcode 4.1, containing gcc 4.2.1 / clang 2.1 (configure seems to choose gcc). I've tried this with both Python 2.7.5 and 2.7.6. I get the tarball, unpack it, then: $ configure --prefix=/Users/build/python $ make $ make install $ export PATH=/Users/build/python/bin:$PATH $ export PYTHONHOME=/Users/build/python $ python2.7 --version Python 2.7.6 Simple scripts seem to work fine. But then I need to install various packages; for example pycrypto. When I try to build that it fails to build: $ cd src/pycrypto-2.6 python2.7 setup.py build Traceback (most recent call last): File setup.py, line 45, in module import struct File /Users/build/python/lib/python2.7/struct.py, line 1, in module from _struct import * ImportError: dlopen(/Users/build/python/lib/python2.7/lib-dynload/_struct.so, 2): Symbol not found: _PyObject_Free Referenced from: /Users/build/python/lib/python2.7/lib-dynload/_struct.so Expected in: dynamic lookup This happens with every module that is not pure-Python: boto, paramiko, etc. Looking at the various python2.7/lib-dynload/*.so files with nm I see that actually a number of them refer to an unresolved (U) symbol _PyObject_Free (and other _PyObject_* symbols). I can't find any shared library being built by the python build (.dylib or .so or whatever); there's just a libpython2.7.a file (I tried forcing --enable-shared on the configure line and it didn't change anything). There shouldn't be any problems with what you are trying to do. It works for me with Python 2.7.6 and pycrypto-2.6.1. Some suggestions: - Avoid --enable-shared on OS X at least initially. There are too many ways things can go wrong. If you've built with it, suggest starting with a fresh Python source directory just to be sure. - Check the dynamic library dependencies of _struct. On OS X: otool -L /Users/build/python/lib/python2.7/lib-dynload/_struct.so For a non-shared build, the only library dependency should be /usr/lib/libSystem.B.dylib. - Make sure you are *really* building pycrypto and friends with your Python and not with some other one. - On OS X with an installed Python, you shouldn't need to set PYTHONHOME. -- Ned Deily, n...@acm.org -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
In article nad-e7cc3b.14401113112...@news.gmane.org, Ned Deily n...@acm.org wrote: There shouldn't be any problems with what you are trying to do. It works for me with Python 2.7.6 and pycrypto-2.6.1. Some suggestions: - Avoid --enable-shared on OS X at least initially. There are too many ways things can go wrong. If you've built with it, suggest starting with a fresh Python source directory just to be sure. - Check the dynamic library dependencies of _struct. On OS X: otool -L /Users/build/python/lib/python2.7/lib-dynload/_struct.so For a non-shared build, the only library dependency should be /usr/lib/libSystem.B.dylib. - Make sure you are *really* building pycrypto and friends with your Python and not with some other one. - On OS X with an installed Python, you shouldn't need to set PYTHONHOME. - Check your other environment variables and make sure you are not setting any DYLD_ or LD_ env variables. -- Ned Deily, n...@acm.org -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Nov 13, 2013, at 14:59 , Paul Smith p...@mad-scientist.net wrote: Thanks for the response Ned! On Wed, 2013-11-13 at 14:40 -0800, Ned Deily wrote: There shouldn't be any problems with what you are trying to do. It works for me with Python 2.7.6 and pycrypto-2.6.1. Some suggestions: - Avoid --enable-shared on OS X at least initially. There are too many ways things can go wrong. If you've built with it, suggest starting with a fresh Python source directory just to be sure. I've tried it with all three options: no flag, --disable-shared, and --enable-shared, and don't notice any difference in the results; even with --enable-shared I don't seem to get any shared libs created. - Check the dynamic library dependencies of _struct. On OS X: otool -L /Users/build/python/lib/python2.7/lib-dynload/_struct.so I get a libgcc_s reference as well, since I'm building with GCC: /Users/build/python/lib/python2.7/lib-dynload/_struct.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1105.0.0) I wonder if using the GCC linker etc. is part of the problem? That shouldn't be an issue. What you do don't want to see is references to python libraries. BTW, Xcode 4.1 is quite old and outdated for OS X 10.7. Suggest you update to the current Xcode 4.6.3 or Command Line Tools for 10.7. The reason I've set PYTHONHOME is ultimately I need this installation to be relocatable. It's going to be shared across lots of different systems and they'll have the ability to copy it wherever they want. That could be problematic. You need to be *really* careful about how you do that. You stand a chance with a non-shared installation. You still should not need to set PYTHONHOME. Also, be aware that executables and libraries built on one version of OS X are not guaranteed to work on other versions, particularly older versions unless you take certain precautions. Even non-shared Pythons on OS X dynamically link with system-supplied libraries which can vary across os releases. And not all libraries are supplied, so, depending on your needs, you may need to supply some additional third-party libraries. If you need to support various OS X releases, the safest approach is to build on the oldest release to be supported, say, 10.6. The resultant executables and libraries should then work on 10.7 through 10.9 - except every once in a while there will be a gotcha like the incompatible change in libedit for 10.9. (Python 2.7.6 can deal with that problem.) There is also the question if all the systems you will need to support are of the same architecture. 10.6 supports both 32-bit-only and 64-bit Intel machines. 10.5 supports in addition PPC machines. The solution to that is to build OS X universal binaries. If you really are careful and lucky, it is in theory possible to build on a newer system and use an OS X SDK and set MACOSX_DEPLOYMENT_TARGET for an older version and it will be downward compatible. But thorough testing is called for in that case since it is easy to have a hidden dependency. For the python.org OS X binary installers, we go to a fair amount of trouble to build P ythons that will work across a range of OS X releases. You might want to consider using one of them as a base. It's usually a lot less work than trying to make it work yourself. - Check your other environment variables and make sure you are not setting any DYLD_ or LD_ env variables. Hm; I am setting LD_LIBRARY_PATH to find the Python .so files. Does python figure out where to look for them by itself? Yes -- Ned Deily n...@acm.org -- [] -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
Thanks for the response Ned! On Wed, 2013-11-13 at 14:40 -0800, Ned Deily wrote: There shouldn't be any problems with what you are trying to do. It works for me with Python 2.7.6 and pycrypto-2.6.1. Some suggestions: - Avoid --enable-shared on OS X at least initially. There are too many ways things can go wrong. If you've built with it, suggest starting with a fresh Python source directory just to be sure. I've tried it with all three options: no flag, --disable-shared, and --enable-shared, and don't notice any difference in the results; even with --enable-shared I don't seem to get any shared libs created. - Check the dynamic library dependencies of _struct. On OS X: otool -L /Users/build/python/lib/python2.7/lib-dynload/_struct.so I get a libgcc_s reference as well, since I'm building with GCC: /Users/build/python/lib/python2.7/lib-dynload/_struct.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0) /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1105.0.0) I wonder if using the GCC linker etc. is part of the problem? - Make sure you are *really* building pycrypto and friends with your Python and not with some other one. I'm pretty sure but I'll triple-check. The reason I've set PYTHONHOME is ultimately I need this installation to be relocatable. It's going to be shared across lots of different systems and they'll have the ability to copy it wherever they want. - Check your other environment variables and make sure you are not setting any DYLD_ or LD_ env variables. Hm; I am setting LD_LIBRARY_PATH to find the Python .so files. Does python figure out where to look for them by itself? Thanks for this info; I'll get back to you. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Nov 13, 2013, at 17:24 , Paul Smith p...@mad-scientist.net wrote: On Wed, 2013-11-13 at 16:00 -0800, Ned Deily wrote: The reason I've set PYTHONHOME is ultimately I need this installation to be relocatable. It's going to be shared across lots of different systems and they'll have the ability to copy it wherever they want. That could be problematic. You need to be *really* careful about how you do that. You stand a chance with a non-shared installation. You mean, --disable-shared? That's what I want, ultimately, but I was going to start with the default configuration first. --disable-shared is the default, the same as not specifying --enable-shared. I'm discovering that this is tricky. I don't want to bring OS wars into it, but this kind of thing is so simple and just works on GNU/Linux. I guess I've been spoiled :-). Well, if you are building from scratch, you would likely run into some of the same issues on Linux systems, too. There are many variations in configurations that are mutually incompatible. If you mean due to the lack of a system package manager like yum or apt-get, there are various third-party package managers for OS X: MacPorts, homebrew, and fink being the most well-known. They all provide Pythons and many third-party Python packages. I'm somewhat dreading my next effort after MacOS: the same thing, on Windows :-/. And another task, which seems like it will be fun: building GDB on MacOSX with Python support enabled... Keep in mind that on newer OS X releases, Apple no longer ships gcc or gdb. The standard compiler is clang and it has its own debuggers. For the python.org OS X binary installers, we go to a fair amount of trouble to build Pythons that will work across a range of OS X releases. You might want to consider using one of them as a base. It's usually a lot less work than trying to make it work yourself. Hm, that's an idea. I don't HAVE to build Python myself, actually, I just need (a) it to be relocatable, and (b) to add these extra modules to it so I can use it across systems without installing them individually by hand. I'm not sure I know what you mean by relocatable. Like the third-party package manager solutions, the python.org installers place the Pythons they install in fixed locations (under /Library/Frameworks with links from /usr/local/bin) but one that does not conflict with any system files, including the Apple-supplied system Pythons. I don't know exactly what you are trying to do but a couple of suggestions. A number of widely-used Python projects, like Numpy, provide their own OS X installer package (.mpkg or .pkg) that installs into the site-packages directory of the python.org Python 2.7. That's the sort of traditional way of doing it. A more recent development is deploying Python packages (not Python itself) using the wheel format. (http://wheel.readthedocs.org/en/latest/). The wheel is a cross-platform format and replaces earlier binary distribution formats like eggs. It is supported by recent versions of pip (https://pypi.python.org/pypi/pip), which itself replaces earlier installer commands like easy-install or using python setup.py install. Using pip and wheels have been endorsed by the Python core development team as the new one obvious way for distributions (PEP 453). pip will be automatically included with Python 3.4 but it is very easy to install and use with current Pythons (3.3.x and 2.7.x). You should be able to use pip and wheels, possibly platform-specific wheels, for deploying to all of your platforms: Linux, OS X, and Windows. For OS X and Windows, the deployment would be very similar: install Python from a python.org installer, install pip (and its current setuptools dependency), install additional wheels. On the various Linux systems, some of those steps could be replaced by using the system-supplied packages. Also, if it is not desired or possible to install everything into a shared location on the deployed systems, e.g. there are multiple users or the user does not have admin privileges, you could either use the pip --user option or use virtual environments (virtualenv) in which to install wheels. That may not help in your situation but those are the paths of least resistance these days. -- Ned Deily n...@acm.org -- [] -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2.7.x on MacOSX: failed dlopen() on .so's
On Wed, 2013-11-13 at 16:00 -0800, Ned Deily wrote: The reason I've set PYTHONHOME is ultimately I need this installation to be relocatable. It's going to be shared across lots of different systems and they'll have the ability to copy it wherever they want. That could be problematic. You need to be *really* careful about how you do that. You stand a chance with a non-shared installation. You mean, --disable-shared? That's what I want, ultimately, but I was going to start with the default configuration first. I'm discovering that this is tricky. I don't want to bring OS wars into it, but this kind of thing is so simple and just works on GNU/Linux. I guess I've been spoiled :-). I'm somewhat dreading my next effort after MacOS: the same thing, on Windows :-/. And another task, which seems like it will be fun: building GDB on MacOSX with Python support enabled... You still should not need to set PYTHONHOME. Also, be aware that executables and libraries built on one version of OS X are not guaranteed to work on other versions, particularly older versions unless you take certain precautions. Even non-shared Pythons on OS X dynamically link with system-supplied libraries which can vary across os releases. And not all libraries are supplied, so, depending on your needs, you may need to supply some additional third-party libraries. This is why I'm building on this very old system, and am loath to update it. One saving grace is that while I need my installation to be relocatable, I _don't_ need it to be infinitely portable across MacOSX systems. I'm using it internally only and so I have some control over the version of MacOS and the hardware that it's running on. I don't need to worry about non-Intel hardware, or versions of MacOS prior to the one I'm using here. For the python.org OS X binary installers, we go to a fair amount of trouble to build Pythons that will work across a range of OS X releases. You might want to consider using one of them as a base. It's usually a lot less work than trying to make it work yourself. Hm, that's an idea. I don't HAVE to build Python myself, actually, I just need (a) it to be relocatable, and (b) to add these extra modules to it so I can use it across systems without installing them individually by hand. Thanks, I'll look into this further. -- https://mail.python.org/mailman/listinfo/python-list
