FWIW, the Spack Python builds (for 3.5 and 2.7 at least) work on El Capitan, thanks to a number of recent contributions. They don't use any patches.
Spack bridges the everything-in-its-own-prefix gap by allowing you to activate/deactivate python packages on the fly. You install the module in its own prefix then `spack activate <module>`, and it is symlinked into place within the python prefix (with special handling for eggs, .pth files, etc.) Think of it as though each python installation is its own virtualenv. Modules are installed in their own prefix but can be symlinked into the python prefix on demand: http://software.llnl.gov/spack/basic_usage.html#extensions-python-support http://software.llnl.gov/spack/packaging_guide.html#extensions We use this to manage Python installs at LLNL. The same support can be implemented for other languages that have their own module systems via Spack extensions (e.g., we're thinking about lua). Also, we install our own setuptools, and packages that use it need to depend on it, or you'll get the aforementioned problem where everything wants to bootstrap its own setuptools. Depending on setuptools ensures that one is already installed. -Todd From: <[email protected]> on behalf of Elizabeth Fischer <[email protected]> Reply-To: "[email protected]" <[email protected]> List-Post: [email protected] Date: Tuesday, February 9, 2016 at 9:31 AM To: "[email protected]" <[email protected]> Subject: Re: [easybuild] More on EasyBuild and (DY)LD_LIBRARY_PATH > > > On Tue, Feb 9, 2016 at 11:04 AM, Kenneth Hoste <[email protected]> wrote: >> Hi Elizabeth, >> >> On 09/02/16 16:55, Elizabeth Fischer wrote: >>> So... I'm compiling my Python environment with MacPorts, and then stuff on >>> top of that with EasyBuild. Why MacPorts? Because I just want it built, >>> and lots of patches are required to compile any other way. It's one big >>> **** mess. >> >> What kind of problems are you running into exactly when compiling Python with >> EasyBuild on OS X? > > To summarize... lots of little things. Some of the MacPorts builds are > heavily patched to fix these problems. The MacPort for Python34 has no fewer > than seven such patches: > > https://trac.macports.org/browser/trunk/dports/lang/python34/files > > I forget exactly what was causing me the problem... but I realized it was > addressed in one of the patches. And if I fixed it, I'd have another 6 to go. > I need a better way... > >> As you know, OS X isn't our main target, but depending on how much effort is >> required, we can try and get the issues fixed, if we know about them... > > EasyBuild does not have the resources to replicate the frequently-extensive > patching work of MacPorts on even a fraction of the ports. That is a losing > proposition... > > What MIGHT work would be an EasyBlock that's able to look at the MacPorts > files for a particular port, and then do the same thing within the EasyBuild > system. This would allow EasyBuild to piggy-back on existing MacPorts when > running on a Mac. > > We would lose any semblance of cross-platform at that point. But maybe we'd > have a special mac/ directory of EasyConfigs for Mac, which would override the > standard directory of cross-platform easyconfigs. > >>> Anyway... >>> >>> I've found that setting (DY)LD_LIBRARY_PATH to include the MacPorts >>> directory breaks various Python modules. To be specific: it causes the Mac >>> loader to find dynamic libraries from MacPorts instead of the /System path. >>> This bug is described here: >>> >>> https://lists.macosforge.org/pipermail/macports-users/2010-November/022662.h >>> tml >> >> Well, if you set (DY)LD_LIBRARY_PATH, that's exactly what you want, i.e. >> libraries in that location being picked up, no? > > Apparently, no. For whatever reason, the binaries expected the System version > of the libraries. Setting DYLD_LIBRARY_PATH causes them to get the MacPorts > version of the libraries instead, which doesn't work. This kind of DLL-hell > is why people say LD_LIBRARY_PATH is evil. >>> I'm compiling my stuff (libraries and Python extensions) with rpath. So I >>> don't need DYLD_LIBRARY_PATH. The solution, for now, is to run Python >>> without any DYLD_LIBRARY_PATH. >>> >>> But this is convincing me that enforcing RPATH on all compiles, as is done >>> in Spack, should be a high priority for EasyBuild. >> >> We're actively looking into RPATH support, keep an eye on >> https://github.com/hpcugent/easybuild-framework/issues/651 (and the notes of >> the EasyBuild conf calls). > > Awesome! > > To change the topic slightly... my larger problem is getting a useful Python > distro built, one that I can build Python extensions against. EasyBuild > failed for the basic Python (on a Mac). MacPorts works, but not (yet) on El > Capitain + GCC (Apple changed the Objective-C language again). Even just > compiling the plain Python is a real nightmare. > > But there's more... Cython wants to compile extensions with the same C > compiler used to compile Python. This means that if you build your Python > with Apple's Clang, then Cython will try to use that for your (C++) > extensions. If you use Fortran+C++, then you really need GCC. ==> <== > > I'm looking into ways that I can compile Cython extensions with a DIFFERENT > compiler than the one used to build Python. If that works out, If that works > out, then maybe something like Anaconda would be my friend --- it gives you a > lot out-of-the-box, but it decides how to build Python itself. Then I would > just need to use EasyBuild for the stuff that EasyBuild was intended for --- > i.e. C/Fortran libraries. > > Still one big mess. > > -- Elizabeth >
