On Tue, Feb 9, 2016 at 11:04 AM, Kenneth Hoste <[email protected]> wrote:
> Hi Elizabeth, > > On 09/02/16 16:55, Elizabeth Fischer wrote: > > So... I'm compiling my Python environment with MacPorts, and then stuff on > top of that with EasyBuild. Why MacPorts? Because I just want it built, > and lots of patches are required to compile any other way. It's one big > **** mess. > > > What kind of problems are you running into exactly when compiling Python > with EasyBuild on OS X? > To summarize... lots of little things. Some of the MacPorts builds are heavily patched to fix these problems. The MacPort for Python34 has no fewer than seven such patches: https://trac.macports.org/browser/trunk/dports/lang/python34/files I forget exactly what was causing me the problem... but I realized it was addressed in one of the patches. And if I fixed it, I'd have another 6 to go. I need a better way... As you know, OS X isn't our main target, but depending on how much effort > is required, we can try and get the issues fixed, if we know about them... > EasyBuild does not have the resources to replicate the frequently-extensive patching work of MacPorts on even a fraction of the ports. That is a losing proposition... What MIGHT work would be an EasyBlock that's able to look at the MacPorts files for a particular port, and then do the same thing within the EasyBuild system. This would allow EasyBuild to piggy-back on existing MacPorts when running on a Mac. We would lose any semblance of cross-platform at that point. But maybe we'd have a special mac/ directory of EasyConfigs for Mac, which would override the standard directory of cross-platform easyconfigs. Anyway... > > I've found that setting (DY)LD_LIBRARY_PATH to include the MacPorts > directory breaks various Python modules. To be specific: it causes the Mac > loader to find dynamic libraries from MacPorts instead of the /System > path. This bug is described here: > > > https://lists.macosforge.org/pipermail/macports-users/2010-November/022662.html > > > Well, if you set (DY)LD_LIBRARY_PATH, that's exactly what you want, i.e. > libraries in that location being picked up, no? > Apparently, no. For whatever reason, the binaries expected the System version of the libraries. Setting DYLD_LIBRARY_PATH causes them to get the MacPorts version of the libraries instead, which doesn't work. This kind of DLL-hell is why people say LD_LIBRARY_PATH is evil. > I'm compiling my stuff (libraries and Python extensions) with rpath. So I > don't need DYLD_LIBRARY_PATH. The solution, for now, is to run Python > without any DYLD_LIBRARY_PATH. > > But this is convincing me that enforcing RPATH on all compiles, as is > done in Spack, should be a high priority for EasyBuild. > > > We're actively looking into RPATH support, keep an eye on > https://github.com/hpcugent/easybuild-framework/issues/651 (and the notes > of the EasyBuild conf calls). > Awesome! To change the topic slightly... my larger problem is getting a useful Python distro built, one that I can build Python extensions against. EasyBuild failed for the basic Python (on a Mac). MacPorts works, but not (yet) on El Capitain + GCC (Apple changed the Objective-C language again). Even just compiling the plain Python is a real nightmare. But there's more... Cython wants to compile extensions with the same C compiler used to compile Python. This means that if you build your Python with Apple's Clang, then Cython will try to use that for your (C++) extensions. If you use Fortran+C++, then you really need GCC. ==> <== I'm looking into ways that I can compile Cython extensions with a DIFFERENT compiler than the one used to build Python. If that works out, If that works out, then maybe something like Anaconda would be my friend --- it gives you a lot out-of-the-box, but it decides how to build Python itself. Then I would just need to use EasyBuild for the stuff that EasyBuild was intended for --- i.e. C/Fortran libraries. Still one big mess. -- Elizabeth
