Hello Uwe, Yes, it would appear that was the case. Aggregating yours and Wes' suggestions, I'm now:
- Compiling arrow-cpp myself, explicitly with the new ABI, and publishing this on our internal conda channel (as gsa-arrow-cpp, so it can be explicitly depended upon). Did this by tweaking https://github.com/conda-forge/arrow-cpp-feedstock/tree/master/recipe a little bit. - Compiling arrow-matlab (which has fromArrowStream and fromArrowFile) also explicitly with the new ABI, depending on the aforementioned gsa-arrow-cpp - Launch MATLAB with an explicit LD_PRELOAD for a sufficiently modern libstdc++ (also from Anaconda) And now it all works fine! I'm not sure if this is the most elegant solution - it does feel a bit awkward - but it does allow us to make progress. Thanks, -J On Tue, Mar 12, 2019 at 8:49 PM Uwe L. Korn <uw...@xhochy.com> wrote: > Hello Joris, > > > '/data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64': > > /data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64: > undefined symbol: _ZNK5arrow6Status8ToStringEv. > > sounds like fromArrowStream.mexa64 was still compiled using > -D_GLIBCXX_USE_CXX11_ABI=0, you might try to explicitly pass > -D_GLIBCXX_USE_CXX11_ABI=1 or when building an conda environment, be sure > to have the compilers package installed and use the $CC and $CXX > environment variables to pick the right compilers. You may also need to > LD_PRELOAD the libstdc++.so that is coming with conda and not the one > coming from the system. > > Anaconda/defaults and conda-forge based Arrow packages are both nowadays > built with the new ABI. But they are built with slightly different > toolchains, so it is best to only install packages from one of the two > repositories and don't mix them. > > Uwe > > > On Tue, Mar 12, 2019, at 5:32 PM, Wes McKinney wrote: > > hi Joris, > > > > You probably ran into the conda-forge compiler migration. I'm not sure > > about Anaconda's Apache Arrow libraries since they maintain those > > recipes. > > > > If you need shared libaries using the gcc 4.x ABI you may have to > > build them yourself right, or use the Linux packages for the platform > > where you are working. It would be useful to have a Dockerfile that > > produces "portable" shared libraries with the RedHat devtoolset-2 > > compiler > > > > - Wes > > > > On Tue, Mar 12, 2019 at 11:22 AM Joris Peeters > > <joris.mg.peet...@gmail.com> wrote: > > > > > > [A] Short background: > > > > > > We are working on a MEX library that converts a binary array > (representing > > > an Arrow stream or file) into MATLAB structs. This is in > > > parallel/complement to what already exists in the main Arrow project, > which > > > focuses on feather, but the hope is certainly to contribute back some > of > > > this work. We talked to MathWorks about this already (as GSA Capital). > > > > > > The library (e.g. fromArrowStream.mexa64) gets published on > > > (company-internal) Anaconda, and upon installation the dependencies on > > > arrow-cpp, boost-cpp etc are resolved (from remote channels). All > .so's end > > > up in a user-local conda environment's ../lib, which in MATLAB we make > > > available through addpath. Compilation uses > -D_GLIBCXX_USE_CXX11_ABI=0. > > > > > > [B] The issue I'm facing ... > > > > > > For quite a while (when we depended on arrow-cpp=0.10 from conda-forge) > > > this has worked fine, but lately I've encountered increasing issues > wrt ABI > > > compatibility, amongst arrow, gtest (both at build time) and MATLAB (at > > > run-time). > > > > > > Is arrow-cpp 0.11 only built with the new ABI? I loaded it from both > > > defaults and conda-forge channel, and it seems different in this regard > > > than conda-forge's 0.10. Either way, I'm now attempting to built my > library > > > without the -D_GLIBCXX_USE_CXX11_ABI=0 compile flag, as that seems to > be > > > the more sustainable way forward. > > > > > > Question: is it possible to load a MEX library that has been compiled > with > > > the new ABI? When doing this naively, I get an error like: > > > > > > Invalid MEX-file > > > > '/data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64': > > > > /data/home/jpeeter/apps/matlab/MATLAB/R2018a/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6: > > > version `CXXABI_1.3.11' not found (required by > > > /data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/libarrow.so.11). > > > > > > which is fair enough. > > > > > > Alternatively, though, when loading MATLAB with an LD_PRELOAD, like > > > LD_PRELOAD=/usr/lib64/libstdc++.so.6 > ~/apps/matlab/MATLAB/R2018a/bin/matlab > > > > > > I get this error: > > > Invalid MEX-file > > > > '/data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64': > > > > /data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64: > > > undefined symbol: _ZNK5arrow6Status8ToStringEv. > > > > > > If this isn't possible, is there a reliable/recommended Anaconda-way to > > > only bring in libraries that have been compiled with the old ABI? My > > > impression was that conda-forge libraries satisfied that, but > > > - This no longer seems to be true for arrow-cpp=0.11? I might be > mistaken > > > here. > > > - We are resolving our dependencies through Anaconda, so it could be > quite > > > brittle for users to explicitly have to specify a channel for certain > > > libraries. There are further subtleties wrt resolving boost & arrow > from > > > different channels etc. Ideally - but not necessarily - users don't > depend > > > on conda-forge at all, but only on the default channels (as an aside: > we > > > use Artifactory internally). > > > > > > Essentially, I'm happy with either, > > > - having a way for MATLAB to load MEX with the new ABI, or, > > > - reliably depending on libraries compiled with the old ABI > > > > > > but I've been struggling for a while now to achieve either one of > those. > > > Any pointers would be most welcome! > > > > > > (For once, this turned out to be easier on Windows than Linux!) > > > > > > Best, > > > -Joris. > > >
