[A] Short background: We are working on a MEX library that converts a binary array (representing an Arrow stream or file) into MATLAB structs. This is in parallel/complement to what already exists in the main Arrow project, which focuses on feather, but the hope is certainly to contribute back some of this work. We talked to MathWorks about this already (as GSA Capital).
The library (e.g. fromArrowStream.mexa64) gets published on (company-internal) Anaconda, and upon installation the dependencies on arrow-cpp, boost-cpp etc are resolved (from remote channels). All .so's end up in a user-local conda environment's ../lib, which in MATLAB we make available through addpath. Compilation uses -D_GLIBCXX_USE_CXX11_ABI=0. [B] The issue I'm facing ... For quite a while (when we depended on arrow-cpp=0.10 from conda-forge) this has worked fine, but lately I've encountered increasing issues wrt ABI compatibility, amongst arrow, gtest (both at build time) and MATLAB (at run-time). Is arrow-cpp 0.11 only built with the new ABI? I loaded it from both defaults and conda-forge channel, and it seems different in this regard than conda-forge's 0.10. Either way, I'm now attempting to built my library without the -D_GLIBCXX_USE_CXX11_ABI=0 compile flag, as that seems to be the more sustainable way forward. Question: is it possible to load a MEX library that has been compiled with the new ABI? When doing this naively, I get an error like: Invalid MEX-file '/data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64': /data/home/jpeeter/apps/matlab/MATLAB/R2018a/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/libarrow.so.11). which is fair enough. Alternatively, though, when loading MATLAB with an LD_PRELOAD, like LD_PRELOAD=/usr/lib64/libstdc++.so.6 ~/apps/matlab/MATLAB/R2018a/bin/matlab I get this error: Invalid MEX-file '/data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64': /data/home/jpeeter/apps/anaconda3/envs/testarrow/lib/fromArrowStream.mexa64: undefined symbol: _ZNK5arrow6Status8ToStringEv. If this isn't possible, is there a reliable/recommended Anaconda-way to only bring in libraries that have been compiled with the old ABI? My impression was that conda-forge libraries satisfied that, but - This no longer seems to be true for arrow-cpp=0.11? I might be mistaken here. - We are resolving our dependencies through Anaconda, so it could be quite brittle for users to explicitly have to specify a channel for certain libraries. There are further subtleties wrt resolving boost & arrow from different channels etc. Ideally - but not necessarily - users don't depend on conda-forge at all, but only on the default channels (as an aside: we use Artifactory internally). Essentially, I'm happy with either, - having a way for MATLAB to load MEX with the new ABI, or, - reliably depending on libraries compiled with the old ABI but I've been struggling for a while now to achieve either one of those. Any pointers would be most welcome! (For once, this turned out to be easier on Windows than Linux!) Best, -Joris.
