You might want to take a look at [1] which has some discussion from a previous (unfortunately, ultimately unsuccessful) attempt at getting pybind11 to work. If you give us a high level overview of what you are trying to achieve we may also be able to suggest a better solution. For example, you may be able to achieve what you want using the C data interface [2] which is going to be more stable.
If you want to use pybind11 however you will need to ensure that the C & C++ libraries (e.g. glibc / libstdc++) used to build the binary shared objects shipped with pyarrow are the same as the libraries used to build your own projects. My guess is that the error you are seeing in (2) is a binary incompatibility between libstdc++ versions. Since you are installing pyarrow packages from PIP you may want to read up on manylinux, which is PIP's attempt to handle this challenge. Pyarrow's PIP wheels are manylinux wheels. Pyarrow distributes both a manylinux2010 [3] wheel which supports: GLIBC_2.12 CXXABI_1.3.3 GLIBCXX_3.4.13 GCC_4.5.0 ...and a manylinux2014 [4] wheel which supports: GLIBC_2.17 CXXABI_1.3.7, CXXABI_TM_1 is also allowed GLIBCXX_3.4.19 GCC_4.8.0 I hope this helps get you started. [1] https://github.com/apache/arrow/issues/10488 [2] https://arrow.apache.org/docs/format/CDataInterface.html [3] https://www.python.org/dev/peps/pep-0571/ [4] https://www.python.org/dev/peps/pep-0599/ On Mon, Sep 20, 2021 at 9:04 AM Arthur Andres <[email protected]> wrote: > > Hello all, > > I want to use pybind11 with arrow, but I can't get it to work. I get an > "undefined symbol" error at run time. > > I've put a simple version of what I'm trying to do on github: > https://github.com/0x26res/pyarrowbind > > In a nutshell, I'm trying to follow the instructions here: > https://arrow.apache.org/docs/python/extending.html#c-api > * I have a virtual environment with pybind, pyarrow (and numpy) installed > * I've made sure I've called `pa.create_library_symlinks()` > * I'm linking against libarrow_python.so and libarrow.so > > Here's how far I got to far: > 1. If my C++ code only tries to call `arrow::py::import_pyarrow()`, it works > (and returns 0) > 2. If my C++ code tries to do anything else with arrow (like creating a > schema), I get this error: > ``` > Traceback (most recent call last): > File "<string>", line 1, in <module> > ImportError: > /home/arthur/source/pyarrowbind/build/bindings.cpython-38-x86_64-linux-gnu.so: > undefined symbol: _ZNK5arrow6Schema8ToStringB5cxx11Eb > ``` > > 3. I tried to link against the "arrow_static" library, provided by > `FIND_PACKAGE(Arrow REQUIRED)` in cmake. When I do that I get a different > error: > ``` > Traceback (most recent call last): > File "<string>", line 1, in <module> > ImportError: > /home/arthur/source/pyarrowbind/build/bindings.cpython-38-x86_64-linux-gnu.so: > undefined symbol: ERR_error_string_n > ``` > This is pointing to a symbol defined in openssl. I tried to fix this, then > got a similar error with curl symbols, which I fixed and eventually got a > segfault. Which makes me think I must be doing something wrong. > > > Do you have any idea about what I'm doing wrong? > > I'm using: > Ubuntu 20.04.3 LTS > Python 3.8.10 > pyarrow==5.0.0 > pybind11==2.7.1 > numpy==1.21.2 > > Thanks.
