I missed that you have an application that embeds the Python interpreter. That definitely makes things more complicated, I don't know enough to be able to help with this further
On Wed, Jan 8, 2020 at 10:01 AM Raúl Bocanegra Algarra <[email protected]> wrote: > > Hi Wes, > > I am checking out tag 0.15.1 from arrow repo then build my own pyarrow to > have the c++11 ABI as you recommended here: > http://mail-archives.apache.org/mod_mbox/arrow-dev/201808.mbox/%3cCAJPUwMA2rF4ZqbLnRD0-bmfGf6ybj8n3Xi=hnkfexs3dvc5...@mail.gmail.com%3e. > So I do: > $ export > PYARROW_CMAKE_OPTIONS=-DCMAKE_TOOLCHAIN_FILE=/home/raul.bocanegra/Code/vcpkg/scripts/buildsystems/vcpkg.cmake > $ export LD_LIBRARY_PATH=$VCPKG_ROOT/installed/x64-linux/lib > $ python setup.py install > > I modified arrow/cpp/CMakeLists.txt the line: "find_package(Arrow REQUIRED)" > to "find_package(arrow CONFIG REQUIRED)" for CMake to find the config scripts > from vcpkg. > This way as you say, I don't have the bundled arrow libs, but the vcpkg ones. > If I run ldd on the python module _fs for example inside the pyarrow folder > it links against the vcpkg arrow and arrow_python libs. But on runtime it > throws the fatal python error. > > Anyway I am going to try building Arrow without vcpkg. I assume that the > correct way to use Arrow C++ and Arrow Python C++ api is to tell CMake that > arrow libs are in pyarrow pip installation, right? I mean, we must avoid > linking libarrow.so on vcpkg folder or any other install folder and link > libarrow_python.so from pyarrow pip installation. We must link both from > pyarrow installation. Is that correct? > > Thanks for your support! > > Regards, > > Raúl Bocanegra Algarra > > Software Engineer > > M: + 34 617 83 64 45 - E: [email protected] > > > > ________________________________ > From: Wes McKinney <[email protected]> > Sent: Tuesday, January 7, 2020 6:28 PM > To: [email protected] <[email protected]> > Cc: Sutou Kouhei <[email protected]> > Subject: Re: Using Pyarrow and C++ API > > How did you install pyarrow, with pip? It has its own C++ libraries > bundled inside and so will surely conflict with the vcpkg libraries > > On Tue, Jan 7, 2020 at 11:18 AM Raúl Bocanegra Algarra > <[email protected]> wrote: > > > > Hi Wes, > > > > I just added "-DARROW_PYTHON=ON" on vcpkg/ports/arrow/portfile.cmake on the > > "vcpkg_configure_cmake" function call. > > I also created an improvement for this in its repo: > > https://github.com/microsoft/vcpkg/issues/9350 > > Also edited its vcpkg/triplets/x64-linux.cmake as they recommend on the > > docs: > > https://vcpkg.readthedocs.io/en/latest/users/triplets/#per-port-customization > > Just edited: > > set(VCPKG_LIBRARY_LINKAGE static) > > if(PORT MATCHES "arrow") > > set(VCPKG_LIBRARY_LINKAGE dynamic) > > endif() > > > > Thanks for your work. > > > > Raúl. > > ________________________________ > > From: Wes McKinney <[email protected]> > > Sent: Tuesday, January 7, 2020 6:06:21 PM > > To: [email protected] <[email protected]> > > Cc: Sutou Kouhei <[email protected]> > > Subject: Re: Using Pyarrow and C++ API > > > > It doesn't seem like vcpkg should have libarrow_python available based on > > > > https://github.com/microsoft/vcpkg/tree/master/ports/arrow > > > > How are you installing pyarrow? > > > > On Tue, Jan 7, 2020 at 10:14 AM Raúl Bocanegra Algarra > > <[email protected]> wrote: > > > > > > Hi Sutou, > > > > > > Thanks for your help. I tried that option but unfortunately arrow build > > > scripts can't find the libarrow installation from vcpkg it seems to be > > > using a custom findArrow.cmake script. Anyway I hacked the CMakeLists > > > from arrow in order to find vcpkg's arrow libraries and it worked, I run > > > ldd and the libraries are the ones from vcpkg. So now I have my Python > > > extension and my application linked against vcpkg's arrow and > > > arrow_python. But now it still doesn't work I am observing a couple of > > > weird issues. > > > 1.- In the python script that my embedded python interpreter runs I do: > > > "from statsmodels.tsa.stattools import adfuller as adf" but it gets stuck > > > there. So I removed that import. Then: > > > 2.- After removing the import now I have a SIGABRT and the following > > > message when my extensions calls "arrow::py::wrap_array": > > > "Fatal Python error: PyThreadState_Get: no current thread". > > > I did some Google and I only found a similar issue on Macs with different > > > python interpreters installed, but I am on an Ubuntu 18.4 with only > > > python 3.6.9 installed and I am using a venv for pyarrow. > > > I don't know what to do next. Did any of you experienced something > > > similar? > > > > > > Thanks! > > > > > > > > > Raúl Bocanegra Algarra > > > > > > Software Engineer > > > > > > M: + 34 617 83 64 45 - E: [email protected] > > > > > > > > > > > > ________________________________ > > > From: Sutou Kouhei <[email protected]> > > > Sent: Sunday, January 5, 2020 10:29 PM > > > To: [email protected] <[email protected]> > > > Subject: Re: Using Pyarrow and C++ API > > > > > > Hi, > > > > > > How about install pyarrow with "pip install --no-binary :all: pyarrow"? > > > Then you will be able to build your pyarrow with your > > > libarrow.so and libarrow_python.so. > > > > > > Thanks, > > > -- > > > kou > > > > > > In > > > > > > <am0pr01mb641703d6ba43cf50596dfd34e7...@am0pr01mb6417.eurprd01.prod.exchangelabs.com> > > > "Using Pyarrow and C++ API " on Sun, 5 Jan 2020 03:45:21 +0000, > > > Raúl Bocanegra Algarra <[email protected]> wrote: > > > > > > > Hi! > > > > > > > > I am trying to use pyarrow with arrow C++ API in an application that > > > > embeds a python3 interpreter and loads an extension module using > > > > pybind11. Documentation says C++ headers and libraries are bundled with > > > > pyarrow but I am having some segfaults when calling some API functions > > > > like the wrap/unwrap ones. I am calling import_pyarrow and also > > > > import_numpy but segfaults still happening. I feel the reason is that I > > > > compile and link with my own arrow and arrow_python libs built with > > > > vcpkg so my app links with those, but the extension module imported by > > > > the embedded python interpreter is loading the arrow_python from the > > > > site-packages folder where pip installed pyarrow, and that mismatch > > > > makes the segfault happen. So I was wondering if the correct approach > > > > for a situation like this with an embedded interpreter and an extension > > > > module that imports pyarrow is to use the headers and libs from the > > > > pyarrow installation removing the ones from vcpkg or if you know > > > > another option I haven't contemplated yet. > > > > > > > > Thanks for your work. > > > > > > > > Best regards, > > > > > > > > Raúl Bocanegra Algarra. C++ Software Engineer. > > > >
