I didn't clearly mention the config I used, export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH
Before starting I added these ENV vars. My objective is to get all the libs inside my build setting. I use arrow as one of the core libraries in my project. I have to build it in the following way, 1. Arrow clone from git 2. Arrow CPP built 3. Using arrow CPP I develop custom functions upon loaded data 4. Integrate Cython APIs for these custom functions 5. Use Pyarrow Cython to provide more functionality from Python end to Cython to my CPP lib which uses Arrow. This is the build that I am trying to formulate. So I have to keep those libs there, and the idea is to build python from the same cloned source. The challenge was to keep all the shared libs from arrow, my-cython libs to point in the right direction. Is this a clear description? With Regards, Vibhatha Abeykoon, Research Assistant, Intelligent Systems Engineering, Indiana University Bloomington, Cell : +1-812-955-1394 Web: https://www.vibhatha.org <https://www.linkedin.com/in/vibhathaabeykoon/> On Fri, May 1, 2020 at 1:28 PM Wes McKinney <[email protected]> wrote: > This part doesn't look correct > > -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > > The usual incantation is > > -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > -DCMAKE_INSTALL_LIBDIR=lib > > The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that some systems > will install libraries in lib32 or lib64 instead of just lib > > > On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon <[email protected]> > wrote: > > > > Hi Neal, > > > > Yes, I added the flag. But I installed my libs not to /usr/lib, but to a > different folder. > > > > This is the way I built arrow C++, > > > > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > > > > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > \ > > -DARROW_WITH_BZ2=OFF \ > > -DARROW_WITH_ZLIB=OFF \ > > -DARROW_WITH_ZSTD=OFF \ > > -DARROW_WITH_LZ4=OFF \ > > -DARROW_WITH_SNAPPY=OFF \ > > -DARROW_WITH_BROTLI=OFF \ > > -DARROW_PARQUET=ON \ > > -DARROW_PYTHON=ON \ > > -DARROW_BUILD_TESTS=ON \ > > > > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > \ > > .. > > > > > > > > > > With Regards, > > Vibhatha Abeykoon, > > Research Assistant, > > Intelligent Systems Engineering, > > Indiana University Bloomington, > > Cell : +1-812-955-1394 > > Web: https://www.vibhatha.org > > > > > > > > On Fri, May 1, 2020 at 11:29 AM Neal Richardson < > [email protected]> wrote: > >> > >> Hi Vibhatha, > >> Did you build Arrow C++ with -DARROW_PYTHON=ON? > >> > >> Neal > >> > >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon <[email protected]> > wrote: > >>> > >>> Hi, > >>> > >>> I am trying to integrate Arrow with an application that I am > developing. Here I build Arrow from the source (CPP) and use the API to > develop some custom functions to do a scientific calculation after data > loaded with Arrow table API. On top of this, I develop a Cython API to > design a python API. > >>> > >>> In the current stage, I have a new necessity where I need to consume > Arrow Cython API for my code. > >>> > >>> Here It was hard to link the build libarrow.so.16 with the > libarrow_python.so.16 from the installed pyarrow (separately from pip). > What I realised was everything has to be built from the same source, so > that I can install pyarrow from the source in my virtual environment. > >>> > >>> Before going through deeper things, I started by just building from > source (CPP) and then moving towards installing pyarrow from the source. > >>> > >>> I tried to follow the guideline form here, > >>> > >>> https://arrow.apache.org/docs/developers/python.html, > >>> > >>> But when I found issues in the python build, I followed this source, > >>> (but still, I used the clone from the master, not a released version) > >>> > >>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 > >>> > >>> My environmental variables are as follows, > >>> > >>> python3 setup.py build_ext --inplace > >>> running build_ext > >>> -- Running cmake for pyarrow > >>> cmake > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on > -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python > >>> -- System processor: x86_64 > >>> -- Arrow build warning level: PRODUCTION > >>> Using ld linker > >>> Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > >>> -- Build Type: RELEASE > >>> -- Build output directory: > /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release > >>> -- Arrow version: 0.18.0 (HOME: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) > >>> -- Arrow SO and ABI version: 18 > >>> -- Arrow full SO version: 18.0.0 > >>> -- Found the Arrow core shared library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >>> -- Found the Arrow core import library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >>> -- Found the Arrow core static library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a > >>> CMake Error at > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > (message): > >>> Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found > >>> version "0.18.0") > >>> Call Stack (most recent call first): > >>> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > (_FPHSA_FAILURE_MESSAGE) > >>> cmake_modules/FindArrowPython.cmake:76 > (find_package_handle_standard_args) > >>> CMakeLists.txt:210 (find_package) > >>> > >>> > >>> -- Configuring incomplete, errors occurred! > >>> See also > "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". > >>> error: command 'cmake' failed with exit status 1 > >>> > >>> Maybe I am missing some step and I am not quite sure what is the issue. > >>> > >>> Any pointers to solve this issue? > >>> > >>> With Regards, > >>> Vibhatha > >>> > >>> >
