Try this instead export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH
Make sure to use -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib I renamed from "arrowmylibs" to "dist" so that it's less confusing what that directory represents (it doesn't contains the libs directory, but rather the directories include/, lib/, etc.) On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon <[email protected]> wrote: > > I didn't clearly mention the config I used, > > export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH > > Before starting I added these ENV vars. > > My objective is to get all the libs inside my build setting. I use arrow as > one of the core libraries in my project. > I have to build it in the following way, > > 1. Arrow clone from git > 2. Arrow CPP built > 3. Using arrow CPP I develop custom functions upon loaded data > 4. Integrate Cython APIs for these custom functions > 5. Use Pyarrow Cython to provide more functionality from Python end to Cython > to my CPP lib which uses Arrow. > > This is the build that I am trying to formulate. So I have to keep those libs > there, and the idea is to build python from > the same cloned source. The challenge was to keep all the shared libs from > arrow, my-cython libs to point in the right > direction. Is this a clear description? > > With Regards, > Vibhatha Abeykoon, > Research Assistant, > Intelligent Systems Engineering, > Indiana University Bloomington, > Cell : +1-812-955-1394 > Web: https://www.vibhatha.org > > > > On Fri, May 1, 2020 at 1:28 PM Wes McKinney <[email protected]> wrote: >> >> This part doesn't look correct >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> >> The usual incantation is >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> -DCMAKE_INSTALL_LIBDIR=lib >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that some systems >> will install libraries in lib32 or lib64 instead of just lib >> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon <[email protected]> wrote: >> > >> > Hi Neal, >> > >> > Yes, I added the flag. But I installed my libs not to /usr/lib, but to a >> > different folder. >> > >> > This is the way I built arrow C++, >> > >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> > >> > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> > \ >> > -DARROW_WITH_BZ2=OFF \ >> > -DARROW_WITH_ZLIB=OFF \ >> > -DARROW_WITH_ZSTD=OFF \ >> > -DARROW_WITH_LZ4=OFF \ >> > -DARROW_WITH_SNAPPY=OFF \ >> > -DARROW_WITH_BROTLI=OFF \ >> > -DARROW_PARQUET=ON \ >> > -DARROW_PYTHON=ON \ >> > -DARROW_BUILD_TESTS=ON \ >> > >> > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> > \ >> > .. >> > >> > >> > >> > >> > With Regards, >> > Vibhatha Abeykoon, >> > Research Assistant, >> > Intelligent Systems Engineering, >> > Indiana University Bloomington, >> > Cell : +1-812-955-1394 >> > Web: https://www.vibhatha.org >> > >> > >> > >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson >> > <[email protected]> wrote: >> >> >> >> Hi Vibhatha, >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON? >> >> >> >> Neal >> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon <[email protected]> >> >> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I am trying to integrate Arrow with an application that I am developing. >> >>> Here I build Arrow from the source (CPP) and use the API to develop some >> >>> custom functions to do a scientific calculation after data loaded with >> >>> Arrow table API. On top of this, I develop a Cython API to design a >> >>> python API. >> >>> >> >>> In the current stage, I have a new necessity where I need to consume >> >>> Arrow Cython API for my code. >> >>> >> >>> Here It was hard to link the build libarrow.so.16 with the >> >>> libarrow_python.so.16 from the installed pyarrow (separately from pip). >> >>> What I realised was everything has to be built from the same source, so >> >>> that I can install pyarrow from the source in my virtual environment. >> >>> >> >>> Before going through deeper things, I started by just building from >> >>> source (CPP) and then moving towards installing pyarrow from the source. >> >>> >> >>> I tried to follow the guideline form here, >> >>> >> >>> https://arrow.apache.org/docs/developers/python.html, >> >>> >> >>> But when I found issues in the python build, I followed this source, >> >>> (but still, I used the clone from the master, not a released version) >> >>> >> >>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 >> >>> >> >>> My environmental variables are as follows, >> >>> >> >>> python3 setup.py build_ext --inplace >> >>> running build_ext >> >>> -- Running cmake for pyarrow >> >>> cmake >> >>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> >>> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >> >>> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >> >>> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >> >>> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off >> >>> -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off >> >>> -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off >> >>> -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on >> >>> -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python >> >>> -- System processor: x86_64 >> >>> -- Arrow build warning level: PRODUCTION >> >>> Using ld linker >> >>> Configured for RELEASE build (set with cmake >> >>> -DCMAKE_BUILD_TYPE={release,debug,...}) >> >>> -- Build Type: RELEASE >> >>> -- Build output directory: >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >> >>> -- Arrow version: 0.18.0 (HOME: >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) >> >>> -- Arrow SO and ABI version: 18 >> >>> -- Arrow full SO version: 18.0.0 >> >>> -- Found the Arrow core shared library: >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >>> -- Found the Arrow core import library: >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >>> -- Found the Arrow core static library: >> >>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >> >>> CMake Error at >> >>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> >>> (message): >> >>> Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found >> >>> version "0.18.0") >> >>> Call Stack (most recent call first): >> >>> >> >>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> >>> (_FPHSA_FAILURE_MESSAGE) >> >>> cmake_modules/FindArrowPython.cmake:76 >> >>> (find_package_handle_standard_args) >> >>> CMakeLists.txt:210 (find_package) >> >>> >> >>> >> >>> -- Configuring incomplete, errors occurred! >> >>> See also >> >>> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". >> >>> error: command 'cmake' failed with exit status 1 >> >>> >> >>> Maybe I am missing some step and I am not quite sure what is the issue. >> >>> >> >>> Any pointers to solve this issue? >> >>> >> >>> With Regards, >> >>> Vibhatha >> >>> >> >>>
