Hi, > -DCMAKE_INSTALL_LIBDIR=lib \ > > this path is relative to ARROW_HOME, correct me if I am wrong?
It's relative to CMAKE_INSTALL_PREFIX not ARROW_HOME. > So with this source build, it doesn't necessarily copy files to /usr/lib ? You don't need to copy. BTW, do you still want to use custom CMAKE_INSTALL_LIBDIR? If you want to do, you can't use the ARROW_HOME environment variable. Again, we don't recommend that you use custom CMAKE_INSTALL_LIBDIR if you aren't a CMake expert. Thanks, -- kou In <CABC-QT=0Me4TQcCQbK7uqK1v-0=d0dy=hwycjjaeu58xtlp...@mail.gmail.com> "Re: PyArrow building from source issue" on Sat, 2 May 2020 19:01:14 -0400, Vibhatha Abeykoon <[email protected]> wrote: > One thing I noticed, > > -DCMAKE_INSTALL_LIBDIR=lib \ > > this path is relative to ARROW_HOME, correct me if I am wrong? > > So with this source build, it doesn't necessarily copy files to /usr/lib ? > > With Regards, > Vibhatha Abeykoon > > > On Sat, May 2, 2020 at 6:12 PM Vibhatha Abeykoon <[email protected]> wrote: > >> So far this is the updated steps, >> >> pwd => /home/vibhatha/sandbox/arrow/repos >> >> export PYTHON_EXEC=/home/vibhatha/sandbox/arrow/repos/ENVARROW/bin/python3 >> export LIB_DIR=$(pwd)/libs >> >> mkdir $LIB_DIR >> >> ## ENV CONFIGS >> >> export ARROW_HOME=export ARROW_HOME=$(pwd)/dist >> export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH >> >> ## MAKE BUILD DIRS >> >> mkdir arrow/cpp/build >> pushd arrow/cpp/build >> >> cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> -DCMAKE_INSTALL_LIBDIR=$LIB_DIR \ >> -DARROW_WITH_BZ2=OFF \ >> -DARROW_WITH_ZLIB=OFF \ >> -DARROW_WITH_ZSTD=OFF \ >> -DARROW_WITH_LZ4=OFF \ >> -DARROW_WITH_SNAPPY=OFF \ >> -DARROW_WITH_BROTLI=OFF \ >> -DARROW_PARQUET=OFF \ >> -DARROW_PYTHON=ON \ >> -DARROW_BUILD_TESTS=ON \ >> -DPYTHON_EXECUTABLE=$PYTHON_EXEC \ >> .. >> >> make -j4 >> make install >> >> popd >> >> export LD_LIBRARY_PATH=$LIB_DIR:$LD_LIBRARY_PATH >> >> pushd arrow/python >> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH="${LIB_DIR}"/cmake/arrow" >> python3 setup.py install >> >> >> *Getting an error * >> >> CMake Error at >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> (message): >> Could NOT find Arrow (missing: ARROW_LIB_DIR) (found version "0.16.0") >> Call Stack (most recent call first): >> >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> (_FPHSA_FAILURE_MESSAGE) >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake:420 >> (find_package_handle_standard_args) >> >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:46 >> (find_package) >> CMakeLists.txt:204 (find_package) >> >> >> Then, I added the following in FindArrow.cmake >> >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake >> >> set(ARROW_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs) >> >> Re-ran the script, >> >> *Getting Error* >> >> CMake Error at >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> (message): >> Could NOT find ArrowPython (missing: ARROW_PYTHON_LIB_DIR) (found version >> "0.16.0") >> Call Stack (most recent call first): >> >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> (_FPHSA_FAILURE_MESSAGE) >> >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:78 >> (find_package_handle_standard_args) >> CMakeLists.txt:204 (find_package) >> >> Added the following >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake >> >> set(ARROW_PYTHON_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs) >> >> >> *It seems a path I added could be wrong or a build flag is missing* >> >> -- Arrow version: 0.16.0 (HOME: /home/vibhatha/sandbox/arrow/repos/dist) >> -- Arrow SO and ABI version: 16 >> -- Arrow full SO version: 16.0.0 >> >> >> >> >> >> >> *-- Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND-- Found >> the Arrow core import library: -- Found the Arrow core static library: >> ARROW_static_lib-NOTFOUND-- Found the Arrow Python by HOME: >> /home/vibhatha/sandbox/arrow/repos/dist-- Found the Arrow Python shared >> library: ARROW_PYTHON_shared_lib-NOTFOUND-- Found the Arrow Python import >> library: -- Found the Arrow Python static library: >> ARROW_PYTHON_static_lib-NOTFOUND* >> -- Configuring done >> -- Generating done >> -- Build files have been written to: >> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8 >> -- Finished cmake for pyarrow >> -- Running cmake --build for pyarrow >> cmake --build . --config release -- >> [ 6%] Compiling Cython CXX source for _compute... >> [ 6%] Built target _compute_pyx >> Scanning dependencies of target _compute >> [ 13%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o >> [ 20%] Linking CXX shared module release/_ >> compute.cpython-38-x86_64-linux-gnu.so >> >> >> */usr/bin/ld: cannot find -larrow_shared/usr/bin/ld: cannot find >> -larrow_python_shared*collect2: error: ld returned 1 exit status >> make[2]: *** [CMakeFiles/_compute.dir/build.make:84: release/_ >> compute.cpython-38-x86_64-linux-gnu.so] Error 1 >> make[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/_compute.dir/all] Error 2 >> make: *** [Makefile:84: all] Error 2 >> error: command 'cmake' failed with exit status 2 >> >> The log also shows that the shared or static libraries have not been >> found, >> >> >> With Regards, >> Vibhatha Abeykoon >> >> >> On Fri, May 1, 2020 at 8:23 PM Vibhatha Abeykoon <[email protected]> >> wrote: >> >>> Thank you for the clarification. >>> >>> On Fri, May 1, 2020 at 8:10 PM Sutou Kouhei <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> It'll work. >>>> Note that the LD_LIBRARY_PATH is the environment variable at >>>> run time. You need to specify the correct ARROW_HOME or >>>> -DCMAKE_MODULE_PATH at build time too. >>>> >>>> >>>> Thanks, >>>> -- >>>> kou >>>> >>>> In <CABC-QT=u_+ae8lammhwfhjp+4wvvkj0y-z7styk0txtrjyz...@mail.gmail.com> >>>> "Re: PyArrow building from source issue" on Fri, 1 May 2020 18:13:54 >>>> -0400, >>>> Vibhatha Abeykoon <[email protected]> wrote: >>>> >>>> > I will elaborate a couple of reasons, >>>> > >>>> > When there are a couple of versions of Arrow, used for different >>>> projects >>>> > depending on various development choices, it is convenient for me to >>>> keep >>>> > them pointed towards a folder of my choice. >>>> > Then refer to it and continue the work. Correct me if I am wrong, what >>>> if I >>>> > point to this folder of my choice and add it to the LD_LIBRARY_PATH. >>>> > Will this cause issues? >>>> > >>>> > With Regards, >>>> > Vibhatha Abeykoon >>>> > >>>> > >>>> > On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon <[email protected]> >>>> wrote: >>>> > >>>> >> Okay, thank you for your response. >>>> >> >>>> >> With Regards, >>>> >> Vibhatha Abeykoon >>>> >> >>>> >> >>>> >> On Fri, May 1, 2020 at 5:17 PM Sutou Kouhei <[email protected]> >>>> wrote: >>>> >> >>>> >>> Hi, >>>> >>> >>>> >>> > I used these settings, I still want the libs to be in a custom >>>> >>> directory, >>>> >>> > not in lib/ >>>> >>> >>>> >>> Why? We don't recommend it if you don't a CMake expert. >>>> >>> >>>> >>> You can't do this if you use ARROW_HOME environment >>>> >>> variable. >>>> >>> >>>> >>> You may able to do this by removing ARROW_HOME environment >>>> >>> variable and adding >>>> >>> >>>> >>> >>>> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow" >>>> >>> environment variable or something. >>>> >>> >>>> >>> >>>> >>> Thanks, >>>> >>> -- >>>> >>> kou >>>> >>> >>>> >>> In <CABC-QTmsekZwjWG4X= >>>> [email protected]> >>>> >>> "Re: PyArrow building from source issue" on Fri, 1 May 2020 >>>> 15:05:51 >>>> >>> -0400, >>>> >>> Vibhatha Abeykoon <[email protected]> wrote: >>>> >>> >>>> >>> > export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >>>> >>> > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH >>>> >>> > >>>> >>> > export PYARROW_WITH_PARQUET=1 >>>> >>> > export PYARROW_WITH_PYTHON=1 >>>> >>> > export PYARROW_WITH_BUILD_TESTS=1 >>>> >>> > >>>> >>> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >>>> >>> > >>>> >>> > >>>> >>> >>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >>>> >>> > \ >>>> >>> > -DARROW_WITH_BZ2=OFF \ >>>> >>> > -DARROW_WITH_ZLIB=OFF \ >>>> >>> > -DARROW_WITH_ZSTD=OFF \ >>>> >>> > -DARROW_WITH_LZ4=OFF \ >>>> >>> > -DARROW_WITH_SNAPPY=OFF \ >>>> >>> > -DARROW_WITH_BROTLI=OFF \ >>>> >>> > -DARROW_PARQUET=ON \ >>>> >>> > -DARROW_PYTHON=ON \ >>>> >>> > -DARROW_BUILD_TESTS=ON \ >>>> >>> > >>>> >>> > >>>> >>> >>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >>>> >>> > \ >>>> >>> > .. >>>> >>> > >>>> >>> > >>>> >>> > I used these settings, I still want the libs to be in a custom >>>> >>> directory, >>>> >>> > not in lib/ >>>> >>> > >>>> >>> > Does it make things not work? >>>> >>> > >>>> >>> > Now I get the following error, >>>> >>> > >>>> >>> > python setup.py build_ext --inplace >>>> >>> > WARNING: The wheel package is not available. >>>> >>> > running build_ext >>>> >>> > -- Running cmake for pyarrow >>>> >>> > cmake >>>> >>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python >>>> >>> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >>>> >>> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >>>> >>> > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >>>> >>> > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off >>>> >>> -DPYARROW_BUILD_HDFS=off >>>> >>> > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off >>>> >>> > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off >>>> >>> > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on >>>> >>> > -DCMAKE_BUILD_TYPE=release >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python >>>> >>> > -- System processor: x86_64 >>>> >>> > -- Arrow build warning level: PRODUCTION >>>> >>> > Using ld linker >>>> >>> > Configured for RELEASE build (set with cmake >>>> >>> > -DCMAKE_BUILD_TYPE={release,debug,...}) >>>> >>> > -- Build Type: RELEASE >>>> >>> > -- Build output directory: >>>> >>> > >>>> >>> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >>>> >>> > -- Arrow version: 0.18.0 (HOME: >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist) >>>> >>> > -- Arrow SO and ABI version: 18 >>>> >>> > -- Arrow full SO version: 18.0.0 >>>> >>> > -- Found the Arrow core shared library: >>>> >>> > >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >>>> >>> > -- Found the Arrow core import library: >>>> >>> > >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >>>> >>> > -- Found the Arrow core static library: >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >>>> >>> > -- Found the Arrow Python by HOME: >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >>>> >>> > -- Found the Arrow Python shared library: >>>> >>> > >>>> >>> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so >>>> >>> > -- Found the Arrow Python import library: >>>> >>> > >>>> >>> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so >>>> >>> > -- Found the Arrow Python static library: >>>> >>> > >>>> >>> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a >>>> >>> > CMake Error at >>>> >>> > >>>> >>> >>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >>>> >>> > (message): >>>> >>> > Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found version >>>> >>> "1.5.1") >>>> >>> > Call Stack (most recent call first): >>>> >>> > >>>> >>> > >>>> >>> >>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >>>> >>> > (_FPHSA_FAILURE_MESSAGE) >>>> >>> > cmake_modules/FindParquet.cmake:116 >>>> >>> (find_package_handle_standard_args) >>>> >>> > CMakeLists.txt:426 (find_package) >>>> >>> > >>>> >>> > Does Parquet need to be installed separately? >>>> >>> > >>>> >>> > With Regards, >>>> >>> > Vibhatha Abeykoon, >>>> >>> > Research Assistant, >>>> >>> > Intelligent Systems Engineering, >>>> >>> > Indiana University Bloomington, >>>> >>> > Cell : +1-812-955-1394 >>>> >>> > Web: https://www.vibhatha.org >>>> >>> > <https://www.linkedin.com/in/vibhathaabeykoon/> >>>> >>> > >>>> >>> > >>>> >>> > On Fri, May 1, 2020 at 1:55 PM Wes McKinney <[email protected]> >>>> >>> wrote: >>>> >>> > >>>> >>> >> Try this instead >>>> >>> >> >>>> >>> >> export >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >>>> >>> >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH >>>> >>> >> >>>> >>> >> Make sure to use >>>> >>> >> >>>> >>> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib >>>> >>> >> >>>> >>> >> I renamed from "arrowmylibs" to "dist" so that it's less confusing >>>> >>> >> what that directory represents (it doesn't contains the libs >>>> >>> >> directory, but rather the directories include/, lib/, etc.) >>>> >>> >> >>>> >>> >> >>>> >>> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon < >>>> [email protected]> >>>> >>> >> wrote: >>>> >>> >> > >>>> >>> >> > I didn't clearly mention the config I used, >>>> >>> >> > >>>> >>> >> > export >>>> >>> >> >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >>>> >>> >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH >>>> >>> >> > >>>> >>> >> > Before starting I added these ENV vars. >>>> >>> >> > >>>> >>> >> > My objective is to get all the libs inside my build setting. I >>>> use >>>> >>> arrow >>>> >>> >> as one of the core libraries in my project. >>>> >>> >> > I have to build it in the following way, >>>> >>> >> > >>>> >>> >> > 1. Arrow clone from git >>>> >>> >> > 2. Arrow CPP built >>>> >>> >> > 3. Using arrow CPP I develop custom functions upon loaded data >>>> >>> >> > 4. Integrate Cython APIs for these custom functions >>>> >>> >> > 5. Use Pyarrow Cython to provide more functionality from Python >>>> end >>>> >>> to >>>> >>> >> Cython to my CPP lib which uses Arrow. >>>> >>> >> > >>>> >>> >> > This is the build that I am trying to formulate. So I have to >>>> keep >>>> >>> those >>>> >>> >> libs there, and the idea is to build python from >>>> >>> >> > the same cloned source. The challenge was to keep all the >>>> shared libs >>>> >>> >> from arrow, my-cython libs to point in the right >>>> >>> >> > direction. Is this a clear description? >>>> >>> >> > >>>> >>> >> > With Regards, >>>> >>> >> > Vibhatha Abeykoon, >>>> >>> >> > Research Assistant, >>>> >>> >> > Intelligent Systems Engineering, >>>> >>> >> > Indiana University Bloomington, >>>> >>> >> > Cell : +1-812-955-1394 >>>> >>> >> > Web: https://www.vibhatha.org >>>> >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> >>> >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney < >>>> [email protected]> >>>> >>> wrote: >>>> >>> >> >> >>>> >>> >> >> This part doesn't look correct >>>> >>> >> >> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >>>> >>> >> >> >>>> >>> >> >>>> >>> >>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >>>> >>> >> >> >>>> >>> >> >> The usual incantation is >>>> >>> >> >> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >>>> >>> >> >> -DCMAKE_INSTALL_LIBDIR=lib >>>> >>> >> >> >>>> >>> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that some >>>> systems >>>> >>> >> >> will install libraries in lib32 or lib64 instead of just lib >>>> >>> >> >> >>>> >>> >> >> >>>> >>> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon < >>>> >>> [email protected]> >>>> >>> >> wrote: >>>> >>> >> >> > >>>> >>> >> >> > Hi Neal, >>>> >>> >> >> > >>>> >>> >> >> > Yes, I added the flag. But I installed my libs not to >>>> /usr/lib, >>>> >>> but >>>> >>> >> to a different folder. >>>> >>> >> >> > >>>> >>> >> >> > This is the way I built arrow C++, >>>> >>> >> >> > >>>> >>> >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >>>> >>> >> >> > >>>> >>> >> >>>> >>> >>>> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >>>> >>> >> \ >>>> >>> >> >> > -DARROW_WITH_BZ2=OFF \ >>>> >>> >> >> > -DARROW_WITH_ZLIB=OFF \ >>>> >>> >> >> > -DARROW_WITH_ZSTD=OFF \ >>>> >>> >> >> > -DARROW_WITH_LZ4=OFF \ >>>> >>> >> >> > -DARROW_WITH_SNAPPY=OFF \ >>>> >>> >> >> > -DARROW_WITH_BROTLI=OFF \ >>>> >>> >> >> > -DARROW_PARQUET=ON \ >>>> >>> >> >> > -DARROW_PYTHON=ON \ >>>> >>> >> >> > -DARROW_BUILD_TESTS=ON \ >>>> >>> >> >> > >>>> >>> >> >>>> >>> >>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >>>> >>> >> \ >>>> >>> >> >> > .. >>>> >>> >> >> > >>>> >>> >> >> > >>>> >>> >> >> > >>>> >>> >> >> > >>>> >>> >> >> > With Regards, >>>> >>> >> >> > Vibhatha Abeykoon, >>>> >>> >> >> > Research Assistant, >>>> >>> >> >> > Intelligent Systems Engineering, >>>> >>> >> >> > Indiana University Bloomington, >>>> >>> >> >> > Cell : +1-812-955-1394 >>>> >>> >> >> > Web: https://www.vibhatha.org >>>> >>> >> >> > >>>> >>> >> >> > >>>> >>> >> >> > >>>> >>> >> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson < >>>> >>> >> [email protected]> wrote: >>>> >>> >> >> >> >>>> >>> >> >> >> Hi Vibhatha, >>>> >>> >> >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON? >>>> >>> >> >> >> >>>> >>> >> >> >> Neal >>>> >>> >> >> >> >>>> >>> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon < >>>> >>> [email protected]> >>>> >>> >> wrote: >>>> >>> >> >> >>> >>>> >>> >> >> >>> Hi, >>>> >>> >> >> >>> >>>> >>> >> >> >>> I am trying to integrate Arrow with an application that I >>>> am >>>> >>> >> developing. Here I build Arrow from the source (CPP) and use the >>>> API to >>>> >>> >> develop some custom functions to do a scientific calculation >>>> after data >>>> >>> >> loaded with Arrow table API. On top of this, I develop a Cython >>>> API to >>>> >>> >> design a python API. >>>> >>> >> >> >>> >>>> >>> >> >> >>> In the current stage, I have a new necessity where I need >>>> to >>>> >>> >> consume Arrow Cython API for my code. >>>> >>> >> >> >>> >>>> >>> >> >> >>> Here It was hard to link the build libarrow.so.16 with the >>>> >>> >> libarrow_python.so.16 from the installed pyarrow (separately from >>>> pip). >>>> >>> >> What I realised was everything has to be built from the same >>>> source, so >>>> >>> >> that I can install pyarrow from the source in my virtual >>>> environment. >>>> >>> >> >> >>> >>>> >>> >> >> >>> Before going through deeper things, I started by just >>>> building >>>> >>> from >>>> >>> >> source (CPP) and then moving towards installing pyarrow from the >>>> >>> source. >>>> >>> >> >> >>> >>>> >>> >> >> >>> I tried to follow the guideline form here, >>>> >>> >> >> >>> >>>> >>> >> >> >>> https://arrow.apache.org/docs/developers/python.html, >>>> >>> >> >> >>> >>>> >>> >> >> >>> But when I found issues in the python build, I followed >>>> this >>>> >>> source, >>>> >>> >> >> >>> (but still, I used the clone from the master, not a >>>> released >>>> >>> >> version) >>>> >>> >> >> >>> >>>> >>> >> >> >>> >>>> >>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 >>>> >>> >> >> >>> >>>> >>> >> >> >>> My environmental variables are as follows, >>>> >>> >> >> >>> >>>> >>> >> >> >>> python3 setup.py build_ext --inplace >>>> >>> >> >> >>> running build_ext >>>> >>> >> >> >>> -- Running cmake for pyarrow >>>> >>> >> >> >>> cmake >>>> >>> >> >>>> >>> >>>> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >>>> >>> >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >>>> >>> >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >>>> >>> >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >>>> >>> >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off >>>> >>> -DPYARROW_BUILD_HDFS=off >>>> >>> >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off >>>> >>> >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off >>>> >>> >> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on >>>> >>> >> -DCMAKE_BUILD_TYPE=release >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python >>>> >>> >> >> >>> -- System processor: x86_64 >>>> >>> >> >> >>> -- Arrow build warning level: PRODUCTION >>>> >>> >> >> >>> Using ld linker >>>> >>> >> >> >>> Configured for RELEASE build (set with cmake >>>> >>> >> -DCMAKE_BUILD_TYPE={release,debug,...}) >>>> >>> >> >> >>> -- Build Type: RELEASE >>>> >>> >> >> >>> -- Build output directory: >>>> >>> >> >>>> >>> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >>>> >>> >> >> >>> -- Arrow version: 0.18.0 (HOME: >>>> >>> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) >>>> >>> >> >> >>> -- Arrow SO and ABI version: 18 >>>> >>> >> >> >>> -- Arrow full SO version: 18.0.0 >>>> >>> >> >> >>> -- Found the Arrow core shared library: >>>> >>> >> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >>>> >>> >> >> >>> -- Found the Arrow core import library: >>>> >>> >> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >>>> >>> >> >> >>> -- Found the Arrow core static library: >>>> >>> >> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >>>> >>> >> >> >>> CMake Error at >>>> >>> >> >>>> >>> >>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >>>> >>> >> (message): >>>> >>> >> >> >>> Could NOT find ArrowPython (missing: >>>> ARROW_PYTHON_INCLUDE_DIR) >>>> >>> >> (found >>>> >>> >> >> >>> version "0.18.0") >>>> >>> >> >> >>> Call Stack (most recent call first): >>>> >>> >> >> >>> >>>> >>> >> >>>> >>> >>>> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >>>> >>> >> (_FPHSA_FAILURE_MESSAGE) >>>> >>> >> >> >>> cmake_modules/FindArrowPython.cmake:76 >>>> >>> >> (find_package_handle_standard_args) >>>> >>> >> >> >>> CMakeLists.txt:210 (find_package) >>>> >>> >> >> >>> >>>> >>> >> >> >>> >>>> >>> >> >> >>> -- Configuring incomplete, errors occurred! >>>> >>> >> >> >>> See also >>>> >>> >> >>>> >>> >>>> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". >>>> >>> >> >> >>> error: command 'cmake' failed with exit status 1 >>>> >>> >> >> >>> >>>> >>> >> >> >>> Maybe I am missing some step and I am not quite sure what >>>> is the >>>> >>> >> issue. >>>> >>> >> >> >>> >>>> >>> >> >> >>> Any pointers to solve this issue? >>>> >>> >> >> >>> >>>> >>> >> >> >>> With Regards, >>>> >>> >> >> >>> Vibhatha >>>> >>> >> >> >>> >>>> >>> >> >> >>> >>>> >>> >> >>>> >>> >>>> >> >>>> >>> -- >>> Vibhatha Abeykoon >>> >>
