This explains clearly. On Sat, May 2, 2020 at 9:12 PM Sutou Kouhei <[email protected]> wrote:
> Hi, > > Is the another third party library built with the Apache > Arrow binary you built? If it's true, it works well. > > If the another third party library is built with another > Apache Arrow binary, it may not work. If another Apache > Arrow binary and your Apache Arrow binary use different > build options, it doesn't work. If another Apache Arrow > binary and your Apache Arrow binary use different revision, > it may not work. > > > Thanks, > -- > kou > > In <cabc-qtk0hqancwghqea7dx9q52xvgt0ncvfmoq0eckwdygv...@mail.gmail.com> > "Re: PyArrow building from source issue" on Sat, 2 May 2020 20:42:08 > -0400, > Vibhatha Abeykoon <[email protected]> wrote: > > > I understand. My main question is the following, building arrow cpp and > > pyarrow from the same source would allow seamless integration with > another > > third party library which uses the arrow cpp and cython API? > > > > On Sat, May 2, 2020 at 8:18 PM Sutou Kouhei <[email protected]> wrote: > > > >> Hi, > >> > >> > -DCMAKE_INSTALL_LIBDIR=lib \ > >> > > >> > this path is relative to ARROW_HOME, correct me if I am wrong? > >> > >> It's relative to CMAKE_INSTALL_PREFIX not ARROW_HOME. > >> > >> > So with this source build, it doesn't necessarily copy files to > /usr/lib > >> ? > >> > >> You don't need to copy. > >> > >> > >> BTW, do you still want to use custom CMAKE_INSTALL_LIBDIR? > >> If you want to do, you can't use the ARROW_HOME environment > >> variable. > >> > >> Again, we don't recommend that you use custom > >> CMAKE_INSTALL_LIBDIR if you aren't a CMake expert. > >> > >> > >> Thanks, > >> -- > >> kou > >> > >> In <CABC-QT=0Me4TQcCQbK7uqK1v-0=d0dy=hwycjjaeu58xtlp...@mail.gmail.com> > >> "Re: PyArrow building from source issue" on Sat, 2 May 2020 19:01:14 > >> -0400, > >> Vibhatha Abeykoon <[email protected]> wrote: > >> > >> > One thing I noticed, > >> > > >> > -DCMAKE_INSTALL_LIBDIR=lib \ > >> > > >> > this path is relative to ARROW_HOME, correct me if I am wrong? > >> > > >> > So with this source build, it doesn't necessarily copy files to > /usr/lib > >> ? > >> > > >> > With Regards, > >> > Vibhatha Abeykoon > >> > > >> > > >> > On Sat, May 2, 2020 at 6:12 PM Vibhatha Abeykoon <[email protected]> > >> wrote: > >> > > >> >> So far this is the updated steps, > >> >> > >> >> pwd => /home/vibhatha/sandbox/arrow/repos > >> >> > >> >> export > >> PYTHON_EXEC=/home/vibhatha/sandbox/arrow/repos/ENVARROW/bin/python3 > >> >> export LIB_DIR=$(pwd)/libs > >> >> > >> >> mkdir $LIB_DIR > >> >> > >> >> ## ENV CONFIGS > >> >> > >> >> export ARROW_HOME=export ARROW_HOME=$(pwd)/dist > >> >> export LD_LIBRARY_PATH=$(pwd)/dist/lib:$LD_LIBRARY_PATH > >> >> > >> >> ## MAKE BUILD DIRS > >> >> > >> >> mkdir arrow/cpp/build > >> >> pushd arrow/cpp/build > >> >> > >> >> cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > >> >> -DCMAKE_INSTALL_LIBDIR=$LIB_DIR \ > >> >> -DARROW_WITH_BZ2=OFF \ > >> >> -DARROW_WITH_ZLIB=OFF \ > >> >> -DARROW_WITH_ZSTD=OFF \ > >> >> -DARROW_WITH_LZ4=OFF \ > >> >> -DARROW_WITH_SNAPPY=OFF \ > >> >> -DARROW_WITH_BROTLI=OFF \ > >> >> -DARROW_PARQUET=OFF \ > >> >> -DARROW_PYTHON=ON \ > >> >> -DARROW_BUILD_TESTS=ON \ > >> >> -DPYTHON_EXECUTABLE=$PYTHON_EXEC \ > >> >> .. > >> >> > >> >> make -j4 > >> >> make install > >> >> > >> >> popd > >> >> > >> >> export LD_LIBRARY_PATH=$LIB_DIR:$LD_LIBRARY_PATH > >> >> > >> >> pushd arrow/python > >> >> PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH="${LIB_DIR}"/cmake/arrow" > >> >> python3 setup.py install > >> >> > >> >> > >> >> *Getting an error * > >> >> > >> >> CMake Error at > >> >> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > >> >> (message): > >> >> Could NOT find Arrow (missing: ARROW_LIB_DIR) (found version > "0.16.0") > >> >> Call Stack (most recent call first): > >> >> > >> >> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > >> >> (_FPHSA_FAILURE_MESSAGE) > >> >> > >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake:420 > >> >> (find_package_handle_standard_args) > >> >> > >> >> > >> > /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:46 > >> >> (find_package) > >> >> CMakeLists.txt:204 (find_package) > >> >> > >> >> > >> >> Then, I added the following in FindArrow.cmake > >> >> > >> >> /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrow.cmake > >> >> > >> >> set(ARROW_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs) > >> >> > >> >> Re-ran the script, > >> >> > >> >> *Getting Error* > >> >> > >> >> CMake Error at > >> >> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > >> >> (message): > >> >> Could NOT find ArrowPython (missing: ARROW_PYTHON_LIB_DIR) (found > >> version > >> >> "0.16.0") > >> >> Call Stack (most recent call first): > >> >> > >> >> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > >> >> (_FPHSA_FAILURE_MESSAGE) > >> >> > >> >> > >> > /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake:78 > >> >> (find_package_handle_standard_args) > >> >> CMakeLists.txt:204 (find_package) > >> >> > >> >> Added the following > >> >> > >> > /home/vibhatha/sandbox/arrow/repos/libs/cmake/arrow/FindArrowPython.cmake > >> >> > >> >> set(ARROW_PYTHON_LIB_DIR /home/vibhatha/sandbox/arrow/repos/libs) > >> >> > >> >> > >> >> *It seems a path I added could be wrong or a build flag is missing* > >> >> > >> >> -- Arrow version: 0.16.0 (HOME: > /home/vibhatha/sandbox/arrow/repos/dist) > >> >> -- Arrow SO and ABI version: 16 > >> >> -- Arrow full SO version: 16.0.0 > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> *-- Found the Arrow core shared library: ARROW_shared_lib-NOTFOUND-- > >> Found > >> >> the Arrow core import library: -- Found the Arrow core static > library: > >> >> ARROW_static_lib-NOTFOUND-- Found the Arrow Python by HOME: > >> >> /home/vibhatha/sandbox/arrow/repos/dist-- Found the Arrow Python > shared > >> >> library: ARROW_PYTHON_shared_lib-NOTFOUND-- Found the Arrow Python > >> import > >> >> library: -- Found the Arrow Python static library: > >> >> ARROW_PYTHON_static_lib-NOTFOUND* > >> >> -- Configuring done > >> >> -- Generating done > >> >> -- Build files have been written to: > >> >> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8 > >> >> -- Finished cmake for pyarrow > >> >> -- Running cmake --build for pyarrow > >> >> cmake --build . --config release -- > >> >> [ 6%] Compiling Cython CXX source for _compute... > >> >> [ 6%] Built target _compute_pyx > >> >> Scanning dependencies of target _compute > >> >> [ 13%] Building CXX object CMakeFiles/_compute.dir/_compute.cpp.o > >> >> [ 20%] Linking CXX shared module release/_ > >> >> compute.cpython-38-x86_64-linux-gnu.so > >> >> > >> >> > >> >> */usr/bin/ld: cannot find -larrow_shared/usr/bin/ld: cannot find > >> >> -larrow_python_shared*collect2: error: ld returned 1 exit status > >> >> make[2]: *** [CMakeFiles/_compute.dir/build.make:84: release/_ > >> >> compute.cpython-38-x86_64-linux-gnu.so] Error 1 > >> >> make[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/_compute.dir/all] > >> Error 2 > >> >> make: *** [Makefile:84: all] Error 2 > >> >> error: command 'cmake' failed with exit status 2 > >> >> > >> >> The log also shows that the shared or static libraries have not been > >> >> found, > >> >> > >> >> > >> >> With Regards, > >> >> Vibhatha Abeykoon > >> >> > >> >> > >> >> On Fri, May 1, 2020 at 8:23 PM Vibhatha Abeykoon <[email protected] > > > >> >> wrote: > >> >> > >> >>> Thank you for the clarification. > >> >>> > >> >>> On Fri, May 1, 2020 at 8:10 PM Sutou Kouhei <[email protected]> > >> wrote: > >> >>> > >> >>>> Hi, > >> >>>> > >> >>>> It'll work. > >> >>>> Note that the LD_LIBRARY_PATH is the environment variable at > >> >>>> run time. You need to specify the correct ARROW_HOME or > >> >>>> -DCMAKE_MODULE_PATH at build time too. > >> >>>> > >> >>>> > >> >>>> Thanks, > >> >>>> -- > >> >>>> kou > >> >>>> > >> >>>> In <CABC-QT= > >> [email protected]> > >> >>>> "Re: PyArrow building from source issue" on Fri, 1 May 2020 > 18:13:54 > >> >>>> -0400, > >> >>>> Vibhatha Abeykoon <[email protected]> wrote: > >> >>>> > >> >>>> > I will elaborate a couple of reasons, > >> >>>> > > >> >>>> > When there are a couple of versions of Arrow, used for different > >> >>>> projects > >> >>>> > depending on various development choices, it is convenient for > me to > >> >>>> keep > >> >>>> > them pointed towards a folder of my choice. > >> >>>> > Then refer to it and continue the work. Correct me if I am wrong, > >> what > >> >>>> if I > >> >>>> > point to this folder of my choice and add it to the > LD_LIBRARY_PATH. > >> >>>> > Will this cause issues? > >> >>>> > > >> >>>> > With Regards, > >> >>>> > Vibhatha Abeykoon > >> >>>> > > >> >>>> > > >> >>>> > On Fri, May 1, 2020 at 6:09 PM Vibhatha Abeykoon < > >> [email protected]> > >> >>>> wrote: > >> >>>> > > >> >>>> >> Okay, thank you for your response. > >> >>>> >> > >> >>>> >> With Regards, > >> >>>> >> Vibhatha Abeykoon > >> >>>> >> > >> >>>> >> > >> >>>> >> On Fri, May 1, 2020 at 5:17 PM Sutou Kouhei <[email protected] > > > >> >>>> wrote: > >> >>>> >> > >> >>>> >>> Hi, > >> >>>> >>> > >> >>>> >>> > I used these settings, I still want the libs to be in a > custom > >> >>>> >>> directory, > >> >>>> >>> > not in lib/ > >> >>>> >>> > >> >>>> >>> Why? We don't recommend it if you don't a CMake expert. > >> >>>> >>> > >> >>>> >>> You can't do this if you use ARROW_HOME environment > >> >>>> >>> variable. > >> >>>> >>> > >> >>>> >>> You may able to do this by removing ARROW_HOME environment > >> >>>> >>> variable and adding > >> >>>> >>> > >> >>>> >>> > >> >>>> > >> > PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow" > >> >>>> >>> environment variable or something. > >> >>>> >>> > >> >>>> >>> > >> >>>> >>> Thanks, > >> >>>> >>> -- > >> >>>> >>> kou > >> >>>> >>> > >> >>>> >>> In <CABC-QTmsekZwjWG4X= > >> >>>> [email protected]> > >> >>>> >>> "Re: PyArrow building from source issue" on Fri, 1 May 2020 > >> >>>> 15:05:51 > >> >>>> >>> -0400, > >> >>>> >>> Vibhatha Abeykoon <[email protected]> wrote: > >> >>>> >>> > >> >>>> >>> > export > >> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist > >> >>>> >>> > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH > >> >>>> >>> > > >> >>>> >>> > export PYARROW_WITH_PARQUET=1 > >> >>>> >>> > export PYARROW_WITH_PYTHON=1 > >> >>>> >>> > export PYARROW_WITH_BUILD_TESTS=1 > >> >>>> >>> > > >> >>>> >>> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > >> >>>> >>> > \ > >> >>>> >>> > -DARROW_WITH_BZ2=OFF \ > >> >>>> >>> > -DARROW_WITH_ZLIB=OFF \ > >> >>>> >>> > -DARROW_WITH_ZSTD=OFF \ > >> >>>> >>> > -DARROW_WITH_LZ4=OFF \ > >> >>>> >>> > -DARROW_WITH_SNAPPY=OFF \ > >> >>>> >>> > -DARROW_WITH_BROTLI=OFF \ > >> >>>> >>> > -DARROW_PARQUET=ON \ > >> >>>> >>> > -DARROW_PYTHON=ON \ > >> >>>> >>> > -DARROW_BUILD_TESTS=ON \ > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > >> >>>> >>> > \ > >> >>>> >>> > .. > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > I used these settings, I still want the libs to be in a > custom > >> >>>> >>> directory, > >> >>>> >>> > not in lib/ > >> >>>> >>> > > >> >>>> >>> > Does it make things not work? > >> >>>> >>> > > >> >>>> >>> > Now I get the following error, > >> >>>> >>> > > >> >>>> >>> > python setup.py build_ext --inplace > >> >>>> >>> > WARNING: The wheel package is not available. > >> >>>> >>> > running build_ext > >> >>>> >>> > -- Running cmake for pyarrow > >> >>>> >>> > cmake > >> >>>> >>> > >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python > >> >>>> >>> > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off > >> >>>> >>> > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off > >> >>>> >>> > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on > >> >>>> >>> > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off > >> >>>> >>> -DPYARROW_BUILD_HDFS=off > >> >>>> >>> > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off > >> >>>> >>> > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off > >> >>>> >>> > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on > >> >>>> >>> > -DCMAKE_BUILD_TYPE=release > >> >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python > >> >>>> >>> > -- System processor: x86_64 > >> >>>> >>> > -- Arrow build warning level: PRODUCTION > >> >>>> >>> > Using ld linker > >> >>>> >>> > Configured for RELEASE build (set with cmake > >> >>>> >>> > -DCMAKE_BUILD_TYPE={release,debug,...}) > >> >>>> >>> > -- Build Type: RELEASE > >> >>>> >>> > -- Build output directory: > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release > >> >>>> >>> > -- Arrow version: 0.18.0 (HOME: > >> >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist) > >> >>>> >>> > -- Arrow SO and ABI version: 18 > >> >>>> >>> > -- Arrow full SO version: 18.0.0 > >> >>>> >>> > -- Found the Arrow core shared library: > >> >>>> >>> > > >> >>>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >> >>>> >>> > -- Found the Arrow core import library: > >> >>>> >>> > > >> >>>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >> >>>> >>> > -- Found the Arrow core static library: > >> >>>> >>> > > >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a > >> >>>> >>> > -- Found the Arrow Python by HOME: > >> >>>> >>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist > >> >>>> >>> > -- Found the Arrow Python shared library: > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so > >> >>>> >>> > -- Found the Arrow Python import library: > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so > >> >>>> >>> > -- Found the Arrow Python static library: > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a > >> >>>> >>> > CMake Error at > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > >> >>>> >>> > (message): > >> >>>> >>> > Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found > >> version > >> >>>> >>> "1.5.1") > >> >>>> >>> > Call Stack (most recent call first): > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > >> >>>> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > >> >>>> >>> > (_FPHSA_FAILURE_MESSAGE) > >> >>>> >>> > cmake_modules/FindParquet.cmake:116 > >> >>>> >>> (find_package_handle_standard_args) > >> >>>> >>> > CMakeLists.txt:426 (find_package) > >> >>>> >>> > > >> >>>> >>> > Does Parquet need to be installed separately? > >> >>>> >>> > > >> >>>> >>> > With Regards, > >> >>>> >>> > Vibhatha Abeykoon, > >> >>>> >>> > Research Assistant, > >> >>>> >>> > Intelligent Systems Engineering, > >> >>>> >>> > Indiana University Bloomington, > >> >>>> >>> > Cell : +1-812-955-1394 > >> >>>> >>> > Web: https://www.vibhatha.org > >> >>>> >>> > <https://www.linkedin.com/in/vibhathaabeykoon/> > >> >>>> >>> > > >> >>>> >>> > > >> >>>> >>> > On Fri, May 1, 2020 at 1:55 PM Wes McKinney < > >> [email protected]> > >> >>>> >>> wrote: > >> >>>> >>> > > >> >>>> >>> >> Try this instead > >> >>>> >>> >> > >> >>>> >>> >> export > >> >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist > >> >>>> >>> >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH > >> >>>> >>> >> > >> >>>> >>> >> Make sure to use > >> >>>> >>> >> > >> >>>> >>> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME > -DCMAKE_INSTALL_LIBDIR=lib > >> >>>> >>> >> > >> >>>> >>> >> I renamed from "arrowmylibs" to "dist" so that it's less > >> confusing > >> >>>> >>> >> what that directory represents (it doesn't contains the libs > >> >>>> >>> >> directory, but rather the directories include/, lib/, etc.) > >> >>>> >>> >> > >> >>>> >>> >> > >> >>>> >>> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon < > >> >>>> [email protected]> > >> >>>> >>> >> wrote: > >> >>>> >>> >> > > >> >>>> >>> >> > I didn't clearly mention the config I used, > >> >>>> >>> >> > > >> >>>> >>> >> > export > >> >>>> >>> >> > >> >>>> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > >> >>>> >>> >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH > >> >>>> >>> >> > > >> >>>> >>> >> > Before starting I added these ENV vars. > >> >>>> >>> >> > > >> >>>> >>> >> > My objective is to get all the libs inside my build > setting. > >> I > >> >>>> use > >> >>>> >>> arrow > >> >>>> >>> >> as one of the core libraries in my project. > >> >>>> >>> >> > I have to build it in the following way, > >> >>>> >>> >> > > >> >>>> >>> >> > 1. Arrow clone from git > >> >>>> >>> >> > 2. Arrow CPP built > >> >>>> >>> >> > 3. Using arrow CPP I develop custom functions upon loaded > >> data > >> >>>> >>> >> > 4. Integrate Cython APIs for these custom functions > >> >>>> >>> >> > 5. Use Pyarrow Cython to provide more functionality from > >> Python > >> >>>> end > >> >>>> >>> to > >> >>>> >>> >> Cython to my CPP lib which uses Arrow. > >> >>>> >>> >> > > >> >>>> >>> >> > This is the build that I am trying to formulate. So I > have to > >> >>>> keep > >> >>>> >>> those > >> >>>> >>> >> libs there, and the idea is to build python from > >> >>>> >>> >> > the same cloned source. The challenge was to keep all the > >> >>>> shared libs > >> >>>> >>> >> from arrow, my-cython libs to point in the right > >> >>>> >>> >> > direction. Is this a clear description? > >> >>>> >>> >> > > >> >>>> >>> >> > With Regards, > >> >>>> >>> >> > Vibhatha Abeykoon, > >> >>>> >>> >> > Research Assistant, > >> >>>> >>> >> > Intelligent Systems Engineering, > >> >>>> >>> >> > Indiana University Bloomington, > >> >>>> >>> >> > Cell : +1-812-955-1394 > >> >>>> >>> >> > Web: https://www.vibhatha.org > >> >>>> >>> >> > > >> >>>> >>> >> > > >> >>>> >>> >> > > >> >>>> >>> >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney < > >> >>>> [email protected]> > >> >>>> >>> wrote: > >> >>>> >>> >> >> > >> >>>> >>> >> >> This part doesn't look correct > >> >>>> >>> >> >> > >> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > >> >>>> >>> >> >> > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > >> >>>> >>> >> >> > >> >>>> >>> >> >> The usual incantation is > >> >>>> >>> >> >> > >> >>>> >>> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > >> >>>> >>> >> >> -DCMAKE_INSTALL_LIBDIR=lib > >> >>>> >>> >> >> > >> >>>> >>> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that > some > >> >>>> systems > >> >>>> >>> >> >> will install libraries in lib32 or lib64 instead of just > lib > >> >>>> >>> >> >> > >> >>>> >>> >> >> > >> >>>> >>> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon < > >> >>>> >>> [email protected]> > >> >>>> >>> >> wrote: > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > Hi Neal, > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > Yes, I added the flag. But I installed my libs not to > >> >>>> /usr/lib, > >> >>>> >>> but > >> >>>> >>> >> to a different folder. > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > This is the way I built arrow C++, > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > >> >>>> >>> >> >> > > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > >> >>>> >>> >> \ > >> >>>> >>> >> >> > -DARROW_WITH_BZ2=OFF \ > >> >>>> >>> >> >> > -DARROW_WITH_ZLIB=OFF \ > >> >>>> >>> >> >> > -DARROW_WITH_ZSTD=OFF \ > >> >>>> >>> >> >> > -DARROW_WITH_LZ4=OFF \ > >> >>>> >>> >> >> > -DARROW_WITH_SNAPPY=OFF \ > >> >>>> >>> >> >> > -DARROW_WITH_BROTLI=OFF \ > >> >>>> >>> >> >> > -DARROW_PARQUET=ON \ > >> >>>> >>> >> >> > -DARROW_PYTHON=ON \ > >> >>>> >>> >> >> > -DARROW_BUILD_TESTS=ON \ > >> >>>> >>> >> >> > > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > >> >>>> >>> >> \ > >> >>>> >>> >> >> > .. > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > With Regards, > >> >>>> >>> >> >> > Vibhatha Abeykoon, > >> >>>> >>> >> >> > Research Assistant, > >> >>>> >>> >> >> > Intelligent Systems Engineering, > >> >>>> >>> >> >> > Indiana University Bloomington, > >> >>>> >>> >> >> > Cell : +1-812-955-1394 > >> >>>> >>> >> >> > Web: https://www.vibhatha.org > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > > >> >>>> >>> >> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson < > >> >>>> >>> >> [email protected]> wrote: > >> >>>> >>> >> >> >> > >> >>>> >>> >> >> >> Hi Vibhatha, > >> >>>> >>> >> >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON? > >> >>>> >>> >> >> >> > >> >>>> >>> >> >> >> Neal > >> >>>> >>> >> >> >> > >> >>>> >>> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon < > >> >>>> >>> [email protected]> > >> >>>> >>> >> wrote: > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> Hi, > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> I am trying to integrate Arrow with an application > that > >> I > >> >>>> am > >> >>>> >>> >> developing. Here I build Arrow from the source (CPP) and use > >> the > >> >>>> API to > >> >>>> >>> >> develop some custom functions to do a scientific calculation > >> >>>> after data > >> >>>> >>> >> loaded with Arrow table API. On top of this, I develop a > Cython > >> >>>> API to > >> >>>> >>> >> design a python API. > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> In the current stage, I have a new necessity where I > >> need > >> >>>> to > >> >>>> >>> >> consume Arrow Cython API for my code. > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> Here It was hard to link the build libarrow.so.16 > with > >> the > >> >>>> >>> >> libarrow_python.so.16 from the installed pyarrow (separately > >> from > >> >>>> pip). > >> >>>> >>> >> What I realised was everything has to be built from the same > >> >>>> source, so > >> >>>> >>> >> that I can install pyarrow from the source in my virtual > >> >>>> environment. > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> Before going through deeper things, I started by just > >> >>>> building > >> >>>> >>> from > >> >>>> >>> >> source (CPP) and then moving towards installing pyarrow from > >> the > >> >>>> >>> source. > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> I tried to follow the guideline form here, > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> https://arrow.apache.org/docs/developers/python.html > , > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> But when I found issues in the python build, I > followed > >> >>>> this > >> >>>> >>> source, > >> >>>> >>> >> >> >>> (but still, I used the clone from the master, not a > >> >>>> released > >> >>>> >>> >> version) > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> > >> >>>> >>> > >> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> My environmental variables are as follows, > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> python3 setup.py build_ext --inplace > >> >>>> >>> >> >> >>> running build_ext > >> >>>> >>> >> >> >>> -- Running cmake for pyarrow > >> >>>> >>> >> >> >>> cmake > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > >> >>>> >>> >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off > >> >>>> >>> >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off > >> >>>> >>> >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on > >> >>>> >>> >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off > >> >>>> >>> -DPYARROW_BUILD_HDFS=off > >> >>>> >>> >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off > >> >>>> >>> >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off > >> >>>> >>> >> -DPYARROW_BOOST_USE_SHARED=on > -DPYARROW_PARQUET_USE_SHARED=on > >> >>>> >>> >> -DCMAKE_BUILD_TYPE=release > >> >>>> >>> /home/vibhatha/sandbox/arrow/repos/arrow/python > >> >>>> >>> >> >> >>> -- System processor: x86_64 > >> >>>> >>> >> >> >>> -- Arrow build warning level: PRODUCTION > >> >>>> >>> >> >> >>> Using ld linker > >> >>>> >>> >> >> >>> Configured for RELEASE build (set with cmake > >> >>>> >>> >> -DCMAKE_BUILD_TYPE={release,debug,...}) > >> >>>> >>> >> >> >>> -- Build Type: RELEASE > >> >>>> >>> >> >> >>> -- Build output directory: > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release > >> >>>> >>> >> >> >>> -- Arrow version: 0.18.0 (HOME: > >> >>>> >>> >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) > >> >>>> >>> >> >> >>> -- Arrow SO and ABI version: 18 > >> >>>> >>> >> >> >>> -- Arrow full SO version: 18.0.0 > >> >>>> >>> >> >> >>> -- Found the Arrow core shared library: > >> >>>> >>> >> > >> >>>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >> >>>> >>> >> >> >>> -- Found the Arrow core import library: > >> >>>> >>> >> > >> >>>> > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > >> >>>> >>> >> >> >>> -- Found the Arrow core static library: > >> >>>> >>> >> > >> >>>> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a > >> >>>> >>> >> >> >>> CMake Error at > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > >> >>>> >>> >> (message): > >> >>>> >>> >> >> >>> Could NOT find ArrowPython (missing: > >> >>>> ARROW_PYTHON_INCLUDE_DIR) > >> >>>> >>> >> (found > >> >>>> >>> >> >> >>> version "0.18.0") > >> >>>> >>> >> >> >>> Call Stack (most recent call first): > >> >>>> >>> >> >> >>> > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > >> >>>> >>> >> (_FPHSA_FAILURE_MESSAGE) > >> >>>> >>> >> >> >>> cmake_modules/FindArrowPython.cmake:76 > >> >>>> >>> >> (find_package_handle_standard_args) > >> >>>> >>> >> >> >>> CMakeLists.txt:210 (find_package) > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> -- Configuring incomplete, errors occurred! > >> >>>> >>> >> >> >>> See also > >> >>>> >>> >> > >> >>>> >>> > >> >>>> > >> > "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". > >> >>>> >>> >> >> >>> error: command 'cmake' failed with exit status 1 > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> Maybe I am missing some step and I am not quite sure > >> what > >> >>>> is the > >> >>>> >>> >> issue. > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> Any pointers to solve this issue? > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> With Regards, > >> >>>> >>> >> >> >>> Vibhatha > >> >>>> >>> >> >> >>> > >> >>>> >>> >> >> >>> > >> >>>> >>> >> > >> >>>> >>> > >> >>>> >> > >> >>>> > >> >>> -- > >> >>> Vibhatha Abeykoon > >> >>> > >> >> > >> > > -- > > Vibhatha Abeykoon > -- Vibhatha Abeykoon
