Hi, > I used these settings, I still want the libs to be in a custom directory, > not in lib/
Why? We don't recommend it if you don't a CMake expert. You can't do this if you use ARROW_HOME environment variable. You may able to do this by removing ARROW_HOME environment variable and adding PYARROW_CMAKE_OPTIONS="-DCMAKE_MODULE_PATH=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist/lib/cmake/arrow" environment variable or something. Thanks, -- kou In <CABC-QTmsekZwjWG4X=awrfqzgyi+qldyec_kvbn2njgtwjl...@mail.gmail.com> "Re: PyArrow building from source issue" on Fri, 1 May 2020 15:05:51 -0400, Vibhatha Abeykoon <[email protected]> wrote: > export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist > export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH > > export PYARROW_WITH_PARQUET=1 > export PYARROW_WITH_PYTHON=1 > export PYARROW_WITH_BUILD_TESTS=1 > > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ > > -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs > \ > -DARROW_WITH_BZ2=OFF \ > -DARROW_WITH_ZLIB=OFF \ > -DARROW_WITH_ZSTD=OFF \ > -DARROW_WITH_LZ4=OFF \ > -DARROW_WITH_SNAPPY=OFF \ > -DARROW_WITH_BROTLI=OFF \ > -DARROW_PARQUET=ON \ > -DARROW_PYTHON=ON \ > -DARROW_BUILD_TESTS=ON \ > > -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 > \ > .. > > > I used these settings, I still want the libs to be in a custom directory, > not in lib/ > > Does it make things not work? > > Now I get the following error, > > python setup.py build_ext --inplace > WARNING: The wheel package is not available. > running build_ext > -- Running cmake for pyarrow > cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/ENVARROW/bin/python > -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off > -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off > -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on > -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off > -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off > -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off > -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on > -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python > -- System processor: x86_64 > -- Arrow build warning level: PRODUCTION > Using ld linker > Configured for RELEASE build (set with cmake > -DCMAKE_BUILD_TYPE={release,debug,...}) > -- Build Type: RELEASE > -- Build output directory: > /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release > -- Arrow version: 0.18.0 (HOME: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist) > -- Arrow SO and ABI version: 18 > -- Arrow full SO version: 18.0.0 > -- Found the Arrow core shared library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > -- Found the Arrow core import library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so > -- Found the Arrow core static library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a > -- Found the Arrow Python by HOME: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist > -- Found the Arrow Python shared library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so > -- Found the Arrow Python import library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.so > -- Found the Arrow Python static library: > /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow_python.a > CMake Error at > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 > (message): > Could NOT find Parquet (missing: PARQUET_LIB_DIR) (found version "1.5.1") > Call Stack (most recent call first): > > /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 > (_FPHSA_FAILURE_MESSAGE) > cmake_modules/FindParquet.cmake:116 (find_package_handle_standard_args) > CMakeLists.txt:426 (find_package) > > Does Parquet need to be installed separately? > > With Regards, > Vibhatha Abeykoon, > Research Assistant, > Intelligent Systems Engineering, > Indiana University Bloomington, > Cell : +1-812-955-1394 > Web: https://www.vibhatha.org > <https://www.linkedin.com/in/vibhathaabeykoon/> > > > On Fri, May 1, 2020 at 1:55 PM Wes McKinney <[email protected]> wrote: > >> Try this instead >> >> export ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/dist >> export LD_LIBRARY_PATH=$ARROW_HOME/lib:$LD_LIBRARY_PATH >> >> Make sure to use >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME -DCMAKE_INSTALL_LIBDIR=lib >> >> I renamed from "arrowmylibs" to "dist" so that it's less confusing >> what that directory represents (it doesn't contains the libs >> directory, but rather the directories include/, lib/, etc.) >> >> >> On Fri, May 1, 2020 at 12:51 PM Vibhatha Abeykoon <[email protected]> >> wrote: >> > >> > I didn't clearly mention the config I used, >> > >> > export >> ARROW_HOME=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> > export LD_LIBRARY_PATH=$ARROW_HOME:$LD_LIBRARY_PATH >> > >> > Before starting I added these ENV vars. >> > >> > My objective is to get all the libs inside my build setting. I use arrow >> as one of the core libraries in my project. >> > I have to build it in the following way, >> > >> > 1. Arrow clone from git >> > 2. Arrow CPP built >> > 3. Using arrow CPP I develop custom functions upon loaded data >> > 4. Integrate Cython APIs for these custom functions >> > 5. Use Pyarrow Cython to provide more functionality from Python end to >> Cython to my CPP lib which uses Arrow. >> > >> > This is the build that I am trying to formulate. So I have to keep those >> libs there, and the idea is to build python from >> > the same cloned source. The challenge was to keep all the shared libs >> from arrow, my-cython libs to point in the right >> > direction. Is this a clear description? >> > >> > With Regards, >> > Vibhatha Abeykoon, >> > Research Assistant, >> > Intelligent Systems Engineering, >> > Indiana University Bloomington, >> > Cell : +1-812-955-1394 >> > Web: https://www.vibhatha.org >> > >> > >> > >> > On Fri, May 1, 2020 at 1:28 PM Wes McKinney <[email protected]> wrote: >> >> >> >> This part doesn't look correct >> >> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> >> >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> >> >> >> The usual incantation is >> >> >> >> -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> -DCMAKE_INSTALL_LIBDIR=lib >> >> >> >> The reason we pass -DCMAKE_INSTALL_LIBDIR=lib is that some systems >> >> will install libraries in lib32 or lib64 instead of just lib >> >> >> >> >> >> On Fri, May 1, 2020 at 10:38 AM Vibhatha Abeykoon <[email protected]> >> wrote: >> >> > >> >> > Hi Neal, >> >> > >> >> > Yes, I added the flag. But I installed my libs not to /usr/lib, but >> to a different folder. >> >> > >> >> > This is the way I built arrow C++, >> >> > >> >> > cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \ >> >> > >> >> -DCMAKE_INSTALL_LIBDIR=/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs >> \ >> >> > -DARROW_WITH_BZ2=OFF \ >> >> > -DARROW_WITH_ZLIB=OFF \ >> >> > -DARROW_WITH_ZSTD=OFF \ >> >> > -DARROW_WITH_LZ4=OFF \ >> >> > -DARROW_WITH_SNAPPY=OFF \ >> >> > -DARROW_WITH_BROTLI=OFF \ >> >> > -DARROW_PARQUET=ON \ >> >> > -DARROW_PYTHON=ON \ >> >> > -DARROW_BUILD_TESTS=ON \ >> >> > >> >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> \ >> >> > .. >> >> > >> >> > >> >> > >> >> > >> >> > With Regards, >> >> > Vibhatha Abeykoon, >> >> > Research Assistant, >> >> > Intelligent Systems Engineering, >> >> > Indiana University Bloomington, >> >> > Cell : +1-812-955-1394 >> >> > Web: https://www.vibhatha.org >> >> > >> >> > >> >> > >> >> > On Fri, May 1, 2020 at 11:29 AM Neal Richardson < >> [email protected]> wrote: >> >> >> >> >> >> Hi Vibhatha, >> >> >> Did you build Arrow C++ with -DARROW_PYTHON=ON? >> >> >> >> >> >> Neal >> >> >> >> >> >> On Fri, May 1, 2020 at 8:24 AM Vibhatha Abeykoon <[email protected]> >> wrote: >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> I am trying to integrate Arrow with an application that I am >> developing. Here I build Arrow from the source (CPP) and use the API to >> develop some custom functions to do a scientific calculation after data >> loaded with Arrow table API. On top of this, I develop a Cython API to >> design a python API. >> >> >>> >> >> >>> In the current stage, I have a new necessity where I need to >> consume Arrow Cython API for my code. >> >> >>> >> >> >>> Here It was hard to link the build libarrow.so.16 with the >> libarrow_python.so.16 from the installed pyarrow (separately from pip). >> What I realised was everything has to be built from the same source, so >> that I can install pyarrow from the source in my virtual environment. >> >> >>> >> >> >>> Before going through deeper things, I started by just building from >> source (CPP) and then moving towards installing pyarrow from the source. >> >> >>> >> >> >>> I tried to follow the guideline form here, >> >> >>> >> >> >>> https://arrow.apache.org/docs/developers/python.html, >> >> >>> >> >> >>> But when I found issues in the python build, I followed this source, >> >> >>> (but still, I used the clone from the master, not a released >> version) >> >> >>> >> >> >>> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 >> >> >>> >> >> >>> My environmental variables are as follows, >> >> >>> >> >> >>> python3 setup.py build_ext --inplace >> >> >>> running build_ext >> >> >>> -- Running cmake for pyarrow >> >> >>> cmake >> -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 >> -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off >> -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off >> -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on >> -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off >> -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off >> -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off >> -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on >> -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python >> >> >>> -- System processor: x86_64 >> >> >>> -- Arrow build warning level: PRODUCTION >> >> >>> Using ld linker >> >> >>> Configured for RELEASE build (set with cmake >> -DCMAKE_BUILD_TYPE={release,debug,...}) >> >> >>> -- Build Type: RELEASE >> >> >>> -- Build output directory: >> /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release >> >> >>> -- Arrow version: 0.18.0 (HOME: >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) >> >> >>> -- Arrow SO and ABI version: 18 >> >> >>> -- Arrow full SO version: 18.0.0 >> >> >>> -- Found the Arrow core shared library: >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >> >>> -- Found the Arrow core import library: >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so >> >> >>> -- Found the Arrow core static library: >> /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a >> >> >>> CMake Error at >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 >> (message): >> >> >>> Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) >> (found >> >> >>> version "0.18.0") >> >> >>> Call Stack (most recent call first): >> >> >>> >> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 >> (_FPHSA_FAILURE_MESSAGE) >> >> >>> cmake_modules/FindArrowPython.cmake:76 >> (find_package_handle_standard_args) >> >> >>> CMakeLists.txt:210 (find_package) >> >> >>> >> >> >>> >> >> >>> -- Configuring incomplete, errors occurred! >> >> >>> See also >> "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". >> >> >>> error: command 'cmake' failed with exit status 1 >> >> >>> >> >> >>> Maybe I am missing some step and I am not quite sure what is the >> issue. >> >> >>> >> >> >>> Any pointers to solve this issue? >> >> >>> >> >> >>> With Regards, >> >> >>> Vibhatha >> >> >>> >> >> >>> >>
