Hi, I am trying to integrate Arrow with an application that I am developing. Here I build Arrow from the source (CPP) and use the API to develop some custom functions to do a scientific calculation after data loaded with Arrow table API. On top of this, I develop a Cython API to design a python API.
In the current stage, I have a new necessity where I need to consume Arrow Cython API for my code. Here It was hard to link the build libarrow.so.16 with the libarrow_python.so.16 from the installed pyarrow (separately from pip). What I realised was everything has to be built from the same source, so that I can install pyarrow from the source in my virtual environment. Before going through deeper things, I started by just building from source (CPP) and then moving towards installing pyarrow from the source. I tried to follow the guideline form here, https://arrow.apache.org/docs/developers/python.html, But when I found issues in the python build, I followed this source, (but still, I used the clone from the master, not a released version) https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521 My environmental variables are as follows, python3 setup.py build_ext --inplace running build_ext -- Running cmake for pyarrow cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python -- System processor: x86_64 -- Arrow build warning level: PRODUCTION Using ld linker Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...}) -- Build Type: RELEASE -- Build output directory: /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release -- Arrow version: 0.18.0 (HOME: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs) -- Arrow SO and ABI version: 18 -- Arrow full SO version: 18.0.0 -- Found the Arrow core shared library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so -- Found the Arrow core import library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so -- Found the Arrow core static library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message): Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found version "0.18.0") Call Stack (most recent call first): /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE) cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args) CMakeLists.txt:210 (find_package) -- Configuring incomplete, errors occurred! See also "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log". error: command 'cmake' failed with exit status 1
