Hello,
I've been trying to get py-arrow built with CUDA support, I've had help from
Wes and Perau on user@, and it seems that the docs for building with CUDA are
out of date; Wes suggested That I try here.
I have a sript:
conda create -n pyarrow-dev
conda activate pyarrow-dev
conda install python numpy six setuptools cython pandas pytest \
cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
gflags brotli jemalloc lz4-c zstd \
double-conversion glog autoconf hypothesis numba \
clangdev=6 flake8 gtest gmock \
-c conda-forge
git clone https://github.com/arrow/arrow.git
conda activate pyarrow-dev
cd arrow
export ARROW_BUILD_TYPE=release
export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
export ARROW_HOME=$CONDA_PREFIX
export PARQUET_HOME=$CONDA_PREFIX
export NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.0/nvvm/libdevice
export NUMBAPRO_NVVM=/usr/local/cuda-9.0/nvvm/lib64/libnvvm.so
cd cpp
cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_PARQUET=off -DARROW_PYTHON=on \
-DARROW_PLASMA=off -DARROW_BUILD_TESTS=OFF \
-DARROW_CUDA=on \
-DCLANG_FORMAT_BIN=`which clang-format` \
.
make -j3
make install
cd ../python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-cuda develop
py.test -sv pyarrow/
which is a slightly modified script given to me by Perau (I'd earlier been
trying to build strictly with cmake)
I'm working on the Amazon Deep Learning AMI:
conda 4.6.4
Clang 6.0.1 (need to upgrade this)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11)
cmake version 3.13.2
GNU Make 4.1
Python 3.6.7
CUDA 9.0
I can build without py-arrow, but my needs involve CUDA support.
Any help would be appreciated.
Thanks in advance,
Andy
