Hi Wes, Thanks for the hints!
I have created a very minimal GPU support to pyarrow, see https://github.com/apache/arrow/compare/master...Quansight:pearu-cuda-pyarrow?expand=1 This includes: 1. introducing libarrow_gpu that is built into pyarrow.lib_gpu extension module, currently only CudaBuffer is exposed 2. updating cmake configuration files to detect libarrow_gpu.{so,a} libraries 3. updating setup.py with new options --with{,out}-arrow-gpu and PYARROW_BUILD_ARROW_GPU env variable support. Let me know if something in my approach needs to be changed. Regarding packaging: 1. FYI, there is a cudatoolkit-dev package in conda-forge: https://github.com/conda-forge/cudatoolkit-dev-feedstock 2. One approach is to introduce new conda packages arrow-cpp-gpu and pyarrow-gpu that are build using -DARROW_GPU and --with-arrow-gpu, respectively. Best regards, Pearu On Wed, Aug 29, 2018 at 12:32 AM, Wes McKinney <wesmck...@gmail.com> wrote: > hi Pearu, > > Sounds good to me; I had wanted to do it already but got caught up > working on other things. You'll probably want to create > libarrow_gpu.pxd since this is an optional library. Otherwise it > should be reasonably straightforward. > > some relevant issues > > - https://issues.apache.org/jira/browse/ARROW-1424 > - https://issues.apache.org/jira/browse/ARROW-2512 > > If we wanted to ship Python wheels and conda packages with CUDA > support built in, what would we have to do (currently these binaries > are produced in Circle CI, Travis CI, and Appveyor)? > > - Wes > On Tue, Aug 28, 2018 at 4:34 PM Pearu Peterson > <pearu.peter...@quansight.com> wrote: > > > > Hi, > > > > Is there any interest in implementing CudaBuffer support in pyarrow? Are > > there any obstacles that would prevent to do this? > > > > I would be willing to give it a try. > > > > As I understand, the entry point for this task would be to patch > > pyarrow/includes/libarrow.pxd . Any hints for what to take into account > are > > appreciated. > > > > Best regards, > > Pearu >
