[
https://issues.apache.org/jira/browse/ARROW-16009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Khoje updated ARROW-16009:
-----------------------------------
Description:
I tried to build and test pyarrow on ppc64le platform with CUDA.
I encounter segmentation fault at
[test_cuda.py/test_foreign_buffer#L292|https://github.com/apache/arrow/blob/e90472e35b40f58b17d408438bb8de1641bfe6ef/python/pyarrow/tests/test_cuda.py#L292].
If i comment out the test_foreign_buffer, then rest of the tests pass
successfully.
I'm sharing steps i followed and log {{{}p11-arrow.zip{}}}.
*
## STEPS
----
{{{{{}mkdir p11-arrow-repo{}}}
{{cd p11-arrow-repo}}
{{git clone [https://github.com/apache/arrow.git]}}
{{cd arrow}}
{{git checkout apache-arrow-7.0.0}}
{{# Not available: benchmark (conda_env_cpp.txt) - Removed for now.}}
{{vim ci/conda_env_cpp.txt}}
{{git diff ci/conda_env_cpp.txt}}
{{{}cd ..{}}}}}
{{{{{}conda create -y -n p11-arrow -c conda-forge{}}}
{{{}conda activate p11-arrow{}}}{{{}conda install -y -c conda-forge {{}}}}
{\{ --file arrow/ci/conda_env_unix.txt }}
{\{ --file arrow/ci/conda_env_cpp.txt }}
{\{ --file arrow/ci/conda_env_python.txt }}
{\{ --file arrow/ci/conda_env_gandiva.txt }}
{\{ compilers }}
{\{ python=3.9 }}
{{{} pandas{}}}{{{}conda install -y -c conda-forge cudatoolkit=11.2{}}}}}
{{{{{}pushd arrow{}}}
{{git submodule init}}
{{git submodule update}}
{{export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"}}
{{export ARROW_TEST_DATA="${PWD}/testing/data"}}
{{{}popd{}}}{{{}# CPP{}}}
{{export ARROW_HOME=$CONDA_PREFIX}}
{{{}export LD_LIBRARY_PATH=$CONDA_PREFIX/lib{}}}{{{}mkdir arrow/cpp/build{}}}
{{{}pushd arrow/cpp/build{}}}{{{}cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME {{}}}}
{\{ -DCMAKE_INSTALL_LIBDIR=lib }}
{\{ -DCMAKE_BUILD_TYPE=release }}
{\{ -DARROW_WITH_BZ2=ON }}
{\{ -DARROW_WITH_ZLIB=ON }}
{\{ -DARROW_WITH_ZSTD=ON }}
{\{ -DARROW_WITH_LZ4=ON }}
{\{ -DARROW_WITH_SNAPPY=ON }}
{\{ -DARROW_WITH_BROTLI=ON }}
{\{ -DARROW_PARQUET=ON }}
{\{ -DARROW_PYTHON=ON }}
{\{ -DARROW_BUILD_TESTS=ON }}
{\{ -DARROW_ORC=ON }}
{\{ -DARROW_DATASET=ON }}
{\{ -DARROW_HDFS=ON }}
{{{} -DARROW_CUDA=ON -DARROW_EXTRA_ERROR_CONTEXT=ON ..{}}}}}
{{{{{}make -j 4{}}}
{{make install}}
{{ctest}}
{{{}popd{}}}}}
{{{{{}# Python{}}}
{{pushd arrow/python}}
{{export PYARROW_WITH_PARQUET=1}}
{{export PYARROW_WITH_CUDA=1}}
{{export PYARROW_WITH_ORC=1}}
{{export PYARROW_WITH_DATASET=1}}
{{export PYARROW_WITH_HDFS=1}}
{{python setup.py build_ext --inplace}}
{{python setup.py install}}
{{pip install -e . --no-build-isolation}}
{{pytest -v}}
{{{}popd{}}}}}
was:
I tried to build and test pyarrow on ppc64le platform with CUDA.
I encounter segmentation fault at
[test_cuda.py/test_foreign_buffer#L292|https://github.com/apache/arrow/blob/e90472e35b40f58b17d408438bb8de1641bfe6ef/python/pyarrow/tests/test_cuda.py#L292].
If i comment out the test_foreign_buffer, then rest of the tests pass
successfully.
I'm sharing steps i followed and log ({{{}{{p11-arrow.zip}}{}}}).
*# STEPS*
----
{{{{mkdir p11-arrow-repo}}
{{cd p11-arrow-repo}}
{{git clone https://github.com/apache/arrow.git}}
{{cd arrow}}
{{git checkout apache-arrow-7.0.0}}
{{# Not available: benchmark (conda_env_cpp.txt) - Removed for now.}}
{{vim ci/conda_env_cpp.txt}}
{{git diff ci/conda_env_cpp.txt}}
{{cd ..}}}}
{{{{conda create -y -n p11-arrow -c conda-forge}}
{{{}conda activate p11-arrow{}}}{{{}conda install -y -c conda-forge \{}}}
{{ --file arrow/ci/conda_env_unix.txt \}}
{{ --file arrow/ci/conda_env_cpp.txt \}}
{{ --file arrow/ci/conda_env_python.txt \}}
{{ --file arrow/ci/conda_env_gandiva.txt \}}
{{ compilers \}}
{{ python=3.9 \}}
{{{} pandas{}}}{{{}conda install -y -c conda-forge cudatoolkit=11.2{}}}}}
{{{{pushd arrow}}
{{git submodule init}}
{{git submodule update}}
{{export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"}}
{{export ARROW_TEST_DATA="${PWD}/testing/data"}}
{{{}popd{}}}{{{}# CPP{}}}
{{export ARROW_HOME=$CONDA_PREFIX}}
{{{}export LD_LIBRARY_PATH=$CONDA_PREFIX/lib{}}}{{{}mkdir arrow/cpp/build{}}}
{{{}pushd arrow/cpp/build{}}}{{{}cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \{}}}
{{ -DCMAKE_INSTALL_LIBDIR=lib \}}
{{ -DCMAKE_BUILD_TYPE=release \}}
{{ -DARROW_WITH_BZ2=ON \}}
{{ -DARROW_WITH_ZLIB=ON \}}
{{ -DARROW_WITH_ZSTD=ON \}}
{{ -DARROW_WITH_LZ4=ON \}}
{{ -DARROW_WITH_SNAPPY=ON \}}
{{ -DARROW_WITH_BROTLI=ON \}}
{{ -DARROW_PARQUET=ON \}}
{{ -DARROW_PYTHON=ON \}}
{{ -DARROW_BUILD_TESTS=ON \}}
{{ -DARROW_ORC=ON \}}
{{ -DARROW_DATASET=ON \}}
{{ -DARROW_HDFS=ON \}}
{{ -DARROW_CUDA=ON -DARROW_EXTRA_ERROR_CONTEXT=ON ..}}}}
{{{{make -j 4}}
{{make install}}
{{ctest}}
{{popd}}}}
{{{{# Python}}
{{pushd arrow/python}}
{{export PYARROW_WITH_PARQUET=1}}
{{export PYARROW_WITH_CUDA=1}}
{{export PYARROW_WITH_ORC=1}}
{{export PYARROW_WITH_DATASET=1}}
{{export PYARROW_WITH_HDFS=1}}
{{python setup.py build_ext --inplace}}
{{python setup.py install}}
{{pip install -e . --no-build-isolation}}
{{pytest -v}}
{{popd}}}}
> [Python][ppc64le][cuda] pytest segfault - test_cuda.py/test_foreign_buffer
> --------------------------------------------------------------------------
>
> Key: ARROW-16009
> URL: https://issues.apache.org/jira/browse/ARROW-16009
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 5.0.0
> Environment: RHEL 8, CUDA, ppc64le
> Reporter: Prashant Khoje
> Priority: Critical
> Labels: CUDA
> Attachments: p11-arrow.zip
>
>
> I tried to build and test pyarrow on ppc64le platform with CUDA.
> I encounter segmentation fault at
> [test_cuda.py/test_foreign_buffer#L292|https://github.com/apache/arrow/blob/e90472e35b40f58b17d408438bb8de1641bfe6ef/python/pyarrow/tests/test_cuda.py#L292].
> If i comment out the test_foreign_buffer, then rest of the tests pass
> successfully.
> I'm sharing steps i followed and log {{{}p11-arrow.zip{}}}.
> *
> ## STEPS
> ----
> {{{{{}mkdir p11-arrow-repo{}}}
> {{cd p11-arrow-repo}}
> {{git clone [https://github.com/apache/arrow.git]}}
> {{cd arrow}}
> {{git checkout apache-arrow-7.0.0}}
> {{# Not available: benchmark (conda_env_cpp.txt) - Removed for now.}}
> {{vim ci/conda_env_cpp.txt}}
> {{git diff ci/conda_env_cpp.txt}}
> {{{}cd ..{}}}}}
> {{{{{}conda create -y -n p11-arrow -c conda-forge{}}}
> {{{}conda activate p11-arrow{}}}{{{}conda install -y -c conda-forge {{}}}}
> {\{ --file arrow/ci/conda_env_unix.txt }}
> {\{ --file arrow/ci/conda_env_cpp.txt }}
> {\{ --file arrow/ci/conda_env_python.txt }}
> {\{ --file arrow/ci/conda_env_gandiva.txt }}
> {\{ compilers }}
> {\{ python=3.9 }}
> {{{} pandas{}}}{{{}conda install -y -c conda-forge cudatoolkit=11.2{}}}}}
> {{{{{}pushd arrow{}}}
> {{git submodule init}}
> {{git submodule update}}
> {{export PARQUET_TEST_DATA="${PWD}/cpp/submodules/parquet-testing/data"}}
> {{export ARROW_TEST_DATA="${PWD}/testing/data"}}
> {{{}popd{}}}{{{}# CPP{}}}
> {{export ARROW_HOME=$CONDA_PREFIX}}
> {{{}export LD_LIBRARY_PATH=$CONDA_PREFIX/lib{}}}{{{}mkdir arrow/cpp/build{}}}
> {{{}pushd arrow/cpp/build{}}}{{{}cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME
> {{}}}}
> {\{ -DCMAKE_INSTALL_LIBDIR=lib }}
> {\{ -DCMAKE_BUILD_TYPE=release }}
> {\{ -DARROW_WITH_BZ2=ON }}
> {\{ -DARROW_WITH_ZLIB=ON }}
> {\{ -DARROW_WITH_ZSTD=ON }}
> {\{ -DARROW_WITH_LZ4=ON }}
> {\{ -DARROW_WITH_SNAPPY=ON }}
> {\{ -DARROW_WITH_BROTLI=ON }}
> {\{ -DARROW_PARQUET=ON }}
> {\{ -DARROW_PYTHON=ON }}
> {\{ -DARROW_BUILD_TESTS=ON }}
> {\{ -DARROW_ORC=ON }}
> {\{ -DARROW_DATASET=ON }}
> {\{ -DARROW_HDFS=ON }}
> {{{} -DARROW_CUDA=ON -DARROW_EXTRA_ERROR_CONTEXT=ON ..{}}}}}
> {{{{{}make -j 4{}}}
> {{make install}}
> {{ctest}}
> {{{}popd{}}}}}
>
> {{{{{}# Python{}}}
> {{pushd arrow/python}}
> {{export PYARROW_WITH_PARQUET=1}}
> {{export PYARROW_WITH_CUDA=1}}
> {{export PYARROW_WITH_ORC=1}}
> {{export PYARROW_WITH_DATASET=1}}
> {{export PYARROW_WITH_HDFS=1}}
> {{python setup.py build_ext --inplace}}
> {{python setup.py install}}
> {{pip install -e . --no-build-isolation}}
> {{pytest -v}}
> {{{}popd{}}}}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
