[jira] [Comment Edited] (ARROW-15541) cannot install python wrapper after building arrow/5.0.0

Fri, 04 Feb 2022 03:42:23 -0800 


    [ 
https://issues.apache.org/jira/browse/ARROW-15541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486743#comment-17486743
 ] 

Joris Van den Bossche edited comment on ARROW-15541 at 2/4/22, 11:41 AM:
-------------------------------------------------------------------------

Hi Joris,

Thank you for the prompt response.
I actually read the setup.py file in the repo 
https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/python/setup.py#L45
 and the minimum version is set to 0.29.

I installed a newer version 0.29.27 and tried the build again. It passes the 
breaking point previous throw the error message complains about the cython 
grammar error:)

However, the new build of pyarrow ended up with the following return message
“””
[ 90%] Built target _dataset
[ 93%] Compiling Cython CXX source for _json...
[ 93%] Built target _json_pyx
[ 96%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o
[100%] Linking CXX shared module release/_json.cpython-37m-x86_64-linux-gnu.so
[100%] Built target _json
-- Finished cmake --build for pyarrow
Bundling includes: release/include
Moving built C-extension release/lib.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_fs.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_fs.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_csv.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_csv.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_json.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_json.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_compute.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_compute.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_cuda.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_cuda.cpython-37m-x86_64-linux-gnu.so
Did not find release/_flight.cpython-37m-x86_64-linux-gnu.so
Cython module _flight failure permitted
Moving built C-extension release/_dataset.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_dataset.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_feather.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_feather.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_parquet.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_parquet.cpython-37m-x86_64-linux-gnu.so
Did not find release/_orc.cpython-37m-x86_64-linux-gnu.so
Cython module _orc failure permitted
Did not find release/_plasma.cpython-37m-x86_64-linux-gnu.so
Cython module _plasma failure permitted
Did not find release/_s3fs.cpython-37m-x86_64-linux-gnu.so
Cython module _s3fs failure permitted
Did not find release/_hdfs.cpython-37m-x86_64-linux-gnu.so
Cython module _hdfs failure permitted
Moving built C-extension release/_hdfsio.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_hdfsio.cpython-37m-x86_64-linux-gnu.so
Did not find release/gandiva.cpython-37m-x86_64-linux-gnu.so
Cython module gandiva failure permitted

“”"


Are the complaints about “did not find release/ <cython module>, failure 
permitted” harmless? Given I turned off those modules in my arrow cpp build.

Many thanks again for the support.

Cheers,
Yue



was (Author: JIRAUSER284682):
Hi Joris,

Thank you for the prompt response.
I actually read the setup.py file in the repo 
https://github.com/apache/arrow/blob/4591d76fce2846a29dac33bf01e9ba0337b118e9/python/setup.py#L45
 and the minimum version is set to 0.29.

I installed a newer version 0.29.27 and tried the build again. It passes the 
breaking point previous throw the error message complains about the cython 
grammar error:)

However, the new build of pyarrow ended up with the following return message
“””
[ 90%] Built target _dataset
[ 93%] Compiling Cython CXX source for _json...
[ 93%] Built target _json_pyx
[ 96%] Building CXX object CMakeFiles/_json.dir/_json.cpp.o
[100%] Linking CXX shared module release/_json.cpython-37m-x86_64-linux-gnu.so
[100%] Built target _json
-- Finished cmake --build for pyarrow
Bundling includes: release/include
Moving built C-extension release/lib.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/lib.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_fs.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_fs.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_csv.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_csv.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_json.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_json.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_compute.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_compute.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_cuda.cpython-37m-x86_64-linux-gnu.so to build 
path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_cuda.cpython-37m-x86_64-linux-gnu.so
Did not find release/_flight.cpython-37m-x86_64-linux-gnu.so
Cython module _flight failure permitted
Moving built C-extension release/_dataset.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_dataset.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_feather.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_feather.cpython-37m-x86_64-linux-gnu.so
Moving built C-extension release/_parquet.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_parquet.cpython-37m-x86_64-linux-gnu.so
Did not find release/_orc.cpython-37m-x86_64-linux-gnu.so
Cython module _orc failure permitted
Did not find release/_plasma.cpython-37m-x86_64-linux-gnu.so
Cython module _plasma failure permitted
Did not find release/_s3fs.cpython-37m-x86_64-linux-gnu.so
Cython module _s3fs failure permitted
Did not find release/_hdfs.cpython-37m-x86_64-linux-gnu.so
Cython module _hdfs failure permitted
Moving built C-extension release/_hdfsio.cpython-37m-x86_64-linux-gnu.so to 
build path 
/scratch/z00/yxs900/install/python_modules/arrow/python/pyarrow/_hdfsio.cpython-37m-x86_64-linux-gnu.so
Did not find release/gandiva.cpython-37m-x86_64-linux-gnu.so
Cython module gandiva failure permitted

“”"


Are the complaints about “did not find release/ <cython module>, failure 
permitted” harmless? Given I turned off those modules in my arrow cpp build.

Many thanks again for the support.

Cheers,
Yue

On 3 Feb 2022, at 10:14 pm, Joris Van den Bossche (Jira) 
<[email protected]<mailto:[email protected]>> wrote:


   [ 
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-15541%3Fpage%3Dcom.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel%26focusedCommentId%3D17486382%23comment-17486382&amp;data=04%7C01%7Cyue.s%40anu.edu.au%7C770d6a38700046e9a5c508d9e7064aa3%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637794836476385937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=YemaOft7M012XyFDrUrhcDcn9S6t4OfP4gUmP0kaUp8%3D&amp;reserved=0
 ]

Joris Van den Bossche commented on ARROW-15541:
-----------------------------------------------

Can you try with a more recent version of cython? (0.29.14 is quite old, more 
than two years) It might be we need to set a minimum cython version in our 
build scripts

cannot install python wrapper after building arrow/5.0.0
--------------------------------------------------------

               Key: ARROW-15541
               URL: 
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FARROW-15541&amp;data=04%7C01%7Cyue.s%40anu.edu.au%7C770d6a38700046e9a5c508d9e7064aa3%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637794836476385937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Oi9EwKj5rQMt4Br4TvWUeNZohw7oPyZBv2ryhUJdb20%3D&amp;reserved=0
           Project: Apache Arrow
        Issue Type: Bug
          Reporter: Yue Sun
          Priority: Major

Hi There,
I installed arrow/5.0.0 as the following.
"""
git clone 
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Farrow.git&amp;data=04%7C01%7Cyue.s%40anu.edu.au%7C770d6a38700046e9a5c508d9e7064aa3%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637794836476385937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=FXygpM37YVroWfn5H%2F3BeKCHfalm3Nx%2Fg6qxOSfExE4%3D&amp;reserved=0
cd arrow
git checkout apache-arrow-5.0.0
export ARROW_HOME=/g/data/z00/yxs900/.local/environments/arrow
mkdir cpp/build
cd cpp/build
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DCMAKE_INSTALL_LIBDIR=lib64 \
      -DCMAKE_BUILD_TYPE=release \
      -DPython3_EXECUTABLE=/apps/python3/3.7.4/bin/python3 \
      -DARROW_PARQUET=ON \
      -DARROW_PYTHON=ON \
      -DARROW_CUDA=ON \
      -DARROW_DATASET=ON \
      -DARROW_BUILD_TESTS=OFF \
      ..

make -j1 VERBOSE=1
make install
"""
The build is successful and installed properly. But when I try to install the 
python wrapper as the following
"""
cd ../../python
export PYARROW_WITH_CUDA=1
export PYARROW_WITH_DATASET=1
export PYARROW_WITH_PARQUET=1
export LD_LIBRARY_PATH=${ARROW_HOME}/lib64:${LD_LIBRARY_PATH}
python3 setup.py build_ext --inplace
"""
It stopped at the error message
"""
Error compiling Cython file:
------------------------------------------------------------
...
                                                          Writable):
        pass

    cdef cppclass CInputStream" arrow::io::InputStream"(FileInterface,
                                                        Readable):
        CResult[shared_ptr[const CKeyValueMetadata]] ReadMetadata()
                                ^
------------------------------------------------------------

pyarrow/includes/libarrow.pxd:1199:33: Expected ']', found 'CKeyValueMetadata'
"""
Did you see this error before? I am using cmake/3.21.4, cuda/11.0.3, 
mccl/2.8.4, and python 3.7.4. The cython version is 0.29.14.
Let me know what else do you need for the diagnosis.
Cheers,
Yue



--
This message was sent by Atlassian Jira
(v8.20.1#820001)


> cannot install python wrapper after building arrow/5.0.0
> --------------------------------------------------------
>
>                 Key: ARROW-15541
>                 URL: https://issues.apache.org/jira/browse/ARROW-15541
>             Project: Apache Arrow
>          Issue Type: Bug
>            Reporter: Yue Sun
>            Priority: Major
>
> Hi There,
> I installed arrow/5.0.0 as the following.
> """
> git clone https://github.com/apache/arrow.git
> cd arrow
> git checkout apache-arrow-5.0.0
> export ARROW_HOME=/g/data/z00/yxs900/.local/environments/arrow
> mkdir cpp/build
> cd cpp/build
> cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>       -DCMAKE_INSTALL_LIBDIR=lib64 \
>       -DCMAKE_BUILD_TYPE=release \
>       -DPython3_EXECUTABLE=/apps/python3/3.7.4/bin/python3 \
>       -DARROW_PARQUET=ON \
>       -DARROW_PYTHON=ON \
>       -DARROW_CUDA=ON \
>       -DARROW_DATASET=ON \
>       -DARROW_BUILD_TESTS=OFF \
>       ..
>  
> make -j1 VERBOSE=1
> make install
> """
> The build is successful and installed properly. But when I try to install the 
> python wrapper as the following
> """
> cd ../../python
> export PYARROW_WITH_CUDA=1
> export PYARROW_WITH_DATASET=1
> export PYARROW_WITH_PARQUET=1
> export LD_LIBRARY_PATH=${ARROW_HOME}/lib64:${LD_LIBRARY_PATH}
> python3 setup.py build_ext --inplace
> """
> It stopped at the error message
> """
> Error compiling Cython file:
> ------------------------------------------------------------
> ...
>                                                           Writable):
>         pass
>  
>     cdef cppclass CInputStream" arrow::io::InputStream"(FileInterface,
>                                                         Readable):
>         CResult[shared_ptr[const CKeyValueMetadata]] ReadMetadata()
>                                 ^
> ------------------------------------------------------------
>  
> pyarrow/includes/libarrow.pxd:1199:33: Expected ']', found 'CKeyValueMetadata'
> """
> Did you see this error before? I am using cmake/3.21.4, cuda/11.0.3, 
> mccl/2.8.4, and python 3.7.4. The cython version is 0.29.14. 
> Let me know what else do you need for the diagnosis.
> Cheers,
> Yue



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to