[
https://issues.apache.org/jira/browse/ARROW-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17363806#comment-17363806
]
Giordon Stark commented on ARROW-13085:
---------------------------------------
[~apitrou] - thanks for the nice hint. Switching to {{devtoolset-8}} does fix
the segfault! (Maybe the docs need to have some hint about making sure a
compatible toolchain is used if one is building extensions using the python
libs?)
{code:java}
# ./run_example.sh
/Users/kratsg/awkward-arrow-cmake-pybind11/examples
/Users/kratsg/awkward-arrow-cmake-pybind11
* Reading CSV file 'test.csv' into table
* Read table:
Integers: int64
Strings: string
Timestamps: timestamp[s]
----
Integers:
[
[
1,
2
]
]
Strings:
[
[
"Some",
"data"
]
]
Timestamps:
[
[
2018-11-13 17:11:10,
null
]
]
* Writing table into Arrow IPC file 'test.arrow'
{code}
where the changes I made to the {{Dockerfile}} are
{code:java}
diff --git a/Dockerfile b/Dockerfile
index 1fc311e..87b9289 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -7,6 +7,11 @@ WORKDIR /
SHELL [ "/bin/bash", "-c" ]
COPY build-requirements.txt /tmp/build-requirements.txt
-RUN python -m pip --no-cache-dir install --upgrade pip setuptools wheel && \
+
+RUN yum update -y -q && \
+ yum install -y -q centos-release-scl && \
+ yum install -y -q devtoolset-8 && \
+ echo "source scl_source enable devtoolset-8" >> ~/.bash_profile && \
+ python -m pip --no-cache-dir install --upgrade pip setuptools wheel && \
python -m pip --no-cache-dir install --requirement
/tmp/build-requirements.txt && \
python -m pip list {code}
which is not so bad. This certainly was not obvious at all to me 😀
> Apache Arrow minimal cpp build segfaults with pyarrow libs
> ----------------------------------------------------------
>
> Key: ARROW-13085
> URL: https://issues.apache.org/jira/browse/ARROW-13085
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Affects Versions: 4.0.1
> Environment: See dockerfile:
> https://github.com/kratsg/awkward-arrow-cmake-pybind11/blob/0e196983b2452c4c3dbbde40984db7031397d0a7/Dockerfile
> Reporter: Giordon Stark
> Priority: Major
>
> h6. _Refer to
> [https://github.com/kratsg/awkward-arrow-cmake-pybind11/issues/6#issuecomment-861592692]
> and code-base available here:
> [https://github.com/kratsg/awkward-arrow-cmake-pybind11/tree/0e196983b2452c4c3dbbde40984db7031397d0a7]_
> The minimal build example provided by arrow
> ([https://github.com/apache/arrow/blob/master/cpp/examples/minimal_build/example.cc]
>
> [)|https://github.com/apache/arrow/blob/master/cpp/examples/minimal_build/example.cc]
> works fine using the libs shipped with {{yum}} on CentOS7. Using the libs
> shipped by {{pip install pyarrow}} instead provides a segfault at runtime.
> Here is the gdb dump with debugging flags included
> {code:java}
> [root@dab7bb04d93f examples]# gdb ./../build/example_arrow
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/example_arrow...done.
> (gdb) run
> Starting program:
> /Users/kratsg/awkward-arrow-cmake-pybind11/examples/./../build/example_arrow
> warning: Error disabling address space randomization: Operation not permitted
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> [New Thread 0x7f9c58dff700 (LWP 1643)]
> * Reading CSV file 'test.csv' into table
> Program received signal SIGSEGV, Segmentation fault.
> __destroy<std::basic_string<char>*> (__last=<optimized out>,
> __first=0x3200000000) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
> 103 std::_Destroy(std::__addressof(*__first));
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.17-324.el7_9.x86_64 libgcc-4.8.5-44.el7.x86_64
> libstdc++-4.8.5-44.el7.x86_64
> (gdb) bt
> #0 __destroy<std::basic_string<char>*> (__last=<optimized out>,
> __first=0x3200000000) at /usr/include/c++/4.8.2/bits/stl_construct.h:103
> #1 _Destroy<std::basic_string<char>*> (__last=<optimized out>,
> __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:126
> #2 _Destroy<std::basic_string<char>*, std::basic_string<char> > (__last=0x0,
> __first=<optimized out>) at /usr/include/c++/4.8.2/bits/stl_construct.h:151
> #3 ~vector (this=0x7ffff43602a8, __in_chrg=<optimized out>) at
> /usr/include/c++/4.8.2/bits/stl_vector.h:415
> #4 arrow::csv::ConvertOptions::~ConvertOptions (this=0x7ffff4360220,
> __in_chrg=<optimized out>)
> at
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/include/arrow/csv/options.h:64
> #5 0x000000000040171a in (anonymous namespace)::RunMain (argv=<optimized
> out>, argc=<optimized out>, this=<optimized out>, this=<optimized out>)
> at /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp:39
> #6 0x000000000040135f in main (argc=<optimized out>, argv=<optimized out>)
> at /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp:66 {code}
> Below is the full CMake log + build log
> {code:java}
> # ./build.sh
> -- The CXX compiler identification is GNU 4.8.5
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Check for working CXX compiler: /usr/bin/c++ - skipped
> -- Detecting CXX compile features
> -- Detecting CXX compile features - done
> -- pybind11 v2.6.2
> -- Found PythonInterp: /usr/local/bin/python3.8 (found version "3.8.10")
> -- Found PythonLibs: /usr/local/lib/libpython3.8.so
> -- Performing Test HAS_FLTO
> -- Performing Test HAS_FLTO - Success
> -- PyArrow includes :
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/include
> -- PyArrow libraries : arrow;arrow_python
> -- PyArrow library dirs : /usr/local/venv/lib/python3.8/site-packages/pyarrow
> -- Awkward includes :
> /usr/local/venv/lib/python3.8/site-packages/awkward/include
> -- Awkward libraries: /usr/local/venv/lib/python3.8/site-packages/awkward
> -- Libraries:
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
> /usr/lib64/libdl.so
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> -- pybind11 v2.6.2
> -- PyArrow includes :
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/include
> -- PyArrow libraries : arrow;arrow_python
> -- PyArrow library dirs : /usr/local/venv/lib/python3.8/site-packages/pyarrow
> -- Awkward includes :
> /usr/local/venv/lib/python3.8/site-packages/awkward/include
> -- Awkward libraries: /usr/local/venv/lib/python3.8/site-packages/awkward
> -- Libraries:
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
> /usr/lib64/libdl.so
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
> -- Configuring done
> -- Generating done
> -- Build files have been written to:
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> -- Cache values
> CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
> CMAKE_INSTALL_PREFIX:PATH=/usr/local
> CPU-KERNELS:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
> LIBARROW:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
> LIBARROW_PYTHON:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
> LIBAWKWARD:FILEPATH=/usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
> LIBDL:FILEPATH=/usr/lib64/libdl.so
> PYBIND11_FINDPYTHON:BOOL=OFF
> PYBIND11_INSTALL:BOOL=OFF
> PYBIND11_NOPYTHON:BOOL=OFF
> PYBIND11_PYTHON_VERSION:STRING=
> PYBIND11_TEST:BOOL=OFF
> ###################################################################
> ######################### BUILD LOG BELOW #########################
> ###################################################################
> /usr/local/bin/cmake -S/Users/kratsg/awkward-arrow-cmake-pybind11/src
> -B/Users/kratsg/awkward-arrow-cmake-pybind11/build --check-build-system
> CMakeFiles/Makefile.cmake 0
> /usr/local/bin/cmake -E cmake_progress_start
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles
> /Users/kratsg/awkward-arrow-cmake-pybind11/build//CMakeFiles/progress.marks
> /usr/bin/gmake -f CMakeFiles/Makefile2 all
> gmake[1]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> /usr/bin/gmake -f CMakeFiles/babel.dir/build.make CMakeFiles/babel.dir/depend
> /usr/bin/gmake -f CMakeFiles/example_arrow.dir/build.make
> CMakeFiles/example_arrow.dir/depend
> /usr/bin/gmake -f CMakeFiles/example.dir/build.make
> CMakeFiles/example.dir/depend
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake
> -E cmake_depends "Unix Makefiles"
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/babel.dir/DependInfo.cmake
> --color=
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake
> -E cmake_depends "Unix Makefiles"
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/example_arrow.dir/DependInfo.cmake
> --color=
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> cd /Users/kratsg/awkward-arrow-cmake-pybind11/build && /usr/local/bin/cmake
> -E cmake_depends "Unix Makefiles"
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/src
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles/example.dir/DependInfo.cmake
> --color=
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> /usr/bin/gmake -f CMakeFiles/example.dir/build.make
> CMakeFiles/example.dir/build
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> /usr/bin/gmake -f CMakeFiles/babel.dir/build.make CMakeFiles/babel.dir/build
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> /usr/bin/gmake -f CMakeFiles/example_arrow.dir/build.make
> CMakeFiles/example_arrow.dir/build
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> gmake[2]: Entering directory
> `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> [ 14%] Building CXX object CMakeFiles/example.dir/example/python.cpp.o
> /usr/bin/c++ -Dexample_EXPORTS -isystem
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem
> /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG
> -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT
> CMakeFiles/example.dir/example/python.cpp.o -MF
> CMakeFiles/example.dir/example/python.cpp.o.d -o
> CMakeFiles/example.dir/example/python.cpp.o -c
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/python.cpp
> [ 28%] Building CXX object CMakeFiles/babel.dir/babel/python.cpp.o
> /usr/bin/c++ -Dbabel_EXPORTS -isystem
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/include -isystem
> /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG
> -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT
> CMakeFiles/babel.dir/babel/python.cpp.o -MF
> CMakeFiles/babel.dir/babel/python.cpp.o.d -o
> CMakeFiles/babel.dir/babel/python.cpp.o -c
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/babel/python.cpp
> [ 42%] Building CXX object CMakeFiles/example_arrow.dir/example/arrow.cpp.o
> /usr/bin/c++ -isystem
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/include -O2 -g -DNDEBUG
> -std=gnu++11 -MD -MT CMakeFiles/example_arrow.dir/example/arrow.cpp.o -MF
> CMakeFiles/example_arrow.dir/example/arrow.cpp.o.d -o
> CMakeFiles/example_arrow.dir/example/arrow.cpp.o -c
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/arrow.cpp
> [ 57%] Linking CXX executable example_arrow
> /usr/local/bin/cmake -E cmake_link_script
> CMakeFiles/example_arrow.dir/link.txt --verbose=1
> /usr/bin/c++ -O2 -g -DNDEBUG CMakeFiles/example_arrow.dir/example/arrow.cpp.o
> -o example_arrow
> -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/pyarrow
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> [ 57%] Built target example_arrow
> [ 71%] Building CXX object CMakeFiles/example.dir/example/minimal.cpp.o
> /usr/bin/c++ -Dexample_EXPORTS -isystem
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/pybind11/include -isystem
> /usr/local/venv/lib/python3.8/site-packages/awkward/include -O2 -g -DNDEBUG
> -fPIC -fvisibility=hidden -std=gnu++11 -MD -MT
> CMakeFiles/example.dir/example/minimal.cpp.o -MF
> CMakeFiles/example.dir/example/minimal.cpp.o.d -o
> CMakeFiles/example.dir/example/minimal.cpp.o -c
> /Users/kratsg/awkward-arrow-cmake-pybind11/src/example/minimal.cpp
> [ 85%] Linking CXX shared module example.cpython-38-x86_64-linux-gnu.so
> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/example.dir/link.txt
> --verbose=1
> /usr/bin/c++ -fPIC -O2 -g -DNDEBUG -shared -o
> example.cpython-38-x86_64-linux-gnu.so
> CMakeFiles/example.dir/example/python.cpp.o
> CMakeFiles/example.dir/example/minimal.cpp.o
> -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/awkward
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
> /usr/lib64/libdl.so
> [100%] Linking CXX shared module babel.cpython-38-x86_64-linux-gnu.so
> /usr/local/bin/cmake -E cmake_link_script CMakeFiles/babel.dir/link.txt
> --verbose=1
> /usr/bin/c++ -fPIC -O2 -g -DNDEBUG -shared -o
> babel.cpython-38-x86_64-linux-gnu.so CMakeFiles/babel.dir/babel/python.cpp.o
> -Wl,-rpath,/usr/local/venv/lib/python3.8/site-packages/pyarrow:/usr/local/venv/lib/python3.8/site-packages/awkward
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow.so
> /usr/local/venv/lib/python3.8/site-packages/pyarrow/libarrow_python.so
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward-cpu-kernels.so
> /usr/local/venv/lib/python3.8/site-packages/awkward/libawkward.so
> /usr/lib64/libdl.so
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> [100%] Built target example
> gmake[2]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> [100%] Built target babel
> gmake[1]: Leaving directory `/Users/kratsg/awkward-arrow-cmake-pybind11/build'
> /usr/local/bin/cmake -E cmake_progress_start
> /Users/kratsg/awkward-arrow-cmake-pybind11/build/CMakeFiles 0 {code}
> One can reproduce this by cloning the corresponding repository recursively,
> then building the docker image, compiling the code, and then running it
> {code:java}
> git clone --recursive [email protected]:kratsg/awkward-arrow-cmake-pybind11.git
> make build-image-debug
> make debug # launches docker image
> > ./build.sh
> > ./run_example.sh{code}
> where {{>}} denotes commands run inside the docker image.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
