clamydo opened a new issue, #35394:
URL: https://github.com/apache/arrow/issues/35394
### Describe the usage question you have. Please include as many useful
details as possible.
I'm developing a C++ project using Arrow with python interface that requires
the cxx11 ABI. I'd like to interoperate with the pypi pyarrow python module
that is compiled with cxx3 ABI.
Because I cannot link my C++-python extension/library against libs in
`pyarrow` (ABI conflict), I am trying to instead _statically_ link to
self-compiled cxx11 `libarrow.a` and `libarrow-python.a` libs. This way, I do
not have to link against the precompiled libs from vanilla pyarrow. I thought,
because Arrow is a well specified in-memory format, it should be possible to
pass a pointer from vanilla pyarrow to my python module lib without clashing
into the ABI incompatibility.
But I cannot get it working. My attempt is written below. It contains a
minimal C++-Python extension linked statically to self compiled cxx11 arrow
libs (in `lib`-folder).
In `quicktest.py` I construct a Arrow table with vanilla pyarrow (`pip
install pyarrow`) and pass it to my python module. But I get a segfault. See
backtrace below:
When using a cxx11 version of the python pyarrow module, it works.
`demo.cpp`
```cpp
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <arrow/api.h>
#include <arrow/python/pyarrow.h>
static auto load_arrow_table(PyObject *self, PyObject *o) -> PyObject *
{
auto is_table = arrow::py::is_table(o);
if (!is_table)
{
throw std::invalid_argument("no table");
}
auto tbl = arrow::py::unwrap_table(o);
return PyUnicode_FromString(((*tbl)->ToString()).c_str());
}
static PyMethodDef DemoMethods[] = {
{"load_arrow_table", load_arrow_table, METH_O, "Some description"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
static struct PyModuleDef demomodule = {
PyModuleDef_HEAD_INIT,
"demo", /* name of module */
NULL, /* module documentation, may be NULL */
-1, /* size of per-interpreter state of the module,
or -1 if the module keeps state in global variables. */
DemoMethods};
PyMODINIT_FUNC PyInit_demo(void)
{
arrow::py::import_pyarrow();
return PyModule_Create(&demomodule);
}
```
`setup.py`
```python
from setuptools import Extension, setup
setup(
ext_modules=[
Extension(
name="demo",
include_dirs = [
/include',
],
extra_compile_args=['-std=c++17'],
extra_link_args=[
'-l:libarrow_python.a',
'-l:libarrow.a',
'-l:libarrow_bundled_dependencies.a',
],
library_dirs = [
'lib',
],
sources=["demo.cpp"], # all sources are compiled into a single
binary file
),
]
)
```
`pyproject.toml`
```toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "demo" # as it would appear on PyPI
version = "0.0.0-dev"
requires-python = ">=3.8"
dependencies = [
]
```
`quicktest.py`
```python
import pyarrow as pa
import demo
days = pa.array([1, 12, 17, 23, 28], type=pa.int8())
months = pa.array([1, 3, 5, 7, 1], type=pa.int8())
years = pa.array([1990, 2000, 1995, 2000, 1995], type=pa.int16())
birthdays_table = pa.table([days, months, years],
names=["days", "months", "years"])
out = demo.load_arrow_table(birthdays_table)
print(out)
```
Backtrace
```
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__memmove_sse2_unaligned_erms () at
../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
440 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such
file or directory.
(gdb) bt
#0 __memmove_sse2_unaligned_erms () at
../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#1 0x00007ffff3b01b28 in std::basic_streambuf<char, std::char_traits<char>
>::xsputn(char const*, long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007ffff3af3824 in std::basic_ostream<char, std::char_traits<char> >&
std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char,
std::char_traits<char> >&, char const*, long) () from
/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007fff9d488a70 in arrow::(anonymous
namespace)::SchemaPrinter::PrintField(arrow::Field const&) ()
from demo.cpython-38-x86_64-linux-gnu.so
#4 0x00007fff9d489675 in arrow::(anonymous
namespace)::SchemaPrinter::Print() ()
from demo.cpython-38-x86_64-linux-gnu.so
#5 0x00007fff9d492b4b in arrow::PrettyPrint(arrow::Table const&,
arrow::PrettyPrintOptions const&, std::ostream*) ()
from
/home/fabiankoessel/python_cpp_arrow_test/demo.cpython-38-x86_64-linux-gnu.so
#6 0x00007fff9d4dab12 in arrow::Table::ToString[abi:cxx11]() const () from
demo.cpython-38-x86_64-linux-gnu.so
#7 0x00007fff9d2f4de9 in load_arrow_table (self=<optimized out>,
o=0x7fff9d974130) at /usr/include/c++/9/bits/shared_ptr_base.h:1020
#8 0x00000000005c5080 in ?? ()
#9 0x0000000000570ac2 in _PyEval_EvalFrameDefault ()
#10 0x0000000000569cea in _PyEval_EvalCodeWithName ()
#11 0x000000000068e7b7 in PyEval_EvalCode ()
#12 0x0000000000680001 in ?? ()
#13 0x000000000068007f in ?? ()
#14 0x0000000000680121 in ?? ()
#15 0x0000000000680db7 in PyRun_SimpleFileExFlags ()
#16 0x00000000006b8122 in Py_RunMain ()
#17 0x00000000006b84ad in Py_BytesMain ()
#18 0x00007ffff7de4083 in __libc_start_main (main=0x4ef1e0 <main>, argc=2,
argv=0x7fffffffde88, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7fffffffde78) at
../csu/libc-start.c:308
#19 0x00000000005fb39e in _start ()
```
Using gcc 9.5 on Ubuntu 20.04
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
