[
https://issues.apache.org/jira/browse/ARROW-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112693#comment-16112693
]
Chris Bartak edited comment on ARROW-1282 at 8/3/17 1:20 PM:
-------------------------------------------------------------
Here's a backtrace - I guess it's actually hanging on re-acquiring the GIL -
any theories?
{code
Thread 2 (Thread 0x7fe5e97ab700 (LWP 12562)):
#0 0x00007fe5f6dcda82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fe5eb0f1953 in ?? () from /usr/lib64/libpython3.4m.so.1.0
#2 0x00007fe5eb0f1f7b in PyEval_RestoreThread () from
/usr/lib64/libpython3.4m.so.1.0
#3 0x00007fe5eb11729b in PyGILState_Ensure () from
/usr/lib64/libpython3.4m.so.1.0
#4 0x00007fe5d0fa143d in __pyx_f_7pyarrow_3lib_check_status(arrow::Status
const&) () from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#5 0x00007fe5d101c040 in
__pyx_pw_7pyarrow_3lib_16MemoryMappedFile_5_open(_object*, _object*, _object*)
()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#6 0x00007fe5d0fc4198 in __pyx_pw_7pyarrow_3lib_73memory_map(_object*,
_object*, _object*) ()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#7 0x00007fe5d0fb673d in __pyx_f_7pyarrow_3lib_get_reader(_object*,
std::shared_ptr<arrow::io::RandomAccessFile>*) ()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#8 0x00007fe5d02303ce in
__pyx_pw_7pyarrow_8_parquet_13ParquetReader_3open(_object*, _object*, _object*)
()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/_parquet.cpython-34m.so
#9 0x00007fe5eb0fa765 in PyEval_EvalFrameEx () from
/usr/lib64/libpython3.4m.so.1.0
#10 0x00007fe5eb0fbdee in PyEval_EvalCodeEx () from
/usr/lib64/libpython3.4m.so.1.0
#11 0x00007fe5eb06e1b3 in ?? () from /usr/lib64/libpython3.4m.so.1.0
}
was (Author: chrisb1):
Here's a backtrace - I guess it's actually hanging on re-acquiring the GIL -
any theories?
{code:
Thread 2 (Thread 0x7fe5e97ab700 (LWP 12562)):
#0 0x00007fe5f6dcda82 in pthread_cond_timedwait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fe5eb0f1953 in ?? () from /usr/lib64/libpython3.4m.so.1.0
#2 0x00007fe5eb0f1f7b in PyEval_RestoreThread () from
/usr/lib64/libpython3.4m.so.1.0
#3 0x00007fe5eb11729b in PyGILState_Ensure () from
/usr/lib64/libpython3.4m.so.1.0
#4 0x00007fe5d0fa143d in __pyx_f_7pyarrow_3lib_check_status(arrow::Status
const&) () from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#5 0x00007fe5d101c040 in
__pyx_pw_7pyarrow_3lib_16MemoryMappedFile_5_open(_object*, _object*, _object*)
()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#6 0x00007fe5d0fc4198 in __pyx_pw_7pyarrow_3lib_73memory_map(_object*,
_object*, _object*) ()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#7 0x00007fe5d0fb673d in __pyx_f_7pyarrow_3lib_get_reader(_object*,
std::shared_ptr<arrow::io::RandomAccessFile>*) ()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/lib.cpython-34m.so
#8 0x00007fe5d02303ce in
__pyx_pw_7pyarrow_8_parquet_13ParquetReader_3open(_object*, _object*, _object*)
()
from
/opt/python/run/venv/lib64/python3.4/site-packages/pyarrow/_parquet.cpython-34m.so
#9 0x00007fe5eb0fa765 in PyEval_EvalFrameEx () from
/usr/lib64/libpython3.4m.so.1.0
#10 0x00007fe5eb0fbdee in PyEval_EvalCodeEx () from
/usr/lib64/libpython3.4m.so.1.0
#11 0x00007fe5eb06e1b3 in ?? () from /usr/lib64/libpython3.4m.so.1.0
}
> Large memory reallocation by Arrow causes hang in jemalloc
> ----------------------------------------------------------
>
> Key: ARROW-1282
> URL: https://issues.apache.org/jira/browse/ARROW-1282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Jeff Knupp
> Fix For: 0.7.0
>
>
> When reallocating a large amount of memory, Arrow is either triggering a bug
> in jemalloc or has a bug itself in the memory manager (many different
> applications reporting same issue but not clear from jemalloc issue
> description if they're sure it's in jemalloc or caused by other issues like
> using multiple memory allocation libraries in the same process, multithreaded
> access, etc).
> Link to stack trace is here:
> https://gist.github.com/jeffknupp/73879feacf9c560afd4f1a20213dc6ef
> Link to issue in jemalloc GitHub is here:
> https://github.com/jemalloc/jemalloc/issues/802
> Originally observed in redis, discussed with jemalloc maintainer here:
> https://github.com/antirez/redis/issues/3799
> *This is entirely reproducible on Ubuntu 16.04 xenial, which uses version
> 3.6.0 according to `apt` metadata.*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
