[
https://issues.apache.org/jira/browse/ARROW-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103395#comment-16103395
]
Wes McKinney commented on ARROW-1282:
-------------------------------------
I see. From the comment at
https://github.com/antirez/redis/issues/3799#issuecomment-297832432 this is
definitely concerning since there are plenty of other libraries in the stack
that will not be using jemalloc. Particularly with multithreaded Parquet reads,
there might be two threads which are using different allocators (e.g. the
Python interpreter is using the system allocator, while Parquet/Arrow are using
jemalloc)
[~xhochy] the conservative approach on this might be to disable jemalloc in our
conda and PyPI packages until we understand what is going on. What do you think?
> Large memory reallocation by Arrow causes hang in jemalloc
> ----------------------------------------------------------
>
> Key: ARROW-1282
> URL: https://issues.apache.org/jira/browse/ARROW-1282
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Jeff Knupp
> Fix For: 0.6.0
>
>
> When reallocating a large amount of memory, Arrow is either triggering a bug
> in jemalloc or has a bug itself in the memory manager (many different
> applications reporting same issue but not clear from jemalloc issue
> description if they're sure it's in jemalloc or caused by other issues like
> using multiple memory allocation libraries in the same process, multithreaded
> access, etc).
> Link to stack trace is here:
> https://gist.github.com/jeffknupp/73879feacf9c560afd4f1a20213dc6ef
> Link to issue in jemalloc GitHub is here:
> https://github.com/jemalloc/jemalloc/issues/802
> Originally observed in redis, discussed with jemalloc maintainer here:
> https://github.com/antirez/redis/issues/3799
> *This is entirely reproducible on Ubuntu 16.04 xenial, which uses version
> 3.6.0 according to `apt` metadata.*
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
