Bug#953832: cannot allocate memory in static TLS block
Hi Faidon, On Tue, Mar 17, 2020 at 01:29:59PM +0200, Faidon Liambotis wrote: > Hi Andreas, > > Thanks for reaching out. It sounds like this is already reported as > #951704 (Cc'ed now). OK. > I'll need to give this a closer look, but I hope I > can have an update within the next couple of weeks. Does this work? Well, drmaa is certainly not the most important package inside Debian so I see no reason to push you. But it would be great to have all those Python3 issues from the table sooner or later since it generated noise for the most active Python 2->3 migrators. Just take the time you need. See you Andreas. -- http://fam-tille.de
Bug#953832: cannot allocate memory in static TLS block
Hi Andreas, Thanks for reaching out. It sounds like this is already reported as #951704 (Cc'ed now). I'll need to give this a closer look, but I hope I can have an update within the next couple of weeks. Does this work? Thanks! Faidon On Sun, Mar 15, 2020 at 10:33:20AM +0100, Andreas Tille wrote: > Hi Faidon, > > could you imagine to build jemalloc with --disable-initial-exec-tls > as Sergio suggests below to fix the issue in drmaa (and possibly other > packages)? > > Should I open a separate bug report against jemalloc to request this? > > Kind regards > > Andreas. > > On Sat, Mar 14, 2020 at 05:18:49PM -0400, Sergio Durigan Junior wrote: > > > $ python3 > > > Python 3.7.6 (default, Jan 19 2020, 22:34:52) > > > [GCC 9.2.1 20200117] on linux > > > Type "help", "copyright", "credits" or "license" for more information. > > import drmaa > > > Traceback (most recent call last): > > > File "", line 1, in > > > File > > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", > > > line 65, in > > > from .session import JobInfo, JobTemplate, Session > > > File > > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", > > > line 39, in > > > from drmaa.helpers import (adapt_rusage, Attribute, > > > attribute_names_iterator, > > > File > > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", > > > line 36, in > > > from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t, > > > File > > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", > > > line 58, in > > > _lib = CDLL(libpath, mode=RTLD_GLOBAL) > > > File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__ > > > self._handle = _dlopen(self._name, mode) > > > OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate > > > memory in static TLS block > > > > This is an issue with jemalloc's handling of the TLS model when being > > dlopened.. See: > > > > https://github.com/jemalloc/jemalloc/issues/1237 > > > > The recommended way to build a libjemalloc that is suitable for being > > dlopened is to use '--disable-initial-exec-tls' when building it. Take > > a look at the INSTALL.md file, and look for this option: > > > > https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md > > > > There is a way to workaround this bug by doing an LD_PRELOAD of > > libjemalloc when invoking python, but this will only mask the problem > > and we can't expect users to do/know this. > > > > The way I see it, you can try to convince jemalloc's maintainer to > > enable that flag. > > > > BTW, the reason 'find_library' can't find drmaa's library is because the > > .so is being installed in a non-standard directory. I don't know why > > the package was made like this, though. > > > > Thanks, > > > > -- > > Sergio > > GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 > > Please send encrypted e-mail if possible > > http://sergiodj.net/ > > > > -- > http://fam-tille.de >
Bug#953832: cannot allocate memory in static TLS block
Hi Faidon, could you imagine to build jemalloc with --disable-initial-exec-tls as Sergio suggests below to fix the issue in drmaa (and possibly other packages)? Should I open a separate bug report against jemalloc to request this? Kind regards Andreas. On Sat, Mar 14, 2020 at 05:18:49PM -0400, Sergio Durigan Junior wrote: > > $ python3 > > Python 3.7.6 (default, Jan 19 2020, 22:34:52) > > [GCC 9.2.1 20200117] on linux > > Type "help", "copyright", "credits" or "license" for more information. > import drmaa > > Traceback (most recent call last): > > File "", line 1, in > > File > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", > > line 65, in > > from .session import JobInfo, JobTemplate, Session > > File > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", > > line 39, in > > from drmaa.helpers import (adapt_rusage, Attribute, > > attribute_names_iterator, > > File > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", > > line 36, in > > from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t, > > File > > "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", > > line 58, in > > _lib = CDLL(libpath, mode=RTLD_GLOBAL) > > File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__ > > self._handle = _dlopen(self._name, mode) > > OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory > > in static TLS block > > This is an issue with jemalloc's handling of the TLS model when being > dlopened.. See: > > https://github.com/jemalloc/jemalloc/issues/1237 > > The recommended way to build a libjemalloc that is suitable for being > dlopened is to use '--disable-initial-exec-tls' when building it. Take > a look at the INSTALL.md file, and look for this option: > > https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md > > There is a way to workaround this bug by doing an LD_PRELOAD of > libjemalloc when invoking python, but this will only mask the problem > and we can't expect users to do/know this. > > The way I see it, you can try to convince jemalloc's maintainer to > enable that flag. > > BTW, the reason 'find_library' can't find drmaa's library is because the > .so is being installed in a non-standard directory. I don't know why > the package was made like this, though. > > Thanks, > > -- > Sergio > GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36 > Please send encrypted e-mail if possible > http://sergiodj.net/ -- http://fam-tille.de
Bug#953832: cannot allocate memory in static TLS block
On Saturday, March 14 2020, Andreas Tille wrote:
> On Fri, Mar 13, 2020 at 11:09:31PM +0100, Paul Gevers wrote:
>>
>> raise RuntimeError(('Could not find drmaa library. Please specify
>> its ' +
>> RuntimeError: Could not find drmaa library. Please specify its full
>> path using the environment variable DRMAA_LIBRARY_PATH
>
> I've fixed this in Git. Unfortunately I get a new issue when importing
> drmaa
>
> $ python3
> Python 3.7.6 (default, Jan 19 2020, 22:34:52)
> [GCC 9.2.1 20200117] on linux
> Type "help", "copyright", "credits" or "license" for more information.
import drmaa
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py",
> line 65, in
> from .session import JobInfo, JobTemplate, Session
> File
> "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py",
> line 39, in
> from drmaa.helpers import (adapt_rusage, Attribute,
> attribute_names_iterator,
> File
> "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py",
> line 36, in
> from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
> File
> "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py",
> line 58, in
> _lib = CDLL(libpath, mode=RTLD_GLOBAL)
> File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
> self._handle = _dlopen(self._name, mode)
> OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory
> in static TLS block
>
>
> Any help would be welcome
This is an issue with jemalloc's handling of the TLS model when being
dlopened.. See:
https://github.com/jemalloc/jemalloc/issues/1237
The recommended way to build a libjemalloc that is suitable for being
dlopened is to use '--disable-initial-exec-tls' when building it. Take
a look at the INSTALL.md file, and look for this option:
https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md
There is a way to workaround this bug by doing an LD_PRELOAD of
libjemalloc when invoking python, but this will only mask the problem
and we can't expect users to do/know this.
The way I see it, you can try to convince jemalloc's maintainer to
enable that flag.
BTW, the reason 'find_library' can't find drmaa's library is because the
.so is being installed in a non-standard directory. I don't know why
the package was made like this, though.
Thanks,
--
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF 31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/
signature.asc
Description: PGP signature
Bug#953832: cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)
Hi Andreas, On 14-03-2020 14:30, Andreas Tille wrote: > Hi Paul, [...] > Any help would be welcome I can't help you with this. Paul signature.asc Description: OpenPGP digital signature
Bug#953832: cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)
Control: tags -1 help
Control: retitle -1 cannot allocate memory in static TLS block
Hi Paul,
On Fri, Mar 13, 2020 at 11:09:31PM +0100, Paul Gevers wrote:
>
> raise RuntimeError(('Could not find drmaa library. Please specify
> its ' +
> RuntimeError: Could not find drmaa library. Please specify its full
> path using the environment variable DRMAA_LIBRARY_PATH
I've fixed this in Git. Unfortunately I get a new issue when importing
drmaa
$ python3
Python 3.7.6 (default, Jan 19 2020, 22:34:52)
[GCC 9.2.1 20200117] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import drmaa
Traceback (most recent call last):
File "", line 1, in
File
"/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py",
line 65, in
from .session import JobInfo, JobTemplate, Session
File
"/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py",
line 39, in
from drmaa.helpers import (adapt_rusage, Attribute,
attribute_names_iterator,
File
"/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py",
line 36, in
from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
File
"/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py",
line 58, in
_lib = CDLL(libpath, mode=RTLD_GLOBAL)
File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in
static TLS block
Any help would be welcome
Andreas.
--
http://fam-tille.de
