Antoine Pitrou added the comment:
PEP 445 allows you to customize the Python memory allocators, which is a better
solution than shipping several ones with Python ;-)
--
resolution: - wont fix
status: open - closed
___
Python tracker
Charles-François Natali neolo...@free.fr added the comment:
Fortunately, it is easy to solve by defining the following in
dlmalloc:
#define HAVE_MORECORE 0
I was expecting this answer ;-)
Here's a quick demo, on a Linux box:
cf@neobox:~/cpython$ ./python Tools/pybench/pybench.py -n 1
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Sorry for the very late reply; I have been quite busy recently with the
birth of my second daughter, a new job, a new home town and soon a new home.
...
But with your patch, such code wouldn't be thread-safe anymore. This
patch
Changes by Sébastien Sablé sa...@users.sourceforge.net:
Added file: http://bugs.python.org/file22698/patch_dlmalloc_Python_2_7_1.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3526
___
Antoine Pitrou pit...@free.fr added the comment:
Since I am changing of job, I won't be working on AIX anymore (yeah!);
You seem happy about that :)
Does it mean the project to have an AIX buildbot is abandoned?
I also don't expect this patch to be integrated spontaneously without
someone
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Does it mean the project to have an AIX buildbot is abandoned?
We have a buildbot running internally on AIX. I could not get the necessary
modifications integrated upstream in the official Python buildbot so that we
could plug
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Another reason why you should not force dlmalloc for all applications linked
with libpython is because dlmalloc is (by default) not thread safe, while the
system malloc is (generally) thread-safe. It is possible to define a
Charles-François Natali neolo...@free.fr added the comment:
Also that addresses the issue of two threads inside different malloc
implementations at the same time: it is currently not allowed with
PyMem_Malloc.
That's not true.
You can perfectly have one thread inside PyMem_Malloc while
Sébastien Sablé sa...@users.sourceforge.net added the comment:
I'm just proposing an alternative that I find cleaner, simpler and easier to
maintain.
I understand how LD_PRELOAD works but I find it neither clean nor simple to
maintain.
Also by using a wrapper to call Python you still
Antoine Pitrou pit...@free.fr added the comment:
Since I want PyMem_MALLOC to call dlmalloc, I would need to export the
malloc symbol from libpython so that Python extensions could use it
when calling PyMem_MALLOC, but that would impact all malloc calls in
applications which embed Python for
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Yes, I was probably not clear:
When --with-dlmalloc is activated, PyMem_MALLOC/PyMem_Malloc will call
dlmalloc, PyMem_REALLOC/PyMem_Realloc will call dlrealloc and
PyMem_FREE/PyMem_Free will call dlfree.
While calls to
Antoine Pitrou pit...@free.fr added the comment:
Yes, I was probably not clear:
When --with-dlmalloc is activated, PyMem_MALLOC/PyMem_Malloc will call
dlmalloc, PyMem_REALLOC/PyMem_Realloc will call dlrealloc and
PyMem_FREE/PyMem_Free will call dlfree.
While calls to malloc/free/realloc
Floris Bruynooghe floris.bruynoo...@gmail.com added the comment:
On 29 April 2011 17:16, Antoine Pitrou rep...@bugs.python.org wrote:
Antoine Pitrou pit...@free.fr added the comment:
Yes, I was probably not clear:
When --with-dlmalloc is activated, PyMem_MALLOC/PyMem_Malloc will call
Antoine Pitrou pit...@free.fr added the comment:
For the same reason that extension modules can choose between
PyMem_Malloc and plain malloc (or whatever else). Python has never
forced it's malloc on extension modules why should it now?
We're talking about a platform-specific feature
Charles-Francois Natali neolo...@free.fr added the comment:
Even worse than that, mixing to malloc implementations could lead to trouble.
For example, the trimming code ensures that the heap is where it last set it.
So if an allocation has been made by another implementation in the meantime,
Sébastien Sablé sa...@users.sourceforge.net added the comment:
I share the opinion of Floris on this: just because you link your application
with python does not mean you want it to handle all memory management.
If you want the memory to be handled by Python, you should call PyMem_Malloc.
Charles-Francois Natali neolo...@free.fr added the comment:
I don't understand the point concerning trimming/fragmentation/threading by
Charles-Francois: dlmalloc will allocate its own memory segment using mmap
and handle memory inside that segment when you do a
dlmalloc/dlfree/dlrealloc.
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Hi Charles-François,
it is possible to impact the memory allocation system on AIX using some
environment variables (MALLOCOPTIONS and others), but it is not very elegant
(it will impact all applications running with this
Charles-Francois Natali neolo...@free.fr added the comment:
it is possible to impact the memory allocation system on AIX using some
environment variables (MALLOCOPTIONS and others)
LD_PRELOAD won't impact AIX's malloc behaviour, but allows you to
replace it transparently by any other
Floris Bruynooghe floris.bruynoo...@gmail.com added the comment:
So by using dlmalloc on SunOS and AIX you would get the same level
of performance for memory operations that you already probably can
appreciate on Linux systems.
Yes, but with the above trick, you can do that without
Antoine Pitrou pit...@free.fr added the comment:
So how about a --with-dlmalloc=path/to/dlmalloc.c?
Can't you just add dlmalloc to LDFLAGS or something? Or would the
default malloc still be selected?
This is
similar in how python uses e.g. openssl to provide optional extra
Charles-Francois Natali neolo...@free.fr added the comment:
I just noticed there's already a version of dlmalloc in
Modules/_ctypes/libffi/src/dlmalloc.c
Compiling with gcc -shared -fpic -o /tmp/dlmalloc.so
./Modules/_ctypes/libffi/src/dlmalloc.c
Then LD_PRELOAD=/tmp/dlmalloc.so ./python
Charles-Francois Natali neolo...@free.fr added the comment:
Sébastien:
I'm chiming in late, but doesn't AIX have something like LD_PRELOAD?
Why not use it to transparently replace AIX's legacy malloc by another malloc
implementation like dlmalloc or ptmalloc?
That would not require any patching
Changes by Floris Bruynooghe floris.bruynoo...@gmail.com:
--
nosy: +flub
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3526
___
___
Antoine Pitrou pit...@free.fr added the comment:
I guess there are not that many people running Python applications with
a big memory footprint on AIX or SunOS, otherwise this problem would be
more popular.
Not only, but integrating a big chunk of foreign code in something as critical
as
Sébastien Sablé sa...@users.sourceforge.net added the comment:
Well I am still interested in getting this patch officially integrated in
Python.
This patch is integrated in the version of Python that we deploy to our
customers with our products (Sungard GP3). So it runs in production at
Mark Lawrence breamore...@yahoo.co.uk added the comment:
Any SunOS/AIX people interested in keeping this open?
--
nosy: +BreamoreBoy
versions: +Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3526
Sébastien Sablé [EMAIL PROTECTED] added the comment:
My previous patch has a small problem as I believed dlmalloc was always
returning a non-NULL value, even when asking for 0 bytes.
It turns out not to be the case, so here is a new patch
(patch_dlmalloc3.diff) which must be applied after the
Sébastien Sablé [EMAIL PROTECTED] added the comment:
Here is a new patch so that pymalloc can be combined with dlmalloc.
I first added the --with-pymalloc-mmap option to configure.in which
ensures that pymalloc arenas are allocated through mmap when possible.
However I found this was not
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I will try to do that patch in coming weeks (obmalloc mostly allocates
some 256KB arenas so it should nearly always use mmap).
Exactly so. If you can, please also consider supporting Windows, in the
same way.
Anything in obmalloc that is
Martin v. Löwis [EMAIL PROTECTED] added the comment:
OTOH, if for some compelling reason it's believed Python knows better
than platform vendors, then obmalloc should be uglied-up on all paths to
make the enlightened choice.
I'm proposing that obmalloc is changed to know better than system
Tim Peters [EMAIL PROTECTED] added the comment:
I have to admit that if Python /didn't/ know better than platform libc
implementers in some cases, there would be no point to having obmalloc
at all :-(
What you (Martin) suggest is reasonable enough.
___
Sébastien Sablé [EMAIL PROTECTED] added the comment:
[sorry for the late reply, I have been on holidays]
Martin:
you are right that this memory is moved to swap and does not consume any
real memory; however we decided to work on this patch because we
observed on our application some
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le vendredi 08 août 2008 à 22:46 +, Martin v. Löwis a écrit :
Instead, Python's own memory allocate (obmalloc) should be changed to
directly use the virtual memory interfaces of the operating system (i.e.
mmap), bypassing the malloc of
Martin v. Löwis [EMAIL PROTECTED] added the comment:
Instead, Python's own memory allocate (obmalloc) should be changed to
directly use the virtual memory interfaces of the operating system (i.e.
mmap), bypassing the malloc of the C library.
How would that interact with fork()?
Nicely,
Antoine Pitrou [EMAIL PROTECTED] added the comment:
Le samedi 09 août 2008 à 17:28 +, Martin v. Löwis a écrit :
Martin v. Löwis [EMAIL PROTECTED] added the comment:
Instead, Python's own memory allocate (obmalloc) should be changed to
directly use the virtual memory interfaces of the
New submission from Sébastien Sablé [EMAIL PROTECTED]:
Hi,
We run a big application mostly written in Python (with Pyrex/C
extensions) on different systems including Linux, SunOS and AIX.
The memory footprint of our application on Linux is fine; however we
found that on AIX and SunOS, any
Changes by Sébastien Sablé [EMAIL PROTECTED]:
Added file: http://bugs.python.org/file11083/customized_malloc_AIX.pdf
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3526
___
Changes by Sébastien Sablé [EMAIL PROTECTED]:
--
keywords: +patch
Added file: http://bugs.python.org/file11084/patch_dlmalloc.diff
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3526
___
Antoine Pitrou [EMAIL PROTECTED] added the comment:
This is very interesting, although it should probably go through
discussion on python-dev since it involves integrating a big chunk of
external code.
--
components: +Interpreter Core
nosy: +pitrou
priority: - normal
versions: +Python
Martin v. Löwis [EMAIL PROTECTED] added the comment:
I cannot quite see why the problem is serious: even though the memory is
not returned to the system, it will be swapped out to the swap file, so
it doesn't consume any real memory (just swap space).
I don't think Python should integrate a
41 matches
Mail list logo
