STINNER Victor added the comment:
New patch (version 2), more complete:
* add "void *data" argument to all allocator functions
* add "block" API used for pymalloc allocator to allocate arenas. Use mmap or
malloc, but may use VirtualAlloc in a near future (see #13483). Callbacks
prototype:
- void block_malloc (size_t, void*);
- void block_free (void*, size_t, void*);
* remove PY_ALLOC_SYSTEM_API
Main API:
---
#define PY_ALLOC_MEM_API 'm' /* PyMem_Malloc() API */
#define PY_ALLOC_OBJECT_API 'o' /* PyObject_Malloc() API */
PyAPI_FUNC(int) Py_GetAllocators(
char api,
void* (**malloc_p) (size_t size, void *user_data),
void* (**realloc_p) (void *ptr, size_t size, void *user_data),
void (**free_p) (void *ptr, void *user_data),
void **user_data_p
);
PyAPI_FUNC(int) Py_SetAllocators(
char api,
void* (*malloc) (size_t size, void *user_data),
void* (*realloc) (void *ptr, size_t size, void *user_data),
void (*free) (void *ptr, void *user_data),
void *user_data
);
PyAPI_FUNC(void) Py_GetBlockAllocators(
void* (**malloc_p) (size_t size, void *user_data),
void (**free_p) (void *ptr, size_t size, void *user_data),
void **user_data_p
);
PyAPI_FUNC(int) Py_SetBlockAllocators(
void* (*malloc) (size_t size, void *user_data),
void (*free) (void *ptr, size_t size, void *user_data),
void *user_data
);
---
I see the following use cases using allocators:
* Don't use malloc nor mmap but your own allocator: replace PyMem and PyObject
allocators
* Track memory leaks (my pytracemalloc project, or Antoine's simple
_Py_AllocatedBlocks counter): hook PyMem and PyObject allocators
* Fill newly allocated memory with a pattern and check for buffer underflow and
overflow: hook PyMem and PyObject allocators
"Hook" means adding extra code before and/or after calling the original
function.
The final API should allow to hook the APIS multiple times and replacing
allocators. So it should be possible to track memory leaks, detect buffer
overflow and our your own allocators. It is not yet possible with the patch 2,
because _PyMem_DebugMalloc() calls directly malloc().
_PyMem_DebugMalloc is no more used by PyObject_Malloc. This code should be
rewritten to use the hook approach instead of replacing memory allocators.
Example tracing PyMem calls using the hook approach:
-----------------------------------
typedef struct {
void* (*malloc) (size_t, void*);
void* (*realloc) (void*, size_t, void*);
void (*free) (void*, void*);
void *data;
} allocators_t;
allocators_t pymem, pyobject;
void* trace_malloc (size_t size, void* data)
{
allocators_t *alloc = (allocators_t *)data;
printf("malloc(%z)\n", size);
return alloc.malloc(size, alloc.data);
}
void* trace_realloc (void* ptr, size_t size, void* data)
{
allocators_t *alloc = (allocators_t *)data;
printf("realloc(%p, %z)\n", ptr, size);
return alloc.realloc(ptr, size, alloc.data);
}
void trace_free (void* ptr, void* data)
{
allocators_t *alloc = (allocators_t *)data;
printf("free(%p)\n", ptr);
alloc.free(ptr, alloc.data);
}
void hook_pymem(void)
{
Py_GetAllocators(PY_ALLOC_MEM_API, &pymem.malloc, &pymem.realloc,
&pymem.free, &pymem.data);
Py_SetAllocators(PY_ALLOC_MEM_API, trace_malloc, trace_realloc, trace_free,
&pymem);
Py_GetAllocators(PY_ALLOC_OBJECT_API, &pyobject.malloc, &pyobject.realloc,
&pyobject.free, &pyobject.data);
Py_SetAllocators(PY_ALLOC_OBJECT_API, trace_malloc, trace_realloc,
trace_free, &pyobject);
}
-----------------------------------
I didn't try the example :-p It is just to give you an idea of the API and how
to use it.
----------
Added file: http://bugs.python.org/file30453/py_setallocators-2.patch
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
