> -----Original Message-----
> Take a look at the benchmark suite at
> http://hg.python.org/benchmarks/
> The test runner has an -m option that profiles memory usage, you could take
> a look at how that is implemented
>
Yes, out of process monitoring of memory as reported by the OS. We do gather
those counters as well on clients and servers.
But they don't give you the granularity you want when checking for memory leaks
and memory usage by certain algorithms.
In the same way that the unittests have reference leak reports, they could just
have memory usage reports, if the underlying allocator supported that.
FYI the current state of affairs of the cPython 2.7 branch we use is as follows:
1) We allow the API user to specify the base allocator python uses, both for
regular allocs and allocating blocks for the obmalloc one, using:
/* Support for custom allocators */
typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int
line, const char *msg);
typedef void *(*PyCCP_Realloc_t)(void *ptr, size_t size, void *arg, const char
*file, int line, const char *msg);
typedef void (*PyCCP_Free_t)(void *ptr, void *arg, const char *file, int line,
const char *msg);
typedef size_t (*PyCCP_Msize_t)(void *ptr, void *arg);
typedef struct PyCCP_CustomAllocator_t
{
PyCCP_Malloc_t pMalloc;
PyCCP_Realloc_t pRealloc;
PyCCP_Free_t pFree;
PyCCP_Msize_t pMsize; /* can be NULL, or return -1 if no size info is
avail. */
void *arg; /* opaque argument for the functions */
} PyCCP_CustomAllocator_t;
/* To set an allocator! use 0 for the regular allocator, 1 for the block
allocator.
* pass a null pointer to reset to internal default
*/
PyAPI_FUNC(void) PyCCP_SetAllocator(int which, const PyCCP_CustomAllocator_t
*); /* for BLUE to set the current context */
/* internal data member */
extern PyCCP_CustomAllocator_t _PyCCP_CustomAllocator[];
2) using ifdefs, the macros will delegate all final allocations through these
allocators. This includes all the "naked" malloc calls scattered about, they
are patched up using #defines.
3) Additionally, there is an internal layer of management, before delegating to
the external allocators. This internal manager provides statistics, exposed
through the "sys" module.
The layering is something like this, all more or less definable by
pre-processor macros. (raw malloc() is turned into something else via
pre-processor magic and a special "patch_malloc.h" file added to the modules
which uses raw malloc())
PyMem_Malloc() PyObject_Malloc()
| |
v v
Mem bookkeeping obj bookkeeping
| |
| v
malloc() | obmallocator
| | |
v v v
PyMem_MALLOC_RAW() PyObject_MALLOC_RAW
| |
v v
malloc() or vectored allocator specified through API function
Cheers,
K
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
