Kristján Valur Jónsson added the comment:
I'd like to add some argument to providing a "file" and "line number" to the
allocation api. I know that currently this is not provided e.g. by the
PyMem_Allocate() functions, but I think it would be wise to provide a "debug"
version of these functions that pass in the call sites. An allocator api that
then also allows for these values to be provided to the malloc/realloc/free
routines is then future-proof in that respect.
Case in point: We have a memory profiler running which uses a allocator hook
system similar to what Victor is proposing. But in addition, it provides a
"file " and "line" argument to every function.
Now, the profiler is currently not using this code. Here how the "malloc"
function looks:
static void *
PyMalloc(size_t size, void *arg, const char *file, int line, const char *msg)
{
void *r = DustMalloc(size);
if (r) {
tmAllocEx(g_telemetryContext, file, line, r, size, "Python alloc: %s",
msg);
ReportAllocInfo(AllocEvent, 0, r, size);
}
return r;
}
tmAllocEx is calling the Telemetry memory profiles and passing in the
allocation site. (http://www.radgametools.com/telemetry.htm, also my blog
about using it:
http://cosmicpercolator.com/2012/05/25/optimizing-python-condition-variables-with-telemetry/
But our profiler, called with ReportAllocInfo, isn't using this. It relies
solely on extracting the python callstack.
Today, I got this email (see attached file Capture.jpg)
Basically, the profiler sees a lot of allocated memory with no python call
stack. Now it would be useful if we had the C call site information, to know
where it came from.
So: My suggestion is that the allocator api be
1) a struct, which allows for a cleaner api function
2) Include C filename and line number.
Even though the current python memory API (e.g. PyMem_Malloc(),
PyObject_Malloc()) do not currently support it, this would allow us to
internally have _extended_ versions of these apis that do support it and macros
that pass in that information. This can be added at a later stage. Having it
in the allcoator api function would make it more future proof.
See also my "pymem.h" and "ccpmem.h" files attached to this defect for examples
on how we have tweaked python's internal memory apis to support this
information.
----------
Added file: http://bugs.python.org/file30496/Capture.JPG
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
