Nathaniel Smith added the comment:
I think we're talking past each other :-).
> If I change tracemalloc, it's not to fullfit numpy requirements, it must
> remain very generic
Nothing about what I'm saying is relevant to numpy -- the patch attached to
this bug report is already plenty for what numpy needs. (Well, it still needs a
public API like PyMem_Track/Untrack or something, but never mind for now.)
The only reason I'm speaking up now is that if you're adding a manual
track/untrack API, then a relatively trivial addition now makes tracemalloc
vastly more powerful, so I don't want to miss this opportunity.
> void* allows to implement the rejected option of also storing the C filename
> an C line number:
And if you want to attach some extra metadata to traces, then that's an
interesting idea that I can potentially imagine various use cases for. But it's
not the idea I'm talking about :-). (FWIW, I think the biggest challenge for
your idea will be how the allocation sites -- which might be in arbitrary user
code -- are supposed to figure out what kind of metadata they should be
attaching. And if it's information that tracemalloc can compute itself -- like
C backtraces -- then there's no reason for it to be in the public API, which is
the thing I'm concerned about here.)
What I'm talking about is different: I think it should be possible to re-use
the tracemalloc infrastructure to track other resources besides "heap
allocations". So for my use case, it's crucial that we index by (domain,
pointer), because the address 0xdeadbeef on the heap is different from the
address 0xdeadbeef on the GPU. We'll never want to group by pointer alone
without the domain, because that would cause us to actually misinterpret the
data (if I do PyMem_Track("gpu", 0xdeadbeef); PyMem_Untrack("heap",
0xdeadbeef), then this should not cause tracemalloc to forget about the gpu
allocation! I think this is very different than your C backtrace example). And,
it's always obvious to callers what kind of thing to pass here, because they
know perfectly well whether they just allocated memory on the heap or on the
GPU, so the public API is an appropriate place for this information. And, it's
immediately obvious that for this use case, there will only be a few different
domains in use a
t one time, so it's very inefficient to literally store (domain, pointer)
pairs -- replacing the current pointer => trace design with a domain =>
(pointer => trace) design would indeed require changing tracemalloc's design a
bit, but not, I think, in any fundamental way?
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26530>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
