[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track
> > However, I still wonder: is there anyway to support `PyTraceMalloc_Track` > API without being dependent to `tracemalloc`? I know there is not many > memory tracing tools but I mean I still feel like there should be a generic > way of doing this: A very vague example for demonstration: > > PyMemHooks_GetAllocators/PyMemHooks_SetAllocators/PyMemHooks_TrackAlloc > which are public APIs and tracemalloc using them internally instead of > allocator APIs? > This can be investigated, but this is equivalent to add tracing support to the memory allocators and then make tracemalloc a client of such APIs. This can be advantageous to tool authors but it can raise the maintenance cost of the allocators as we would need to make these APIs generic enough as different tools have different constraints (like having to hold the Gil or not to call these APIs, what information can be passed down (do you want to know the pointer and the kind of allocator it was used...etc). There are even tools that want to link the allocated blocks to python objects directly (which is not possible without a considerable redesign). I would recommend opening an issue in bugs.python.org where this can be discussed and (maybe) implemented. Please, feel free to add me to the issue once open :) Regards from rainy London, Pablo Galindo Salgado ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDZCL6OB2B535ZXZYVVCFMV4S6RO32GQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track
> In other words, the allocators are concerned about > memory, not tracing or anything else that can be done by overriding them. Ok I now understand that `tracemalloc`'s use of allocator APIs is just an implementation detail. Allocator APIs were used for tracing but they are not designed for that: which makes perfect sense. However, I still wonder: is there anyway to support `PyTraceMalloc_Track` API without being dependent to `tracemalloc`? I know there is not many memory tracing tools but I mean I still feel like there should be a generic way of doing this: A very vague example for demonstration: PyMemHooks_GetAllocators/PyMemHooks_SetAllocators/PyMemHooks_TrackAlloc which are public APIs and tracemalloc using them internally instead of allocator APIs? >Regards from rainy London, Cheers from a cold Istanbul, :) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BBHAJQGIUIBRTW7ZWUYHSP3JACLCTIHS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track
The memory allocators don't have any context of tracing, they just allocate. Tracemalloc is a trampoline based allocator that also trace what's going on, bit from the point of view of the python allocator system is just another allocator. There is no concept of "notify the python allocator" because the python allocator system doesn't have the concept of external allocation events. You can override it entirely to do something extra, but it is designed (as many other allocators) to not be aware of the existence of other systems running in parallel. In other words, the allocators are concerned about memory, not tracing or anything else that can be done by overriding them. That's why there is no "notify allocator" APIs in the python allocators. Regards from rainy London, Pablo Galindo Salgado On Tue, 15 Feb 2022, 13:57 Sümer Cip, wrote: > Hi everyone, > > I would like to ask a question about an issue that we faced regarding > profiling memory usage: > > We have implemented a custom memory profiler using > `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, > we are facing an issue with numpy: numpy seems to have its own memory > allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc > about the allocation. A simple search on GitHub reveals there are more > projects using this approach: > https://github.com/search?q=PyTraceMalloc_Track=code which is fine. > I believe it is common to use custom memory allocators for scientific > computation which makes perfect sense. > > However, I would have expected to have at least some form of > `PyMem_NotifyAllocator` type of API instead of a one that is specific to > `tracemalloc`? I might be missing some context here. > > WDYT? > > Best, > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/BHOIDGRUWPM5WEOB3EIDPOJLDMU4WQ4F/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ATULLVYZMDGNOE2NYC53ZUZXBBAHLNSO/ Code of Conduct: http://python.org/psf/codeofconduct/
