C extensions "should" use PyMem_Malloc/PyMem_Free to benefit of PEP 445 hooks like debug hooks and tracemalloc.
* Debug hooks (enabled by -X dev) helps to debug buffer overflow: https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks * tracemalloc: https://docs.python.org/dev/library/tracemalloc.html * PEP 445, API for memory allocators: https://www.python.org/dev/peps/pep-0445/ tracemalloc is used more and more often in CPython internals to provide more information when an issue is detected. Example with ResourceWarning: https://docs.python.org/dev/library/devmode.html#resourcewarning-example Buffer overflow: "On error, the debug hooks use the tracemalloc module to get the traceback where a memory block was allocated. The traceback is only displayed if tracemalloc is tracing Python memory allocations and the memory block was traced." Also, PyMem_Malloc/PyMem_Free now use pymalloc memory allocator which is more efficient than libc malloc/free for memory blocks smaller than or equal to 512 bytes. When I modified them to use pymalloc, I saw a global speedup around 10% faster. But these functions require the caller to hold the GIL. Use PyMem_RawMalloc/PyMem_RawFree if you don't hold the GIL. Note: The Python ssl module doesn't use PyMem_Malloc/PyMem_Free for OpenSSL. Last time I tried, the OpenSSL function to set a custom memory allocator didn't work at all. Maybe it has been fixed in the meanwhile. https://bugs.python.org/issue18227#msg191610 See https://bugs.python.org/issue18227#msg191250 for the list of C extensions which may be modified to use Python memory allocators. Victor Le mer. 13 mai 2020 à 14:23, Pablo Galindo Salgado <pablog...@gmail.com> a écrit : > > > But again this is for PyObjects only. > > Not really, we check also memory blocks: > > https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py#L72 > > as long as you don't directly call malloc and use one of the Python specific > APIs like PyMem_Malloc > then the reflect code should catch that. > > On Wed, 13 May 2020 at 12:29, Giampaolo Rodola' <g.rod...@gmail.com> wrote: >> >> On Wed, May 13, 2020 at 9:17 AM Ammar Askar <am...@ammaraskar.com> wrote: >>> >>> > Py_DECREF calls in the C code >>> >>> I think this part specifically is already covered through refleak >>> checks: >>> https://github.com/python/cpython/blob/master/Lib/test/libregrtest/refleak.py >>> >>> Since it can involve the repetition of tests many times, these aren't >>> run on the CI though, they do get run on the refleak buildbots so >>> issues get caught eventually: >>> https://buildbot.python.org/all/#/builders?tags=%2Brefleak >>> >>> But again this is for PyObjects only. Were you able to find any memory >>> leaks with your proof-of-concept? I don't think there's a lot of >>> chances of someone missing a PyMem_Free call and there's not a lot of >>> other manual memory management but I could be wrong. Anything found >>> there could help motivate adding this a bit more. >> >> >> Yeah, I agree it depends on how many PyMem_* occurrences are there, and it >> probably makes more sense to cover those ones only. Under Modules/* I found: >> >> - 24 occurrences for PyMem_RawMalloc >> - 2 for PyMem_RawCalloc >> - 106 for PyMem_Malloc >> - 12 for PyMem_Calloc >> - 39 for PyMem_New >> - 5 for " = malloc(" >> >> I spent an hour covering around 20 of those and didn't find any leak. It's a >> boring work. I will try to work on it over the next few weeks. >> >> _______________________________________________ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/C7ZZRDPGIUS7Q6Q4AS4YFPD2OOF56JBO/ >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/YNBGAWNCCHF3XK3MBHJWZI4NKTNEPKV6/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/APHU4VQMD2IRUNI4YMOQ7IWTMXYN6AIY/ Code of Conduct: http://python.org/psf/codeofconduct/
