2013/6/15 Antoine Pitrou <solip...@pitrou.net>: >> http://hg.python.org/cpython/rev/6661a8154eb3 >> ... >> Issue #3329: Add new APIs to customize memory allocators >> >> * Add a new PyMemAllocators structure >> * New functions: >> >> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory >> allocator functions >> - PyMem_GetRawAllocators(), PyMem_SetRawAllocators() >> - PyMem_GetAllocators(), PyMem_SetAllocators() >> - PyMem_SetupDebugHooks() >> - _PyObject_GetArenaAllocators(), _PyObject_SetArenaAllocators() > > My two cents, but I would prefer if this whole changeset was reverted. > I think it adds too much complexity in the memory allocation APIs, > for a pretty specialized benefit. IMHO, we should be able to get by with > less allocation APIs (why the new _Raw APIs) and less hook-setting > functions.
Ok, I reverted my commit. I posted my initial patch 3 months ago on the bug tracker. I got some reviews and discussed with Kristján Valur Jónsson who heavily modified Python for his game at CCP. I started two threads on python-dev this week (ok, only two days ago). I thaugh that the last known issues were fixed with the addition of PyMem_SetupDebugHooks() (to avoid an environment variable, as asked by Nick) and PyMem_RawMalloc() (have a GIL-free allocator). I will work on a PEP to explain all these new functions and their use cases. ** The addition of PyMem_RawMalloc() is motivated by the issue #18203 (Replace calls to malloc() with PyMem_Malloc()). The goal is to be able to setup a custom allocator for *all* allocation made by Python, so malloc() should not be called directly. PyMem_RawMalloc() is required in places where the GIL is not held (ex: in os.getcwd() on Windows). PyMem_Malloc() is misused (called without the GIL held) in different places. Examples: the readline modules and functions called at Python startup, including main(). Replacing PyMem_Malloc() with malloc() would not allow to use the custom allocator everywhere, so PyMem_RawMalloc() is also required here. The last point is an extension of the issue #18203: some external libraries like zlib or OpenSSL are also calling malloc() directly. But Python can configure these libraries to use a custom memory allocator. I plan to configure external libraries to use PyMem_GetRawAllocators() if PyMem_SetRawAllocators() was called (if PyMem_RawMalloc is not simply malloc) and if setting a custom allocator only affect a function and not the whole library. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
