[Python-Dev] Re: PEP 683: "Immortal Objects, Using a Fixed Refcount"

2022-02-15 Thread Inada Naoki
+1 for overall idea. Some comments: > > Also note that "fork" isn't the only operating system mechanism > that uses copy-on-write semantics. > Could you elaborate? mmap, maybe? Generally speaking, fork is very difficult to use in safe. My company's web apps load applications and libraries

[Python-Dev] PEP 683: "Immortal Objects, Using a Fixed Refcount"

2022-02-15 Thread Eric Snow
Eddie and I would appreciate your feedback on this proposal to support treating some objects as "immortal". The fundamental characteristic of the approach is that we would provide stronger guarantees about immutability for some objects. A few things to note: * this is essentially an

[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track

2022-02-15 Thread Pablo Galindo Salgado
> > However, I still wonder: is there anyway to support `PyTraceMalloc_Track` > API without being dependent to `tracemalloc`? I know there is not many > memory tracing tools but I mean I still feel like there should be a generic > way of doing this: A very vague example for demonstration: > >

[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track

2022-02-15 Thread Sümer Cip
> In other words, the allocators are concerned about > memory, not tracing or anything else that can be done by overriding them. Ok I now understand that `tracemalloc`'s use of allocator APIs is just an implementation detail. Allocator APIs were used for tracing but they are not designed for

[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track

2022-02-15 Thread Pablo Galindo Salgado
The memory allocators don't have any context of tracing, they just allocate. Tracemalloc is a trampoline based allocator that also trace what's going on, bit from the point of view of the python allocator system is just another allocator. There is no concept of "notify the python allocator"

[Python-Dev] Custom memory profiler and PyTraceMalloc_Track

2022-02-15 Thread Sümer Cip
Hi everyone, I would like to ask a question about an issue that we faced regarding profiling memory usage: We have implemented a custom memory profiler using `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, we are facing an issue with numpy: numpy seems to have its