That looks pretty cool! I'm really happy that PEP 445 hooks are reused for something different than tracemalloc ;-)
Victor Le mer. 14 août 2019 à 20:12, Yonatan Zunger <zun...@humu.com> a écrit : > > Update: Thanks to Victor's advice and the PEP445 hooks, I put together a > pretty comprehensive logging/sampling heap profiler for Python, and it works > great. The package is now available via pip for anyone who needs it! > > On Thu, Jun 27, 2019 at 4:21 PM Yonatan Zunger <zun...@humu.com> wrote: >> >> Well, then. I think I'm going to have some fun with this. :) >> >> Thank you! >> >> On Thu, Jun 27, 2019 at 4:17 PM Victor Stinner <vstin...@redhat.com> wrote: >>> >>> Le ven. 28 juin 2019 à 01:03, Yonatan Zunger <zun...@humu.com> a écrit : >>> > Although while I have you hear, I do have a further question about how >>> > tracemalloc works: If I'm reading the code correctly, traces get removed >>> > by tracemalloc when objects are free, which means that at equilibrium >>> > (e.g. at the end of a function) the trace would show just the data which >>> > leaked. That's very useful in most cases, but I'm trying to hunt down a >>> > situation where memory usage is transiently spiking -- which might be due >>> > to something being actively used, or to something building up and >>> > overwhelming the GC, or to evil elves in the CPU for all I can tell so >>> > far. Would it be completely insane for tracemalloc to have a mode where >>> > it either records frees separately (e.g. as a malloc of negative space, >>> > at the trace where the free is happening), or where it simply ignores >>> > frees altogether? >>> >>> My very first implementation of tracemalloc produced a log of malloc >>> and free calls. Problem: transferring the log from a slow set top box >>> to a desktop computer was slow, and parsing the log was very slow. >>> Parsing complexity is in O(n) where n is the number of malloc or free >>> calls, knowning that Python calls malloc(), realloc() or free() >>> 270,000 times per second in average: >>> https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator >>> >>> tracemalloc is built on top of PEP 445 -- Add new APIs to customize >>> Python memory allocators: >>> https://www.python.org/dev/peps/pep-0445/ >>> >>> Using these PEP 445 hooks, you should be able to do whatever you want >>> on Python memory allocations and free :-) >>> >>> Example of toy project to inject memory allocation failures: >>> https://github.com/vstinner/pyfailmalloc >>> >>> Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/N3ZAEFYF64Q7WH7LL26E2T2NUV7ENGCY/
