On Tue, Apr 16, 2019 at 2:11 AM Victor Stinner <vstin...@redhat.com> wrote:
> Hi Michael, > > Do you know the tracemalloc module? Did you try it? It works on a > regular Python (compiled in debug mode). > > I would be curious to know if tracemalloc also allows you to track the > memory leak. > > Playing around with it a little it does not seem super helpful here (unless I am missing something): it tracks the allocations based on the python call stack, which doesn't help here, in a C extension module generated from python code. Though, in the the mypyc case, we could implement a debug option for creating dummy frames so that we always have a useful call stack. That seems like less of an option for actual hand-written extension modules, though. (Though on the flip side, the python call stacks might be more useful there.) sys.getobjects() is just a list of objects. Do you have a tool written > on top of it to track memory leaks? If yes, how? > > Not really. We have a very simple memory profiler built on top of gc.get_objects() that just reports how many of different types of objects there are and how much memory they are using: https://github.com/python/mypy/blob/master/mypy/memprofile.py. I swapped out gc.get_objects() for sys.getobjects(), observed that we were leaking int objects, and inspected the live int objects, which gave a pretty good clue where the leak was. > Victor > > Le mar. 16 avr. 2019 à 00:28, Michael Sullivan <su...@msully.net> a écrit > : > > > > > The main question is if anyone ever used Py_TRACE_REFS? Does someone > > > use sys.getobjects() or PYTHONDUMPREFS environment variable? > > > > I used sys.getobjects() today to track down a memory leak in the > mypyc-compiled version of mypy. > > > > We were leaking memory badly but no sign of the leak was showing up in > mypy's gc.get_objects() based profiler. Using a debug build and switching > to sys.getobjects() showed that we were badly leaking int objects. A quick > inspection of the values in question (large and random looking) suggested > we were leaking hash values, and that quickly pointed me to > https://github.com/mypyc/mypyc/pull/562. > > > > I don't have any strong feelings about whether to keep it in the > "default" debug build, though. I was using a debug build that I built > myself with every debug feature that seemed potentially useful. > > > > -sully > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. >
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
