Thanks for tracking that down! Been bugging me as well and the one time I tried to figure it out I got no where, so kudos for sticking with it!
On Wed, 9 Jan 2019 at 09:13, Victor Stinner <vstin...@redhat.com> wrote: > Hi, > > The "AMD64 Windows 8.1 Refleaks 3.x" buildbot (which hunts reference > leaks and memory leaks) was failing on test_asyncio for 1 year: > > https://bugs.python.org/issue32710 > > It was a leak of a single reference: > > test_aiosend leaked [1, 1, 1] references, sum=3 > > I tried multiple times since last year (January 2018) to understand > the leak: it didn't make any sense (well, as any bug at the beginning, > no?). I checked several times the complex asyncio code: all transports > are cleanly closed, the event loop is properly closed, etc. The code > looks correct. > > After a long sleepness night... I still failed to reproduce the bug > :-( But I succeeded to get a way shorter reproducer script. Thanks to > this script, I was able to loop to get 100 reference leaks instead of > leaking a single reference. Using tracemalloc, I found the faulty > line... but it still didn't make sense to me. After additional several > hours of debug, I found that an overlapped object wasn't released > properly: an asynchronous WSASend(). > > The problem was when an overlapped WSASend() failed immediately, the > internal buffer was not released, whereas it holds a reference to the > input byte string. **It means that an asyncio send() failure using the > ProactorEventLoop can cause a memory leak**... I'm very surprised that > nobody noticed such **huge** memory leak previously! > > The _overlapped bugfix: > > > https://github.com/python/cpython/commit/a234e148394c2c7419372ab65b773d53a57f3625 > > Eventually, the "AMD64 Windows 8.1 Refleaks 3.x" buildbot is back to green! > > https://buildbot.python.org/all/#/builders/80 > > It means that it will be easier and faster to spot reference or memory > leak regressions (specific to Windows, the Gentoo Refleaks buildbot > was already green for several months!). > > Since ProactorEventLoop became the default event loop in Python 3.8 > (on Windows, it's specific to Windows), I hope that we fixed all most > obvious bugs! > > This story also means that any very tiny buildbot failure (a single > test method failure on a single very specific buildbot) can hide a > giant bug ;-) Sadly, we have to fix *all* buildbots failures to find > them... Hopefully, almost all buildbots are green currently. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org >
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
