Re: Finding windows and docshells leaked until shutdown
I started a section on dealing with leaks on the MDN mochitest page. Feel free to contribute to it with any tips that may be useful to others. I got pointed to this thread when I was asking for some help on leaks yesterday. I couldn't find any details when searching online for anything around the "leakcheck" mechanism, so I figured docs would be good to add. https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest#Diagnosing_and_fixing_leakcheck_failures On Thu, Oct 3, 2019 at 6:10 AM smaug wrote: > On 10/3/19 1:18 AM, Andrew McCreight wrote: > > On Wed, Oct 2, 2019 at 2:35 PM Geoff Lankow > wrote: > > > >> Hi all, I need some advice. > >> > >> I'm trying to enable Mochitest on debug builds of Thunderbird. It works > >> fine, except for the leak check, which finds a number of leaked windows > >> and docshells. Obviously we don't want to leak these things, but I can't > >> find what's holding onto them. > >> > >> I've ruled out a number of likely candidates and I'm quickly running out > >> of ideas. Can you offer any tips or tricks that might help? > >> > >> Here's an example log: > >> > >> > https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=267576044&repo=try-comm-central&lineNumber=8051 > > > > > > What this leak checker means is that a docshell or window was created > while > > the test was running, but was not destroyed until after the test finished > > (and we ran a bunch of GCs and CCs). However, these docshells and windows > > don't show up in the leakcheck output, which means that they were cleaned > > up before we shut down. > > Ah, good, runtime leak :) > Runtime leaks are in general way easier to debug than leaks which are > there until shutdown. > Add a conditional break point to the Release method of the relevant object > and see all the cases Release is called. One (or more) of the Release > calls ends > up revealing where the leak is coming from. > > To get access to the right C++ object, CC log is one way to get the > address. > > > > > > > A typical cause for these is that there's some top level data structure, > > often in JS but it could be in C++, too, that keeps track of every > docshell > > or window that is created, and does so by just adding a reference to it. > > When we start shutdown, the data structure gets torn down, and we don't > > leak things forever. > > > > One small tip is that if you look at the line after the "leaked until > > shutdown" message and you'll see something like this: > > [task 2019-09-20T03:42:16.942Z] 03:42:16 INFO - TEST-INFO | > > comm/calendar/test/browser/browser_calendarList.js | windows(s) leaked: > > [pid = 1155] [serial = 38], [pid = 1155] [serial = 36], [pid = 1155] > > [serial = 39], [pid = 1155] [serial = 37] > > > > The entries like "[pid = 1155] [serial = 38]" are the descriptors we use > > for windows (and docshells have a similar thing). If you search for > "[pid = > > 1155] [serial = 38]" in the log, you can see where exactly each window > was > > created and destroyed. Sometimes that can give a hint about what is going > > wrong. > > > > Like Emilio said, the best approach to fixing these leaks is going to be > > getting GC/CC logs from the point where it complains about the window > being > > alive, and then you can figure out from there what is keeping the window > > alive, which may or may not point at what the leak is. > > > > In practice, doing all of this leak fixing is rather a big project by > > itself, so I'd recommend that you disable this leak checking in > Thunderbird > > somehow, and then just try to get the rest of the debug tests up and > > running, and then maybe come back to it later. This leak checking is done > > in this file (by doing postprocessing of the browser log): > > testing/mochitest/leaks.py > > > > > > > >> > >> GL > >> ___ > >> dev-platform mailing list > >> dev-platform@lists.mozilla.org > >> https://lists.mozilla.org/listinfo/dev-platform > >> > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Finding windows and docshells leaked until shutdown
On 10/3/19 1:18 AM, Andrew McCreight wrote: On Wed, Oct 2, 2019 at 2:35 PM Geoff Lankow wrote: Hi all, I need some advice. I'm trying to enable Mochitest on debug builds of Thunderbird. It works fine, except for the leak check, which finds a number of leaked windows and docshells. Obviously we don't want to leak these things, but I can't find what's holding onto them. I've ruled out a number of likely candidates and I'm quickly running out of ideas. Can you offer any tips or tricks that might help? Here's an example log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=267576044&repo=try-comm-central&lineNumber=8051 What this leak checker means is that a docshell or window was created while the test was running, but was not destroyed until after the test finished (and we ran a bunch of GCs and CCs). However, these docshells and windows don't show up in the leakcheck output, which means that they were cleaned up before we shut down. Ah, good, runtime leak :) Runtime leaks are in general way easier to debug than leaks which are there until shutdown. Add a conditional break point to the Release method of the relevant object and see all the cases Release is called. One (or more) of the Release calls ends up revealing where the leak is coming from. To get access to the right C++ object, CC log is one way to get the address. A typical cause for these is that there's some top level data structure, often in JS but it could be in C++, too, that keeps track of every docshell or window that is created, and does so by just adding a reference to it. When we start shutdown, the data structure gets torn down, and we don't leak things forever. One small tip is that if you look at the line after the "leaked until shutdown" message and you'll see something like this: [task 2019-09-20T03:42:16.942Z] 03:42:16 INFO - TEST-INFO | comm/calendar/test/browser/browser_calendarList.js | windows(s) leaked: [pid = 1155] [serial = 38], [pid = 1155] [serial = 36], [pid = 1155] [serial = 39], [pid = 1155] [serial = 37] The entries like "[pid = 1155] [serial = 38]" are the descriptors we use for windows (and docshells have a similar thing). If you search for "[pid = 1155] [serial = 38]" in the log, you can see where exactly each window was created and destroyed. Sometimes that can give a hint about what is going wrong. Like Emilio said, the best approach to fixing these leaks is going to be getting GC/CC logs from the point where it complains about the window being alive, and then you can figure out from there what is keeping the window alive, which may or may not point at what the leak is. In practice, doing all of this leak fixing is rather a big project by itself, so I'd recommend that you disable this leak checking in Thunderbird somehow, and then just try to get the rest of the debug tests up and running, and then maybe come back to it later. This leak checking is done in this file (by doing postprocessing of the browser log): testing/mochitest/leaks.py GL ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Finding windows and docshells leaked until shutdown
On Wed, Oct 2, 2019 at 2:35 PM Geoff Lankow wrote: > Hi all, I need some advice. > > I'm trying to enable Mochitest on debug builds of Thunderbird. It works > fine, except for the leak check, which finds a number of leaked windows > and docshells. Obviously we don't want to leak these things, but I can't > find what's holding onto them. > > I've ruled out a number of likely candidates and I'm quickly running out > of ideas. Can you offer any tips or tricks that might help? > > Here's an example log: > > https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=267576044&repo=try-comm-central&lineNumber=8051 What this leak checker means is that a docshell or window was created while the test was running, but was not destroyed until after the test finished (and we ran a bunch of GCs and CCs). However, these docshells and windows don't show up in the leakcheck output, which means that they were cleaned up before we shut down. A typical cause for these is that there's some top level data structure, often in JS but it could be in C++, too, that keeps track of every docshell or window that is created, and does so by just adding a reference to it. When we start shutdown, the data structure gets torn down, and we don't leak things forever. One small tip is that if you look at the line after the "leaked until shutdown" message and you'll see something like this: [task 2019-09-20T03:42:16.942Z] 03:42:16 INFO - TEST-INFO | comm/calendar/test/browser/browser_calendarList.js | windows(s) leaked: [pid = 1155] [serial = 38], [pid = 1155] [serial = 36], [pid = 1155] [serial = 39], [pid = 1155] [serial = 37] The entries like "[pid = 1155] [serial = 38]" are the descriptors we use for windows (and docshells have a similar thing). If you search for "[pid = 1155] [serial = 38]" in the log, you can see where exactly each window was created and destroyed. Sometimes that can give a hint about what is going wrong. Like Emilio said, the best approach to fixing these leaks is going to be getting GC/CC logs from the point where it complains about the window being alive, and then you can figure out from there what is keeping the window alive, which may or may not point at what the leak is. In practice, doing all of this leak fixing is rather a big project by itself, so I'd recommend that you disable this leak checking in Thunderbird somehow, and then just try to get the rest of the debug tests up and running, and then maybe come back to it later. This leak checking is done in this file (by doing postprocessing of the browser log): testing/mochitest/leaks.py > > GL > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Finding windows and docshells leaked until shutdown
Not an expert in leak-hunting, but generally this happens every time you leave something like an observer or such registered and you don't de-register it where the page goes away. It'd keep all the window alive. Probably can also happen with some native event listeners or what not... A recent example if it may help: This patch got backed out due to leaking the layout debugger window: https://hg.mozilla.org/integration/autoland/rev/3f8c2c77cd63 This is the fixed version: https://hg.mozilla.org/integration/autoland/rev/ddfb38a12cdf And this the interdiff: https://phabricator.services.mozilla.com/D47715?vs=170954&id=171025 That one was pretty obvious to fix, but yeah, tracking pre-existing ones is tricky... I think the best way to do that is using: https://github.com/amccreight/heapgraph On the logs generated using: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/GC_and_CC_logs (There are docs there about how to generate those from automation) This is useful to find leaks of individual objects too, where you may want to find where the object is allocated rather than what keeps it alive: https://developer.mozilla.org/en-US/docs/Mozilla/Performance/BloatView But I guess that's not all that useful for chasing window leaks. I'm not an expert on these tools myself, hopefully Andrew or one of the people with more experience with them can help more... Hope it helps, -- Emilio On 10/2/19 11:29 PM, Geoff Lankow wrote: Hi all, I need some advice. I'm trying to enable Mochitest on debug builds of Thunderbird. It works fine, except for the leak check, which finds a number of leaked windows and docshells. Obviously we don't want to leak these things, but I can't find what's holding onto them. I've ruled out a number of likely candidates and I'm quickly running out of ideas. Can you offer any tips or tricks that might help? Here's an example log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=267576044&repo=try-comm-central&lineNumber=8051 GL ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
