Re: [webkit-dev] Questions regarding memory debugging
On Wednesday 10 March 2010 05:56:27 Holger Freyther wrote: HI All, I'm currently running the iexploder tests on GtkLauncher and I'm observing a growth in memory usage of that process. I have used the GNOME memprof utilitiy to look at it and the best candidate for the leak is SharedBuffer::append which will allocate memory. Zoltan has mentioned gdb on irc and I started to wonder how far we can get with GDB's python support in 7.0. Maybe we could create a set of scripts to track lifetime using GDB? I'm going to explore this option maybe later today but within this week. z. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
On Mar 9, 2010, at 8:56 PM, Holger Freyther wrote: Now SharedBuffer is refcounted and can only leak if someone is holding a reference to it and this is where stuff gets tricky. Besides manually searching the tree and adding printf's all over has anyone found a better way of doing this? One of the best techniques we have found is to detect the leak when it is introduced. A technique we use is to add monitoring to classes for which there should be no instances at process exit, and log a complaint if objects are still around. Then the leak message can be spotted on the check-in that introduces it. I’ve done a lot of manual work with gdb to track down reference count mismatches, reference cycles, and such. At the OpenBSC GSM Project we are using this C library talloc. With talloc every allocation is happening inside a (hierachical) context and at runtime we can send a SIGUSR1 one to the application and the memstatistics are dumped on the console. I have attached an example output so you guys can see it. I wonder if such a thing would make sense in a debug build as well? I wonder if we could change the common Class::create methods to take a context, or at least an area (Render, Network, Platform) and if these get created in a special debug build we add these objects to a list for these contexts and we can dump the information... e.g. # of allocated bytes, ref count and such? Does this sound crazy or too invasive? Changes that involve modifying every allocation are challenging and could take a lot of work to do. There are many, many separate calls to new. But more importantly, how would this help with the SharedBuffer problem? It seems like a separate topic. We have memory tools on Mac OS X that group allocations by the call stack when an object is allocated, and we have come up with scripts that help us put allocations into “areas” as you suggest, without adding code to WebKit. But in cases like the SharedBuffer one, there is no mystery about who allocated the memory. The mystery is typically who is holding on to a reference to it. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
This would be a really great topic for the webkit meeting in April. At Palm we spend a significant amount of development time dealing with memory growth (not so much leaks). Google perf tools has worked very well for us on desktop as it gives the complete call stack showing how the allocation was made. On device, we use a lot of live object instance monitoring. Anyway it would be great to share some tips and tricks with other folks. Greg On 3/10/10 9:51 AM, Darin Adler da...@apple.com wrote: On Mar 9, 2010, at 8:56 PM, Holger Freyther wrote: Now SharedBuffer is refcounted and can only leak if someone is holding a reference to it and this is where stuff gets tricky. Besides manually searching the tree and adding printf's all over has anyone found a better way of doing this? One of the best techniques we have found is to detect the leak when it is introduced. A technique we use is to add monitoring to classes for which there should be no instances at process exit, and log a complaint if objects are still around. Then the leak message can be spotted on the check-in that introduces it. I've done a lot of manual work with gdb to track down reference count mismatches, reference cycles, and such. At the OpenBSC GSM Project we are using this C library talloc. With talloc every allocation is happening inside a (hierachical) context and at runtime we can send a SIGUSR1 one to the application and the memstatistics are dumped on the console. I have attached an example output so you guys can see it. I wonder if such a thing would make sense in a debug build as well? I wonder if we could change the common Class::create methods to take a context, or at least an area (Render, Network, Platform) and if these get created in a special debug build we add these objects to a list for these contexts and we can dump the information... e.g. # of allocated bytes, ref count and such? Does this sound crazy or too invasive? Changes that involve modifying every allocation are challenging and could take a lot of work to do. There are many, many separate calls to new. But more importantly, how would this help with the SharedBuffer problem? It seems like a separate topic. We have memory tools on Mac OS X that group allocations by the call stack when an object is allocated, and we have come up with scripts that help us put allocations into areas as you suggest, without adding code to WebKit. But in cases like the SharedBuffer one, there is no mystery about who allocated the memory. The mystery is typically who is holding on to a reference to it. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
On Mar 10, 2010, at 10:06 AM, Greg Simon wrote: This would be a really great topic for the webkit meeting in April. At Palm we spend a significant amount of development time dealing with memory “growth” (not so much leaks). Google perf tools has worked very well for us on desktop as it gives the complete call stack showing how the allocation was made. On device, we use a lot of live object instance monitoring. Anyway it would be great to share some tips and tricks with other folks. I would love to hear more about it. Did any of this development work lead to bug fixes for WebKit? Please note that the meeting in April is for WebKit contributors. People using WebKit but not yet contributing to the open source project should not attend. I mention this because I don’t remember contributions from engineers working at Palm, although I may have missed them. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
While we post the source code to every release of webOS, we have held back in posting the changes directly to webkit org for several reasons: * webOS versions 1.0.x-1.3.x have been based on webkit3, and the code on webkit-trunk has changed significantly since then * many changes we have made are specific to making our JavaScript framework run fast, and quite frankly I suspect these would be rejected by the group since they are JS framework-specific; i.e., I do not want to waste the group's time. That being said, with webOS 1.4 (released last week) we are now only several months off of trunk, so we plan on being much more active now that we are closer to TOT. We will always lag TOT due to our own internal product requirements. Greg On 3/10/10 10:13 AM, Darin Adler da...@apple.com wrote: On Mar 10, 2010, at 10:06 AM, Greg Simon wrote: This would be a really great topic for the webkit meeting in April. At Palm we spend a significant amount of development time dealing with memory growth (not so much leaks). Google perf tools has worked very well for us on desktop as it gives the complete call stack showing how the allocation was made. On device, we use a lot of live object instance monitoring. Anyway it would be great to share some tips and tricks with other folks. I would love to hear more about it. Did any of this development work lead to bug fixes for WebKit? Please note that the meeting in April is for WebKit contributors. People using WebKit but not yet contributing to the open source project should not attend. I mention this because I don't remember contributions from engineers working at Palm, although I may have missed them. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
But more importantly, how would this help with the SharedBuffer problem? It seems like a separate topic. We have memory tools on Mac OS X that group allocations by the call stack when an object is allocated, and we have come up with scripts that help us put allocations into “areas” as you suggest, without adding code to WebKit. But in cases like the SharedBuffer one, there is no mystery about who allocated the memory. The mystery is typically who is holding on to a reference to it. well, you can use same technique to capture call stack in ref inc/dec? Usually buffers shouldn't be a problem even if widely allocedas you can dump a few bytes and get some idea where it came from and who may have been interested in it. Sometimes I will give objects names and make up the name with some helpful info. Personally I don't like ref counts. Maybe in a project like this they are the easiest way to go but there is no substititue for knowing how long your object is needed. I haven't looked at this code but is SharedBuffer thread safe etc? I thought someone was talking about thread safety of reference counts at some point. -- Darin _ Hotmail: Trusted email with Microsoft’s powerful SPAM protection. http://clk.atdmt.com/GBL/go/201469226/direct/01/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
On Mar 10, 2010, at 11:15 AM, Mike Marchywka wrote: I haven't looked at this code but is SharedBuffer thread safe etc? SharedBuffer uses RefCounted, an implementation that works only on a single thread, as opposed to the slower CrossThreadRefCounted. We can test your guess about the memory problems being a thread-safety issue by adding assertions that check ref and deref are called only on one thread. Those would be extremely valuable. Someone was working on assertions of that type a while back, but I don’t think we successfully landed them. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
On Wed, Mar 10, 2010 at 11:27 AM, Darin Adler da...@apple.com wrote: On Mar 10, 2010, at 11:15 AM, Mike Marchywka wrote: I haven't looked at this code but is SharedBuffer thread safe etc? SharedBuffer uses RefCounted, an implementation that works only on a single thread, as opposed to the slower CrossThreadRefCounted. We can test your guess about the memory problems being a thread-safety issue by adding assertions that check ref and deref are called only on one thread. Those would be extremely valuable. Someone was working on assertions of that type a while back, but I don’t think we successfully landed them. Here's the bug if someone wants to take that up: https://bugs.webkit.org/show_bug.cgi?id=31639 https://bugs.webkit.org/show_bug.cgi?id=31639 -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
But in cases like the SharedBuffer one, there is no mystery about who allocated the memory. The mystery is typically who is holding on to a reference to it. I have a technique to capture ref/deref problems with gdb. When the object is created (inside the object constructor), I just put a write break point to the address of m_refCount: watch this-m_refCount in gdb. The backtrace can be inspected every time when this watch point is hit. Ref / derefs must be in pairs, usually easy to find which ref has no deref pair. Gdb also supports executing scripts when a breakpoint is hit, this could be useful to perform some tasks (see the COMMAND command). Zoltan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Questions regarding memory debugging
On Mar 10, 2010, at 11:48 AM, Zoltan Herczeg wrote: I have a technique to capture ref/deref problems with gdb. When the object is created (inside the object constructor), I just put a write break point to the address of m_refCount: watch this-m_refCount in gdb. The backtrace can be inspected every time when this watch point is hit. Ref / derefs must be in pairs, usually easy to find which ref has no deref pair. Gdb also supports executing scripts when a breakpoint is hit, this could be useful to perform some tasks (see the COMMAND command). I do something similar, putting a breakpoint on both ref and deref with the commands p m_refCount, bt, and continue. Then I paste the output into a text file and find the unbalanced call with searching in the text editor. Usually by deleting ref/deref pairs that I decide are balanced. Until one is left. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] Questions regarding memory debugging
HI All, I'm currently running the iexploder tests on GtkLauncher and I'm observing a growth in memory usage of that process. I have used the GNOME memprof utilitiy to look at it and the best candidate for the leak is SharedBuffer::append which will allocate memory. Now SharedBuffer is refcounted and can only leak if someone is holding a reference to it and this is where stuff gets tricky. Besides manually searching the tree and adding printf's all over has anyone found a better way of doing this? At the OpenBSC GSM Project we are using this C library talloc. With talloc every allocation is happening inside a (hierachical) context and at runtime we can send a SIGUSR1 one to the application and the memstatistics are dumped on the console. I have attached an example output so you guys can see it. I wonder if such a thing would make sense in a debug build as well? I wonder if we could change the common Class::create methods to take a context, or at least an area (Render, Network, Platform) and if these get created in a special debug build we add these objects to a list for these contexts and we can dump the information... e.g. # of allocated bytes, ref count and such? Does this sound crazy or too invasive? comments? talloc report on 'vty' (total 26810 bytes in 2351 blocks) save_cwd contains 37 bytes in 1 blocks (ref 0) 0x92d18d0 vty_commandcontains 15093 bytes in 1180 blocks (ref 0) 0x92d1898 vty_vector contains 11680 bytes in 1169 blocks (ref 0) 0x92d1860 full talloc report on 'openbsc' (total 85562 bytes in 54 blocks) struct ia_e1_handlecontains 60 bytes in 1 blocks (ref 0) 0x92f74a8 telnet_connection contains 1 bytes in 1 blocks (ref 0) 0x92c50a0 struct gsm_network contains 84860 bytes in 5 blocks (ref 0) 0x92c3830 struct gsm_bts contains 84604 bytes in 2 blocks (ref 0) 0x92f7ad8 struct gsm_bts_trx contains 82092 bytes in 1 blocks (ref 0) 0x92f84e0 OpenBSCcontains 8 bytes in 1 blocks (ref 0) 0x92c51f8 OpenBSCcontains 8 bytes in 1 blocks (ref 0) 0x92c2cd0 countercontains500 bytes in 26 blocks (ref 0) 0x92ac568 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c6a78 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2ae8 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2aa0 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2a58 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2a10 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b29c8 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2980 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b2938 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c2c10 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c2bc8 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c2b80 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c2b38 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c2af0 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b3b48 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b3b00 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b3ab8 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b3a70 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92ae140 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92ae0f8 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92ae0b0 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b0250 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b0208 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c5058 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92b0178 struct counter contains 20 bytes in 1 blocks (ref 0) 0x92c5110 trau_upq_entry contains 0 bytes in 1 blocks (ref 0) 0x92ac530 trau_map_entry contains 0 bytes in 1 blocks (ref 0) 0x92ac4f8 transactioncontains 0 bytes in 1 blocks (ref 0)
