Hi Andreas, Thanks for figuring this out. I'm a little confused about how starting lots of small processes caused the explosion though... are the page caches allocated per process or per address space (or per core, or something else)? My underlying concern is whether the explosion could be dealt with by better sharing of decode pages rather than (or in addition to) limiting the number of cached pages.
Steve On May 16, 2013 1:50 AM, "Andreas Sandberg" <[email protected]> wrote: > Hi Joel, > > You're right that this problem isn't huge if restoring from a checkpoint > since you start from a clean code cache when restoring. The problem I had > was that I never got to the point where I wanted to take the checkpoint > because I was running out of memory. > > After applying my fix [1], memory usage stabilized at around the guest > memory size + 1GiB instead of growing to more than 8 GiB (that's when gem5 > got killed). > > BTW, you probably want to check out the heap checker in tcmalloc. It's > /way/ faster than Valgrind. > > //Andreas > > [1] https://github.com/andysan/**gem5/commit/** > 99e25dfe196f2fc5edb250024a8bb2**3bb5a5fe7d<https://github.com/andysan/gem5/commit/99e25dfe196f2fc5edb250024a8bb23bb5a5fe7d> > > On 05/15/2013 08:12 PM, Joel Hestness wrote: > >> Hey guys, >> To add a little detail to this, when I cleaned up the memory leaks in >> the >> RubyPort a few weeks ago, I was using Valgrind to find memory errors and >> leaks. I saw numerous decode cache entries reported by Valgrind that >> appear >> to never get cleaned up. The Valgrind output for a record is of the >> following form: >> >> ==18427== 24,016 bytes in 158 blocks are possibly lost in loss record >> 5,121 >> of 5,272 >> ==18427== at 0x4C2B1C7: operator new(unsigned long) (in >> /usr/lib/valgrind/vgpreload_**memcheck-amd64-linux.so) >> ==18427== by 0x68BC01: >> X86ISAInst::X86Macroop::CALL_**NEAR_I::CALL_NEAR_I(X86ISA::**ExtMachInst, >> X86ISA::EmulEnv) (decoder.cc:33404) >> ==18427== by 0x551C0C: X86ISA::Decoder::decodeInst(** >> X86ISA::ExtMachInst) >> (decoder.cc:108772) >> ==18427== by 0x40D393: X86ISA::Decoder::decode(**X86ISA::ExtMachInst, >> unsigned long) (decoder.cc:471) >> ==18427== by 0x40E0E6: X86ISA::Decoder::decode(**X86ISA::PCState&) >> (decoder.cc:516) >> ==18427== by 0x1329AFD: BaseSimpleCPU::preExecute() (base.cc:392) >> ==18427== by 0x131DC85: TimingSimpleCPU::**completeIfetch(Packet*) >> (timing.cc:658) >> ==18427== by 0xCF5DBB: EventQueue::serviceOne() (eventq.cc:207) >> ==18427== by 0xD43392: simulate(unsigned long) (simulate.cc:72) >> ==18427== by 0xAEF414: _wrap_simulate (event_wrap.cc:4798) >> ==18427== by 0x54E73B7: PyEval_EvalFrameEx (in >> /usr/lib/libpython2.7.so.1.0) >> ==18427== by 0x54B2604: PyEval_EvalCodeEx (in >> /usr/lib/libpython2.7.so.1.0) >> >> From a cursory skim, it looks like Valgrind classifies all of these as >> "possibly lost" records, and these records account for about half of all >> the records in a reasonably long simulation. According to the memory size >> of each record, these decode cache entries constitute about 10% of all >> "possibly lost" memory by capacity (~800kB out of 8.4MB possibly lost >> total). >> >> @Andreas: The runs that I'm reporting results from here are when restoring >> from a checkpoint, so I don't see any bloat that might be caused by Linux >> boot. This suggests the problem may not be terrible when restoring from a >> checkpoint. >> >> Joel >> >> >> On Wed, May 15, 2013 at 8:58 AM, Andreas Sandberg<[email protected].** >> se <[email protected]>>wrote: >> >> Hi Everyone, >>> >>> I recently started experimenting with a new x86 user space (based on >>> Debian Wheezy) and ran into a problem with the decode cache. It seems >>> like >>> the boot process (probably udev) is starting a lot of small processes, >>> which causes the decode cache size to explode since it stores every >>> single >>> static instruction that the decoder encounters. On my system, I ended up >>> using more than 8 GiB of memory long before the the boot process >>> completed. >>> >>> My current solution is to set an upper limit on the decode cache size and >>> when that is reached, I randomly remove half of the entries. In practice, >>> it might be better to completely flush the cache since that keeps the >>> code >>> a bit simpler. >>> >>> Does anyone have any opinions about this? Gabe? >>> >>> I'll push my proposed fix to my fixes branch [1] on GitHub later today if >>> anyone wants to have a look. >>> >>> //Andreas >>> >>> [1] >>> https://github.com/andysan/****gem5/tree/fixes<https://github.com/andysan/**gem5/tree/fixes> >>> <https://**github.com/andysan/gem5/tree/**fixes<https://github.com/andysan/gem5/tree/fixes> >>> > >>> >>> ______________________________****_________________ >>> gem5-dev mailing list >>> [email protected] >>> http://m5sim.org/mailman/****listinfo/gem5-dev<http://m5sim.org/mailman/**listinfo/gem5-dev> >>> <http://**m5sim.org/mailman/listinfo/**gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> >>> > >>> >>> >> >> > ______________________________**_________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
