That makes some sense... I agree that we should be robust in the face of ASLR.
I suppose if we wanted to get really complicated we could try and track usage and do some form of LRU eviction, but since the page caches are refilled from the main decode cache anyway, it probably doesn't make a lot of difference. Thanks, Steve On May 16, 2013 1:00 PM, "Andreas Sandberg" <[email protected]> wrote: > Hi Steve, > > I'm not entirely sure of what's going on, but I have a theory that seems > pretty solid. I know that I forgot to disable address space randomization > which means that library addresses (e.g., libc) will be randomized. This > means that a cached translation of a library will never hit in the code > cache. > > If I understand correctly, the code cache is indexed based on the PC > (virtual address only, not process ID or similar) and verified against the > actual instruction. This works fine in most cases since libraries tend to > be loaded at the same address and a tag miss will cause an entry to be > updated with the correct information. This is probably the reason the code > cache doesn't explode in simulations using the example user space. > > I guess you could argue that ASLR should be disabled and that this isn't a > problem in practice, but it is a feature that most modern operating systems > have so we shouldn't fall over when it is enabled. > > //Andreas > > On 05/16/2013 06:08 PM, Steve Reinhardt wrote: > >> Hi Andreas, >> >> Thanks for figuring this out. I'm a little confused about how starting >> lots of small processes caused the explosion though... are the page caches >> allocated per process or per address space (or per core, or something >> else)? My underlying concern is whether the explosion could be dealt with >> by better sharing of decode pages rather than (or in addition to) limiting >> the number of cached pages. >> >> Steve >> On May 16, 2013 1:50 AM, "Andreas Sandberg" <[email protected]> >> wrote: >> >> Hi Joel, >>> >>> You're right that this problem isn't huge if restoring from a checkpoint >>> since you start from a clean code cache when restoring. The problem I had >>> was that I never got to the point where I wanted to take the checkpoint >>> because I was running out of memory. >>> >>> After applying my fix [1], memory usage stabilized at around the guest >>> memory size + 1GiB instead of growing to more than 8 GiB (that's when >>> gem5 >>> got killed). >>> >>> BTW, you probably want to check out the heap checker in tcmalloc. It's >>> /way/ faster than Valgrind. >>> >>> //Andreas >>> >>> [1] >>> https://github.com/andysan/****gem5/commit/**<https://github.com/andysan/**gem5/commit/**> >>> 99e25dfe196f2fc5edb250024a8bb2****3bb5a5fe7d<https://github.** >>> com/andysan/gem5/commit/**99e25dfe196f2fc5edb250024a8bb2**3bb5a5fe7d<https://github.com/andysan/gem5/commit/99e25dfe196f2fc5edb250024a8bb23bb5a5fe7d> >>> > >>> >>> On 05/15/2013 08:12 PM, Joel Hestness wrote: >>> >>> Hey guys, >>>> To add a little detail to this, when I cleaned up the memory leaks >>>> in >>>> the >>>> RubyPort a few weeks ago, I was using Valgrind to find memory errors and >>>> leaks. I saw numerous decode cache entries reported by Valgrind that >>>> appear >>>> to never get cleaned up. The Valgrind output for a record is of the >>>> following form: >>>> >>>> ==18427== 24,016 bytes in 158 blocks are possibly lost in loss record >>>> 5,121 >>>> of 5,272 >>>> ==18427== at 0x4C2B1C7: operator new(unsigned long) (in >>>> /usr/lib/valgrind/vgpreload_****memcheck-amd64-linux.so) >>>> ==18427== by 0x68BC01: >>>> X86ISAInst::X86Macroop::CALL_****NEAR_I::CALL_NEAR_I(X86ISA::*** >>>> *ExtMachInst, >>>> X86ISA::EmulEnv) (decoder.cc:33404) >>>> ==18427== by 0x551C0C: X86ISA::Decoder::decodeInst(** >>>> X86ISA::ExtMachInst) >>>> (decoder.cc:108772) >>>> ==18427== by 0x40D393: X86ISA::Decoder::decode(**** >>>> X86ISA::ExtMachInst, >>>> unsigned long) (decoder.cc:471) >>>> ==18427== by 0x40E0E6: X86ISA::Decoder::decode(****X86ISA::PCState&) >>>> (decoder.cc:516) >>>> ==18427== by 0x1329AFD: BaseSimpleCPU::preExecute() (base.cc:392) >>>> ==18427== by 0x131DC85: TimingSimpleCPU::****completeIfetch(Packet*) >>>> (timing.cc:658) >>>> ==18427== by 0xCF5DBB: EventQueue::serviceOne() (eventq.cc:207) >>>> ==18427== by 0xD43392: simulate(unsigned long) (simulate.cc:72) >>>> ==18427== by 0xAEF414: _wrap_simulate (event_wrap.cc:4798) >>>> ==18427== by 0x54E73B7: PyEval_EvalFrameEx (in >>>> /usr/lib/libpython2.7.so.1.0) >>>> ==18427== by 0x54B2604: PyEval_EvalCodeEx (in >>>> /usr/lib/libpython2.7.so.1.0) >>>> >>>> From a cursory skim, it looks like Valgrind classifies all of these >>>> as >>>> "possibly lost" records, and these records account for about half of all >>>> the records in a reasonably long simulation. According to the memory >>>> size >>>> of each record, these decode cache entries constitute about 10% of all >>>> "possibly lost" memory by capacity (~800kB out of 8.4MB possibly lost >>>> total). >>>> >>>> @Andreas: The runs that I'm reporting results from here are when >>>> restoring >>>> from a checkpoint, so I don't see any bloat that might be caused by >>>> Linux >>>> boot. This suggests the problem may not be terrible when restoring from >>>> a >>>> checkpoint. >>>> >>>> Joel >>>> >>>> >>>> On Wed, May 15, 2013 at 8:58 AM, Andreas Sandberg<[email protected].* >>>> *** >>>> se <[email protected]>>**wrote: >>>> >>>> Hi Everyone, >>>> >>>>> >>>>> I recently started experimenting with a new x86 user space (based on >>>>> Debian Wheezy) and ran into a problem with the decode cache. It seems >>>>> like >>>>> the boot process (probably udev) is starting a lot of small processes, >>>>> which causes the decode cache size to explode since it stores every >>>>> single >>>>> static instruction that the decoder encounters. On my system, I ended >>>>> up >>>>> using more than 8 GiB of memory long before the the boot process >>>>> completed. >>>>> >>>>> My current solution is to set an upper limit on the decode cache size >>>>> and >>>>> when that is reached, I randomly remove half of the entries. In >>>>> practice, >>>>> it might be better to completely flush the cache since that keeps the >>>>> code >>>>> a bit simpler. >>>>> >>>>> Does anyone have any opinions about this? Gabe? >>>>> >>>>> I'll push my proposed fix to my fixes branch [1] on GitHub later today >>>>> if >>>>> anyone wants to have a look. >>>>> >>>>> //Andreas >>>>> >>>>> [1] >>>>> https://github.com/andysan/******gem5/tree/fixes<https://github.com/andysan/****gem5/tree/fixes> >>>>> <https://**github.com/andysan/**gem5/**tree/fixes<https://github.com/andysan/**gem5/tree/fixes> >>>>> > >>>>> <https://**github.com/andysan/**gem5/tree/**fixes<http://github.com/andysan/gem5/tree/**fixes> >>>>> <https://**github.com/andysan/gem5/tree/**fixes<https://github.com/andysan/gem5/tree/fixes> >>>>> > >>>>> >>>>>> >>>>>> >>>>> ______________________________******_________________ >>>>> gem5-dev mailing list >>>>> [email protected] >>>>> http://m5sim.org/mailman/******listinfo/gem5-dev<http://m5sim.org/mailman/****listinfo/gem5-dev> >>>>> <http://**m5sim.org/mailman/**listinfo/**gem5-dev<http://m5sim.org/mailman/**listinfo/gem5-dev> >>>>> > >>>>> <http://**m5sim.org/mailman/**listinfo/**gem5-dev<http://m5sim.org/mailman/listinfo/**gem5-dev> >>>>> <http://**m5sim.org/mailman/listinfo/**gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> >>>>> > >>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> ______________________________****_________________ >>> gem5-dev mailing list >>> [email protected] >>> http://m5sim.org/mailman/****listinfo/gem5-dev<http://m5sim.org/mailman/**listinfo/gem5-dev> >>> <http://**m5sim.org/mailman/listinfo/**gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> >>> > >>> >>> ______________________________**_________________ >> gem5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> >> >> > ______________________________**_________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev> > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
