Hi Steve,
I'm not entirely sure of what's going on, but I have a theory that seems
pretty solid. I know that I forgot to disable address space
randomization which means that library addresses (e.g., libc) will be
randomized. This means that a cached translation of a library will never
hit in the code cache.
If I understand correctly, the code cache is indexed based on the PC
(virtual address only, not process ID or similar) and verified against
the actual instruction. This works fine in most cases since libraries
tend to be loaded at the same address and a tag miss will cause an entry
to be updated with the correct information. This is probably the reason
the code cache doesn't explode in simulations using the example user space.
I guess you could argue that ASLR should be disabled and that this isn't
a problem in practice, but it is a feature that most modern operating
systems have so we shouldn't fall over when it is enabled.
//Andreas
On 05/16/2013 06:08 PM, Steve Reinhardt wrote:
Hi Andreas,
Thanks for figuring this out. I'm a little confused about how starting
lots of small processes caused the explosion though... are the page caches
allocated per process or per address space (or per core, or something
else)? My underlying concern is whether the explosion could be dealt with
by better sharing of decode pages rather than (or in addition to) limiting
the number of cached pages.
Steve
On May 16, 2013 1:50 AM, "Andreas Sandberg" <[email protected]> wrote:
Hi Joel,
You're right that this problem isn't huge if restoring from a checkpoint
since you start from a clean code cache when restoring. The problem I had
was that I never got to the point where I wanted to take the checkpoint
because I was running out of memory.
After applying my fix [1], memory usage stabilized at around the guest
memory size + 1GiB instead of growing to more than 8 GiB (that's when gem5
got killed).
BTW, you probably want to check out the heap checker in tcmalloc. It's
/way/ faster than Valgrind.
//Andreas
[1] https://github.com/andysan/**gem5/commit/**
99e25dfe196f2fc5edb250024a8bb2**3bb5a5fe7d<https://github.com/andysan/gem5/commit/99e25dfe196f2fc5edb250024a8bb23bb5a5fe7d>
On 05/15/2013 08:12 PM, Joel Hestness wrote:
Hey guys,
To add a little detail to this, when I cleaned up the memory leaks in
the
RubyPort a few weeks ago, I was using Valgrind to find memory errors and
leaks. I saw numerous decode cache entries reported by Valgrind that
appear
to never get cleaned up. The Valgrind output for a record is of the
following form:
==18427== 24,016 bytes in 158 blocks are possibly lost in loss record
5,121
of 5,272
==18427== at 0x4C2B1C7: operator new(unsigned long) (in
/usr/lib/valgrind/vgpreload_**memcheck-amd64-linux.so)
==18427== by 0x68BC01:
X86ISAInst::X86Macroop::CALL_**NEAR_I::CALL_NEAR_I(X86ISA::**ExtMachInst,
X86ISA::EmulEnv) (decoder.cc:33404)
==18427== by 0x551C0C: X86ISA::Decoder::decodeInst(**
X86ISA::ExtMachInst)
(decoder.cc:108772)
==18427== by 0x40D393: X86ISA::Decoder::decode(**X86ISA::ExtMachInst,
unsigned long) (decoder.cc:471)
==18427== by 0x40E0E6: X86ISA::Decoder::decode(**X86ISA::PCState&)
(decoder.cc:516)
==18427== by 0x1329AFD: BaseSimpleCPU::preExecute() (base.cc:392)
==18427== by 0x131DC85: TimingSimpleCPU::**completeIfetch(Packet*)
(timing.cc:658)
==18427== by 0xCF5DBB: EventQueue::serviceOne() (eventq.cc:207)
==18427== by 0xD43392: simulate(unsigned long) (simulate.cc:72)
==18427== by 0xAEF414: _wrap_simulate (event_wrap.cc:4798)
==18427== by 0x54E73B7: PyEval_EvalFrameEx (in
/usr/lib/libpython2.7.so.1.0)
==18427== by 0x54B2604: PyEval_EvalCodeEx (in
/usr/lib/libpython2.7.so.1.0)
From a cursory skim, it looks like Valgrind classifies all of these as
"possibly lost" records, and these records account for about half of all
the records in a reasonably long simulation. According to the memory size
of each record, these decode cache entries constitute about 10% of all
"possibly lost" memory by capacity (~800kB out of 8.4MB possibly lost
total).
@Andreas: The runs that I'm reporting results from here are when restoring
from a checkpoint, so I don't see any bloat that might be caused by Linux
boot. This suggests the problem may not be terrible when restoring from a
checkpoint.
Joel
On Wed, May 15, 2013 at 8:58 AM, Andreas Sandberg<[email protected].**
se <[email protected]>>wrote:
Hi Everyone,
I recently started experimenting with a new x86 user space (based on
Debian Wheezy) and ran into a problem with the decode cache. It seems
like
the boot process (probably udev) is starting a lot of small processes,
which causes the decode cache size to explode since it stores every
single
static instruction that the decoder encounters. On my system, I ended up
using more than 8 GiB of memory long before the the boot process
completed.
My current solution is to set an upper limit on the decode cache size and
when that is reached, I randomly remove half of the entries. In practice,
it might be better to completely flush the cache since that keeps the
code
a bit simpler.
Does anyone have any opinions about this? Gabe?
I'll push my proposed fix to my fixes branch [1] on GitHub later today if
anyone wants to have a look.
//Andreas
[1]
https://github.com/andysan/****gem5/tree/fixes<https://github.com/andysan/**gem5/tree/fixes>
<https://**github.com/andysan/gem5/tree/**fixes<https://github.com/andysan/gem5/tree/fixes>
______________________________****_________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/****listinfo/gem5-dev<http://m5sim.org/mailman/**listinfo/gem5-dev>
<http://**m5sim.org/mailman/listinfo/**gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>
______________________________**_________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/**listinfo/gem5-dev<http://m5sim.org/mailman/listinfo/gem5-dev>
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
