Hi Ali, Here is the valgrind report after enabling track-origins
==17391== Memcheck, a memory error detector ==17391== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==17391== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==17391== Command: build/ARM/gem5.opt configs/run/run.py --bench=spec.gcc --caches --cpu-type=detailed -F 1000 -W 1000 -s 1 -I 10000 ==17391== Parent PID: 4754 ==17391== ==17391== Warning: set address range perms: large range [0x393e9000, 0xb93e9000) (defined) ==17391== Invalid read of size 2 ==17391== at 0x192BCF4: EventQueue::serviceOne() (flags.hh:63) ==17391== by 0x143AA3F: BaseSimpleCPU::preExecute() (eventq.hh:399) ==17391== by 0x14276AF: AtomicSimpleCPU::tick() (atomic.cc:492) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== by 0x18EEA70: _wrap_simulate (event_wrap.cc:4605) ==17391== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) ==17391== by 0x4F28EF2: PyEval_EvalCodeEx (ceval.c:3000) ==17391== by 0x4F26C18: PyEval_EvalFrameEx (ceval.c:3846) ==17391== by 0x4F28054: PyEval_EvalFrameEx (ceval.c:3836) ==17391== by 0x4F28EF2: PyEval_EvalCodeEx (ceval.c:3000) ==17391== by 0x4F28F51: PyEval_EvalCode (ceval.c:541) ==17391== Address 0x71ce1a2 is 34 bytes inside a block of size 80 free'd ==17391== at 0x4A20F40: operator delete(void*) (vg_replace_malloc.c:480) ==17391== by 0x197462F: SimLoopExitEvent::~SimLoopExitEvent() (sim_events.hh:40) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x143AA3F: BaseSimpleCPU::preExecute() (eventq.hh:399) ==17391== by 0x14276AF: AtomicSimpleCPU::tick() (atomic.cc:492) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== by 0x18EEA70: _wrap_simulate (event_wrap.cc:4605) ==17391== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) ==17391== by 0x4F28EF2: PyEval_EvalCodeEx (ceval.c:3000) ==17391== by 0x4F26C18: PyEval_EvalFrameEx (ceval.c:3846) ==17391== by 0x4F28054: PyEval_EvalFrameEx (ceval.c:3836) ==17391== ==17391== Invalid read of size 2 ==17391== at 0x192BCF4: EventQueue::serviceOne() (flags.hh:63) ==17391== by 0x143AA3F: BaseSimpleCPU::preExecute() (eventq.hh:399) ==17391== by 0x1431C7C: TimingSimpleCPU::completeIfetch(Packet*) (timing.cc:659) ==17391== by 0x1434DFB: TimingSimpleCPU::IcachePort::recvTimingResp(Packet*) (timing.cc:723) ==17391== by 0x16A5DBA: PacketQueue::trySendTiming() (packet_queue.cc:152) ==17391== by 0x16A83CC: PacketQueue::sendDeferredPacket() (packet_queue.cc:190) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== by 0x18EEA70: _wrap_simulate (event_wrap.cc:4605) ==17391== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) ==17391== by 0x4F28EF2: PyEval_EvalCodeEx (ceval.c:3000) ==17391== by 0x4F26C18: PyEval_EvalFrameEx (ceval.c:3846) ==17391== Address 0xa200e62 is 34 bytes inside a block of size 80 free'd ==17391== at 0x4A20F40: operator delete(void*) (vg_replace_malloc.c:480) ==17391== by 0x197462F: SimLoopExitEvent::~SimLoopExitEvent() (sim_events.hh:40) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x143AA3F: BaseSimpleCPU::preExecute() (eventq.hh:399) ==17391== by 0x1431C7C: TimingSimpleCPU::completeIfetch(Packet*) (timing.cc:659) ==17391== by 0x1434DFB: TimingSimpleCPU::IcachePort::recvTimingResp(Packet*) (timing.cc:723) ==17391== by 0x16A5DBA: PacketQueue::trySendTiming() (packet_queue.cc:152) ==17391== by 0x16A83CC: PacketQueue::sendDeferredPacket() (packet_queue.cc:190) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== by 0x18EEA70: _wrap_simulate (event_wrap.cc:4605) ==17391== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) ==17391== ==17391== Invalid read of size 2 ==17391== at 0x192BCF4: EventQueue::serviceOne() (flags.hh:63) ==17391== by 0x12B591C: FullO3CPU<O3CPUImpl>::instDone(short, RefCountingPtr<BaseO3DynInst<O3CPUImpl> >&) (eventq.hh:399) ==17391== by 0x129292A: DefaultCommit<O3CPUImpl>::updateComInstStats(RefCountingPtr<BaseO3DynInst<O3 CPUImpl> >&) (commit_impl.hh:1344) ==17391== by 0x12A227A: DefaultCommit<O3CPUImpl>::commitHead(RefCountingPtr<BaseO3DynInst<O3CPUImpl> >&, unsigned int) (commit_impl.hh:1179) ==17391== by 0x12AAB8E: DefaultCommit<O3CPUImpl>::commitInsts() (commit_impl.hh:982) ==17391== by 0x12AF3E2: DefaultCommit<O3CPUImpl>::commit() (commit_impl.hh:878) ==17391== by 0x12B01F7: DefaultCommit<O3CPUImpl>::tick() (commit_impl.hh:660) ==17391== by 0x12DC7DE: FullO3CPU<O3CPUImpl>::tick() (cpu.cc:601) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== by 0x18EEAAC: _wrap_simulate (event_wrap.cc:4587) ==17391== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) ==17391== Address 0x7191a72 is 34 bytes inside a block of size 80 free'd ==17391== at 0x4A20F40: operator delete(void*) (vg_replace_malloc.c:480) ==17391== by 0x197462F: SimLoopExitEvent::~SimLoopExitEvent() (sim_events.hh:40) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x12B591C: FullO3CPU<O3CPUImpl>::instDone(short, RefCountingPtr<BaseO3DynInst<O3CPUImpl> >&) (eventq.hh:399) ==17391== by 0x129292A: DefaultCommit<O3CPUImpl>::updateComInstStats(RefCountingPtr<BaseO3DynInst<O3 CPUImpl> >&) (commit_impl.hh:1344) ==17391== by 0x12A227A: DefaultCommit<O3CPUImpl>::commitHead(RefCountingPtr<BaseO3DynInst<O3CPUImpl> >&, unsigned int) (commit_impl.hh:1179) ==17391== by 0x12AAB8E: DefaultCommit<O3CPUImpl>::commitInsts() (commit_impl.hh:982) ==17391== by 0x12AF3E2: DefaultCommit<O3CPUImpl>::commit() (commit_impl.hh:878) ==17391== by 0x12B01F7: DefaultCommit<O3CPUImpl>::tick() (commit_impl.hh:660) ==17391== by 0x12DC7DE: FullO3CPU<O3CPUImpl>::tick() (cpu.cc:601) ==17391== by 0x192BCF3: EventQueue::serviceOne() (eventq.cc:204) ==17391== by 0x19765D2: simulate(unsigned long) (simulate.cc:85) ==17391== ==17391== ==17391== HEAP SUMMARY: ==17391== in use at exit: 20,440,699 bytes in 66,415 blocks ==17391== total heap usage: 484,626 allocs, 418,211 frees, 121,518,828 bytes allocated ==17391== ==17391== LEAK SUMMARY: ==17391== definitely lost: 983,936 bytes in 714 blocks ==17391== indirectly lost: 615,510 bytes in 5,112 blocks ==17391== possibly lost: 1,820,292 bytes in 16,479 blocks ==17391== still reachable: 17,020,961 bytes in 44,110 blocks ==17391== suppressed: 0 bytes in 0 blocks ==17391== Rerun with --leak-check=full to see details of leaked memory ==17391== ==17391== For counts of detected and suppressed errors, rerun with: -v ==17391== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 3823 from 251) Best, Rio Xiangyu -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ali Saidi Sent: Sunday, September 30, 2012 8:31 PM To: gem5 Developer List Subject: Re: [gem5-dev] Intermittent segmentation fault + Valgrind reports "Invalid read" Hi Rio, It looks like you're setting a maximum number of instructions on a thread to execute and the problem occurs when the comInstEventQueue is being checked to see if it has reached that number of instructions. My guess is that the event is created, put on the queue, deleted, but some how never removed from the queue between the fast forwarding and the warmup. Could you run valgrind with --track-origns=yes and see if it provides any more detail? Thanks, Ali On Sep 28, 2012, at 7:18 PM, Rio Xiangyu Dong wrote: > Hi there, > > > > I use gem5 run large-scale batch of simulations. I start to see > intermittent segmentation fault after the SE/FS merge. The simulation > stopped by segmentation fault usually go through one or two rounds of > re-simulations. > > > > Debugging this segmentation fault is extremely hard to me. Especially > it seems to me that debugging on gdb will hide such segmentation > fault. So, I believe it's some kinds of "Invalid read" problem. > Debugging the binary using valgrind somehow validates my guess, I got > 3 "Invalid read" reported by Valgrind: > > > > My command line is: valgrind --log-file=report5 > --suppressions=valgrind-python.supp build/ARM/gem5.opt > configs/run/run.py --bench=spec.gcc --caches --cpu-type=detailed -F > 1000 -W 1000 -s 1 -I 10000 > > (run.py is my customized se.py scripts which automatically loads > benchmark, just that, no other stuffs in it) > > > > ==17033== Invalid read of size 2 > > ==17033== at 0x4B9774: Flags<unsigned short>::isSet(unsigned short) const > (flags.hh:63) > > ==17033== by 0x1AD5CE2: Event::isExitEvent() const (eventq.hh:291) > > ==17033== by 0x1B28013: EventQueue::serviceOne() (eventq.cc:205) > > ==17033== by 0x14DBA72: EventQueue::serviceEvents(unsigned long) > (eventq.hh:399) > > ==17033== by 0x162E34F: BaseSimpleCPU::preExecute() (base.cc:364) > > ==17033== by 0x161D32A: AtomicSimpleCPU::tick() (atomic.cc:492) > > ==17033== by 0x1618CFF: AtomicSimpleCPU::TickEvent::process() > (atomic.cc:72) > > ==17033== by 0x1B28007: EventQueue::serviceOne() (eventq.cc:204) > > ==17033== by 0x1B74180: simulate(unsigned long) (simulate.cc:85) > > ==17033== by 0x1AD48D2: _wrap_simulate__SWIG_1 (event_wrap.cc:4605) > > ==17033== by 0x1AD49AF: _wrap_simulate (event_wrap.cc:4628) > > ==17033== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) > > > > ==17033== Invalid read of size 2 > > ==17033== at 0x4B9774: Flags<unsigned short>::isSet(unsigned short) const > (flags.hh:63) > > ==17033== by 0x1B28079: EventQueue::serviceOne() (eventq.cc:213) > > ==17033== by 0x14DBA72: EventQueue::serviceEvents(unsigned long) > (eventq.hh:399) > > ==17033== by 0x162E34F: BaseSimpleCPU::preExecute() (base.cc:364) > > ==17033== by 0x161D32A: AtomicSimpleCPU::tick() (atomic.cc:492) > > ==17033== by 0x1618CFF: AtomicSimpleCPU::TickEvent::process() > (atomic.cc:72) > > ==17033== by 0x1B28007: EventQueue::serviceOne() (eventq.cc:204) > > ==17033== by 0x1B74180: simulate(unsigned long) (simulate.cc:85) > > ==17033== by 0x1AD48D2: _wrap_simulate__SWIG_1 (event_wrap.cc:4605) > > ==17033== by 0x1AD49AF: _wrap_simulate (event_wrap.cc:4628) > > ==17033== by 0x4F26F9E: PyEval_EvalFrameEx (ceval.c:4060) > > ==17033== by 0x4F28EF2: PyEval_EvalCodeEx (ceval.c:3000) > > > > ==17033== Invalid read of size 2 > > ==17033== at 0x4B9774: Flags<unsigned short>::isSet(unsigned short) const > (flags.hh:63) > > ==17033== by 0x1B28079: EventQueue::serviceOne() (eventq.cc:213) > > ==17033== by 0x14DBA72: EventQueue::serviceEvents(unsigned long) > (eventq.hh:399) > > ==17033== by 0x14F7A14: FullO3CPU<O3CPUImpl>::instDone(short, > RefCountingPtr<BaseO3DynInst<O3CPUImpl> >&) (cpu.cc:1497) > > ==17033== by 0x14D6BA0: > DefaultCommit<O3CPUImpl>::updateComInstStats(RefCountingPtr<BaseO3DynI > nst<O3 > CPUImpl> >&) (commit_impl.hh:1344) > > ==17033== by 0x14D28BF: > DefaultCommit<O3CPUImpl>::commitHead(RefCountingPtr<BaseO3DynInst<O3CP > UImpl> >> &, unsigned int) (commit_impl.hh:1179) > > ==17033== by 0x14CDB1D: DefaultCommit<O3CPUImpl>::commitInsts() > (commit_impl.hh:982) > > ==17033== by 0x14C7DAA: DefaultCommit<O3CPUImpl>::commit() > (commit_impl.hh:878) > > ==17033== by 0x14C4A28: DefaultCommit<O3CPUImpl>::tick() > (commit_impl.hh:660) > > ==17033== by 0x14E9AF8: FullO3CPU<O3CPUImpl>::tick() (cpu.cc:601) > > ==17033== by 0x14FC5F5: FullO3CPU<O3CPUImpl>::TickEvent::process() > (cpu.cc:139) > > ==17033== by 0x1B28007: EventQueue::serviceOne() (eventq.cc:204) > > > > It seems to me that there are some read-after-delete accesses in > serviceOne() or process(). Can anyone take a look into it? > > > > BTW, I always follow the latest update of gem5 repo, my local revision > is 9267. > > > > Thank you! > > > > Best, > > Xiangyu > > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
