Hi Shehab, As Jason pointed out, I won’t be surprised if you are having issues with classic caches running workloads that rely on locking mechanisms. Your pthread implementation is possibly using some synchronization variables which requires cache coherence to maintain its correctness, and classic caches (at least for now) doesn’t support that.
Switch to ruby caches (I suggest MESI Two Level to begin with), and given your kernel version you should be getting stable behavior from gem5. Best, On Fri, Sep 6, 2019 at 11:47 AM Jason Lowe-Power <[email protected]> wrote: > Hi Shehab, > > IIRC, there are some issues when using classic caches + x86 + multiple > cores on full system mode. I suggest using Ruby (MESI_two_level or > MOESI_hammer) for FS simulations. > > Jason > > On Fri, Sep 6, 2019 at 11:24 AM Shehab Elsayed <[email protected]> > wrote: > >> My latest experiments are with the classical memory system, but I >> remember trying Ruby and it was not different. I am using kernel 4.8.13 and >> ubuntu-16.04.1-server-amd64 disk image. I am using Pthreads for my Hello >> World program. >> >> >> On Fri, Sep 6, 2019 at 1:13 PM Pouya Fotouhi <[email protected]> >> wrote: >> >>> Hi Shehab, >>> >>> Can you confirm a few details about the configuration you are using? Are >>> you using classic caches or Ruby? What is the kernel version and disk image >>> you are using? What is the implementation of your "multithreaded hello >>> world" (are you using OMP)? >>> >>> Best, >>> >>> On Fri, Sep 6, 2019 at 8:58 AM Shehab Elsayed <[email protected]> >>> wrote: >>> >>>> First of all, thanks for your replies, Ryan and Jason. >>>> >>>> I have already pulled the latest changes by Pouya and the problem >>>> still persists. >>>> >>>> As for checkpointing, I was originally doing exactly what Jason >>>> mentioned and ran into the same problem. I then switched to not >>>> checkpointing just to avoid any problems that might be caused by >>>> checkpointing (if any). My plan was to go back to checkpointing after >>>> proving that it works without it. >>>> >>>> However, the problem doesn't seem to be related to KVM as linux boots >>>> reliable every time. The problem happens after the benchmarks starts >>>> execution and it seems to be happening only when running multiple cores >>>> (>=4). My latest experiments with a single core and 8 threads for the >>>> benchmark seem to be working fine. But once I increase the number of >>>> simulated cores problems happen. >>>> >>>> Also, I have posted a link to the repo I am using to run my tests in a >>>> previous message. I have also added 2 debug traces with the Exec flag for a >>>> working and non-working examples. >>>> >>>> >>>> On Fri, Sep 6, 2019 at 11:28 AM Jason Lowe-Power <[email protected]> >>>> wrote: >>>> >>>>> Hi Shehab, >>>>> >>>>> One quick note: There is *no way* to have deterministic behavior when >>>>> running with KVM. Since you are using the hardware, the underlying host OS >>>>> will influence the execution path of the workload. >>>>> >>>>> To try to narrow down the bug you're seeing, you can try to take a >>>>> checkpoint after booting with KVM. Then, the execution from the checkpoint >>>>> should be deterministic since it is 100% in gem5. >>>>> >>>>> BTW, I doubt you can run the KVM CPU in a VM since this would require >>>>> your hardware and the VM to support nested virtualization. There *is* >>>>> support for this in the Linux kernel, but I don't think it's been widely >>>>> deployed outside of specific cloud environments. >>>>> >>>>> One other note: Pouya has pushed some changes which implement some x86 >>>>> instructions that were causing issues for him. You can try with the >>>>> current >>>>> gem5 mainline to see if that helps. >>>>> >>>>> Cheers, >>>>> Jason >>>>> >>>>> On Fri, Sep 6, 2019 at 8:22 AM Shehab Elsayed <[email protected]> >>>>> wrote: >>>>> >>>>>> That's interesting. Are you using Full System as well? I don't think >>>>>> FS behavior is supposed to be so dependent on the host environment! >>>>>> >>>>>> On Fri, Sep 6, 2019 at 11:16 AM Gambord, Ryan < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I have found that gem5 behavior is sensitive to the execution >>>>>>> environment. I now run gem5 inside an ubuntu vm on qemu and have had >>>>>>> much >>>>>>> more consistent results. I haven't tried running kvm gem5 inside a kvm >>>>>>> qemu >>>>>>> vm, so not sure how that works, but might be worth trying. >>>>>>> >>>>>>> Ryan >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 6, 2019, 08:07 Shehab Elsayed <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I was wondering if anyone is running into the same problem or if >>>>>>>> anyone has any suggestions on how to proceed with debugging this >>>>>>>> problem. >>>>>>>> >>>>>>>> On Mon, Jul 29, 2019 at 4:57 PM Shehab Elsayed < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Sorry for the spam. I just forgot to mention that the system >>>>>>>>> configuration I am using is mainly from >>>>>>>>> https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs. >>>>>>>>> <https://github.com/darchr/gem5/tree/jason/kvm-testing/configs/myconfigs> >>>>>>>>> >>>>>>>>> >>>>>>>>> Shehab Y. Elsayed, MSc. >>>>>>>>> PhD Student >>>>>>>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer >>>>>>>>> Engineering >>>>>>>>> University of Toronto >>>>>>>>> E-mail: [email protected] >>>>>>>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Jul 29, 2019 at 4:08 PM Shehab Elsayed < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I have set up a repo with gem5 that demonstrates the problem. The >>>>>>>>>> repo includes the latest version of gem5 from gem5's github repo >>>>>>>>>> with a few >>>>>>>>>> patches applied to get KVM working together with the kernel binary >>>>>>>>>> and disk >>>>>>>>>> image I am using. You can get the repo at >>>>>>>>>> https://github.com/ShehabElsayed/gem5_debug.git. >>>>>>>>>> <https://github.com/ShehabElsayed/gem5_debug.git> >>>>>>>>>> >>>>>>>>>> These steps should reproduce the problem: >>>>>>>>>> 1- scons build/X86/gem5.opt >>>>>>>>>> 2- ./scripts/get_fs_stuff.sh >>>>>>>>>> 3- ./scripts/run_fs.sh 8 >>>>>>>>>> >>>>>>>>>> I have also included sample m5term outputs for both a 2 thread >>>>>>>>>> run (m5out_2t) and an 8 thread run (m5out_8t) >>>>>>>>>> >>>>>>>>>> Any help is really appreciated. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 23, 2019 at 11:01 AM Shehab Elsayed < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> When I enable the Exec debug flag I can see that it seems to be >>>>>>>>>>> stuck in a spin lock (queued_spin_lock_slowpath) >>>>>>>>>>> >>>>>>>>>>> On Fri, Jul 19, 2019 at 5:28 PM Shehab Elsayed < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello All, >>>>>>>>>>>> >>>>>>>>>>>> I have a gem5 X86 full system set up that starts with KVM >>>>>>>>>>>> cores and then switches to O3 cores once the benchmark reaches the >>>>>>>>>>>> region >>>>>>>>>>>> of interest. Right now I am testing with a simple multithreaded >>>>>>>>>>>> hello world benchmark. Sometimes the benchmark completes >>>>>>>>>>>> successfully while >>>>>>>>>>>> others gem5 just seems to hang after starting the benchmark. I >>>>>>>>>>>> believe it >>>>>>>>>>>> is still executing some instructions but without making any >>>>>>>>>>>> progress. The >>>>>>>>>>>> chance of this behavior (indeterminism) happening increases as >>>>>>>>>>>> the number of simulated cores or the number of threads created by >>>>>>>>>>>> the >>>>>>>>>>>> benchmark increases. >>>>>>>>>>>> >>>>>>>>>>>> Any ideas what might be the reason for this or how I can start >>>>>>>>>>>> debugging this problem? >>>>>>>>>>>> >>>>>>>>>>>> Note: I have tried the patch in https://gem5-review. >>>>>>>>>>>> googlesource.com/c/public/gem5/+/19568 but the problem >>>>>>>>>>>> persists. >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>> gem5-users mailing list >>>>>>>> [email protected] >>>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>>> >>>>>>> _______________________________________________ >>>>>>> gem5-users mailing list >>>>>>> [email protected] >>>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>>> >>>>>> _______________________________________________ >>>>>> gem5-users mailing list >>>>>> [email protected] >>>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>> >>>>> _______________________________________________ >>>>> gem5-users mailing list >>>>> [email protected] >>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>> >>>> _______________________________________________ >>>> gem5-users mailing list >>>> [email protected] >>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >>> >>> >>> -- >>> Pouya Fotouhi >>> PhD Candidate >>> Department of Electrical and Computer Engineering >>> University of California, Davis >>> _______________________________________________ >>> gem5-users mailing list >>> [email protected] >>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> >> _______________________________________________ >> gem5-users mailing list >> [email protected] >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users > > _______________________________________________ > gem5-users mailing list > [email protected] > http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- Pouya Fotouhi PhD Candidate Department of Electrical and Computer Engineering University of California, Davis
_______________________________________________ gem5-users mailing list [email protected] http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
