This is on an chromebook based on the RK3399 with only ~4GB of RAM which is not ideal, although we have a bigger machine in the works for the future. I agree with your reasoning and don't think option 1 is a problem. We're using static DTBs so I don't think that's an issue either. In my script, I'm not doing anything smart with the event queues, so that's likely at least part of the problem. When I tried using fs_bigLITTLE.py I ran into what looked like a similar issue so that might not be the whole story, but it's definitely something I should fix up. I'll let you know how that goes!
Gabe On Mon, Mar 19, 2018 at 4:30 AM, Andreas Sandberg <[email protected]> wrote: > Hmm, OK, this is very strange. > > What type of hardware are you running on? Is it an A57-based chip or > something else? Also, what's your simulation quantum? I have been able to > run with a 0.5ms quantum (5e8 ticks). > I think the following trace of two CPUs running in KVM should be roughly > equivalent to the trace you shared earlier. It was generated on a > commercially available 8xA57 (16GiB ram) using the following command (gem5 > rev 9dc44b417): > > gem5.opt -r --debug-flags Kvm,KvmIO,KvmRun > configs/example/arm/fs_bigLITTLE.py \ > --sim-quantum '0.5ms' \ > --cpu-type kvm --big-cpus 0 --little-cpus 2 \ > --dtb system/arm/dt/armv8_gem5_v1_2cpu.dtb --kernel > vmlinux.aarch64.4.4-d318f95d0c > > Note that the tick counts are a bit weird since we have three different > event queues at play (1 for devices and one per CPU). > > 0: system.littleCluster.cpus0: KVM: Executing for 500000000 ticks > 0: system.littleCluster.cpus1: KVM: Executing for 500000000 ticks > 0: system.littleCluster.cpus0: KVM: Executed 79170 instructions in > 176363 cycles (88181504 ticks, sim cycles: 176363). > 88182000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6) > 88182000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr: > 0x1c090024, len: 4) > 88332000: system.littleCluster.cpus0: Entering KVM... > 88332000: system.littleCluster.cpus0: KVM: Executing for 411668000 ticks > 88332000: system.littleCluster.cpus0: KVM: Executed 4384 instructions in > 16854 cycles (8427000 ticks, sim cycles: 16854). > 96759000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6) > 96759000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr: > 0x1c090030, len: 4) > 0: system.littleCluster.cpus1: KVM: Executed 409368 instructions in > 666400 cycles (333200000 ticks, sim cycles: 666400). > 333200000: system.littleCluster.cpus1: Entering KVM... > 333200000: system.littleCluster.cpus1: KVM: Executing for 166800000 ticks > 96909000: system.littleCluster.cpus0: Entering KVM... > 96909000: system.littleCluster.cpus0: KVM: Executing for 403091000 ticks > 96909000: system.littleCluster.cpus0: KVM: Executed 4384 instructions in > 15257 cycles (7628500 ticks, sim cycles: 15257). > 104538000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6) > 104538000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr: > 0x1c0100a0, len: 4) > 333200000: system.littleCluster.cpus1: KVM: Executed 47544 instructions in > 200820 cycles (100410000 ticks, sim cycles: 200820). > 433610000: system.littleCluster.cpus1: Entering KVM... > 433610000: system.littleCluster.cpus1: KVM: Executing for 66390000 ticks > 104688000: system.littleCluster.cpus0: Entering KVM... > 104688000: system.littleCluster.cpus0: KVM: Executing for 395312000 ticks > 104688000: system.littleCluster.cpus0: KVM: Executed 4382 instructions in > 14942 cycles (7471000 ticks, sim cycles: 14942). > > Comparing this trace to yours, I'd say that there the frequent KVM exits > look a bit suspicious. I would expect secondary CPUs to make very little > process while the main CPU initializes the system and starts the early boot > code. > > There area couple of possibilities that might be causing issues: > > 1) There is some CPU ID weirdness that confuses the boot code and puts > both CPUs in the holding pen. This seems unlikely since there are some > writes to the UART. > > 2) Some device is incorrectly mapped to the CPU event queues and causes > frequent KVM exits. Have a look at _build_kvm in fs_bigLITTLE.py, it > doesn't use configs/common, so no need to tear your eyes out. ;) Do you map > event queues in the same way? It's mapping all simulated devices to one > event queue and the CPUs to private event queues. It's important to remap > CPU child devices to the device queue instead of the CPU queue. Failing to > do this will cause chaos, madness, and quite possibly result in Armageddon. > > 3) You're using DTB autogeneration. This doesn't work for KVM guests due > to issues with the timer interrupt specification. We have a patch for the > timer that we are testing internally. Sorry. :( > > Regards, > Andreas > On 16/03/2018 23:20, Gabe Black wrote: > > Ok, diving into this a little deeper, it looks like execution is > progressing but is making very slow progress for some reason. I added a > call to "dump()" before each ioctl invocation which enters the VM and > looked at the PC to get an idea of what it was up to. I made sure to put > that before the timers to avoid taking up VM time with printing debug > stuff. In any case, I see that neither CPU gets off of PC 0 for about 2ms > simulated time (~500Hz), and that's EXTREMELY slow for a CPU which is > supposed to be running in the ballpark of 2GHz. It's not clear to me why > it's making such slow progress, but that would explain why I'm getting very > little out on the simulated console. It's just taking forever to make it > that far. > > Any idea why it's going so slow, or how to debug further? > > Gabe > > On Wed, Mar 14, 2018 at 7:42 PM, Gabe Black <[email protected]> wrote: > >> Some output which I think is suspicious: >> >> 55462000: system.cpus0: Entering KVM... >> 55462000: system.cpus0: KVM: Executing for 1506000 ticks >> 55462000: system.cpus0: KVM: Executed 5159 instructions in 13646 cycles >> (6823000 ticks, sim cycles: 13646). >> 56968000: system.cpus1: Entering KVM... >> 56968000: system.cpus1: KVM: Executing for 5317000 ticks >> 56968000: system.cpus1: KVM: Executed 7229 instructions in 14379 cycles >> (7189500 ticks, sim cycles: 14379). >> 62285000: system.cpus0: Entering KVM... >> 62285000: system.cpus0: KVM: Executing for 1872500 ticks >> 62285000: system.cpus0: KVM: Executed 5159 instructions in 13496 cycles >> (6748000 ticks, sim cycles: 13496). >> 64157500: system.cpus1: Entering KVM... >> 64157500: system.cpus1: KVM: Executing for 4875500 ticks >> 64157500: system.cpus1: KVM: Executed 6950 instructions in 13863 cycles >> (6931500 ticks, sim cycles: 13863). >> 69033000: system.cpus0: Entering KVM... >> 69033000: system.cpus0: KVM: Executing for 2056000 ticks >> 69033000: system.cpus0: KVM: Executed 5159 instructions in 13454 cycles >> (6727000 ticks, sim cycles: 13454). >> 71089000: system.cpus1: Entering KVM... >> 71089000: system.cpus1: KVM: Executing for 4671000 ticks >> 71089000: system.cpus1: KVM: Executed 6950 instructions in 13861 cycles >> (6930500 ticks, sim cycles: 13861). >> 75760000: system.cpus0: Entering KVM... >> 75760000: system.cpus0: KVM: Executing for 2259500 ticks >> 75760000: system.cpus0: KVM: Executed 5159 instructions in 13688 cycles >> (6844000 ticks, sim cycles: 13688). >> >> [...] >> >> 126512000: system.cpus0: handleKvmExit (exit_reason: 6) >> 126512000: system.cpus0: KVM: Handling MMIO (w: 1, addr: 0x1c090024, len: >> 4) >> 126512000: system.cpus0: In updateThreadContext(): >> >> [...] >> >> 126512000: system.cpus0: PC := 0xd8 (t: 0, a64: 1) >> >> On Wed, Mar 14, 2018 at 7:37 PM, Gabe Black <[email protected]> wrote: >> >>> I tried it just now, and I still don't see anything on the console. I >>> switched back to using my own script since it's a bit simpler (it doesn't >>> use all the configs/common stuff), and started looking at the KVM debug >>> output. I see that both cpus claim to execute instructions, although cpu1 >>> didn't take an exit in the output I was looking at. cpu0 took four exits, >>> two which touched some UART registers, and two which touched RealView >>> registes, the V2M_SYS_CFGDATA and V2M_SYS_CFGCTRL registers judging by the >>> comments in the bootloader assembly file. >>> >>> After that they claim to be doing stuff, although I see no further >>> console output or KVM exits. The accesses themselves and their PCs are from >>> the bootloader blob, and so I'm pretty confident that it's starting that >>> and executing some of those instructions. One thing that looks very odd now >>> that I think about it, is that the KVM messages about entering and >>> executing instructions (like those below) seem to say that cpu0 has >>> executed thousands of instructions, but the exits I see seem to correspond >>> to the first maybe 50 instructions it should be seeing in the bootloader >>> blob. Are those values bogus for some reason? Is there some existing debug >>> output which would let me see where KVM thinks it is periodically to see if >>> it's in the kernel or if it went bananas and is executing random memory >>> somewhere? Or if it just got stuck waiting for some event that's not going >>> to show up? >>> >>> Are there any important CLs which haven't made their way into upstream >>> somehow? >>> >>> Gabe >>> >>> On Wed, Mar 14, 2018 at 4:28 AM, Andreas Sandberg < >>> [email protected]> wrote: >>> >>>> Have you tried using the fs_bigLITTLE script in configs/examples/arm? >>>> That's the script I have been using for testing. >>>> >>>> I just tested the script with 8 little CPUs and 0 big CPUs and it seems >>>> to work. Timing is a bit temperamental though, so you might need to >>>> override the simulation quantum. The default is 1ms, you might need to >>>> decrease it to something slightly smaller (I'm currently using 0.5ms). >>>> Another caveat is that there seem to be some issues related to dtb >>>> auto-generation that affect KVM guests. We are currently testing a >>>> solution for this issue. >>>> >>>> Cheers, >>>> Andreas >>>> >>>> >>>> >>>> On 12/03/2018 22:26, Gabe Black wrote: >>>> >>>>> I'm trying to run in FS mode, to boot android/linux. >>>>> >>>>> Gabe >>>>> >>>>> On Mon, Mar 12, 2018 at 3:26 PM, Dutu, Alexandru < >>>>> [email protected]> >>>>> wrote: >>>>> >>>>> Hi Gabe, >>>>>> >>>>>> Are you running SE or FS mode? >>>>>> >>>>>> Thanks, >>>>>> Alex >>>>>> >>>>>> -----Original Message----- >>>>>> From: gem5-dev [mailto:[email protected]] On Behalf Of Gabe >>>>>> Black >>>>>> Sent: Friday, March 9, 2018 5:46 PM >>>>>> To: gem5 Developer List <[email protected]> >>>>>> Subject: [gem5-dev] Multicore ARM v8 KVM based simulation >>>>>> >>>>>> Hi folks. I have a config script set up where I can run a KVM based >>>>>> ARM v8 >>>>>> simulation just fine when I have a single CPU in it, but when I try >>>>>> running >>>>>> with more than one CPU, it just seems to get lost and not do >>>>>> anything. Is >>>>>> this a configuration that's supported? If so, are there any caveats >>>>>> to how >>>>>> it's set up? I may be missing something simple, but it's not apparent >>>>>> to me >>>>>> at the moment. >>>>>> >>>>>> Gabe >>>>>> _______________________________________________ >>>>>> gem5-dev mailing list >>>>>> [email protected] >>>>>> http://m5sim.org/mailman/listinfo/gem5-dev >>>>>> _______________________________________________ >>>>>> gem5-dev mailing list >>>>>> [email protected] >>>>>> http://m5sim.org/mailman/listinfo/gem5-dev >>>>>> >>>>> _______________________________________________ >>>>> gem5-dev mailing list >>>>> [email protected] >>>>> http://m5sim.org/mailman/listinfo/gem5-dev >>>>> >>>> >>>> IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>>> >>> >>> >> > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
