Hmm, OK, this is very strange.
What type of hardware are you running on? Is it an A57-based chip or something
else? Also, what's your simulation quantum? I have been able to run with a
0.5ms quantum (5e8 ticks).
I think the following trace of two CPUs running in KVM should be roughly
equivalent to the trace you shared earlier. It was generated on a commercially
available 8xA57 (16GiB ram) using the following command (gem5 rev 9dc44b417):
gem5.opt -r --debug-flags Kvm,KvmIO,KvmRun configs/example/arm/fs_bigLITTLE.py \
--sim-quantum '0.5ms' \
--cpu-type kvm --big-cpus 0 --little-cpus 2 \
--dtb system/arm/dt/armv8_gem5_v1_2cpu.dtb --kernel
vmlinux.aarch64.4.4-d318f95d0c
Note that the tick counts are a bit weird since we have three different event
queues at play (1 for devices and one per CPU).
0: system.littleCluster.cpus0: KVM: Executing for 500000000 ticks
0: system.littleCluster.cpus1: KVM: Executing for 500000000 ticks
0: system.littleCluster.cpus0: KVM: Executed 79170 instructions in 176363
cycles (88181504 ticks, sim cycles: 176363).
88182000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6)
88182000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr:
0x1c090024, len: 4)
88332000: system.littleCluster.cpus0: Entering KVM...
88332000: system.littleCluster.cpus0: KVM: Executing for 411668000 ticks
88332000: system.littleCluster.cpus0: KVM: Executed 4384 instructions in 16854
cycles (8427000 ticks, sim cycles: 16854).
96759000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6)
96759000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr:
0x1c090030, len: 4)
0: system.littleCluster.cpus1: KVM: Executed 409368 instructions in 666400
cycles (333200000 ticks, sim cycles: 666400).
333200000: system.littleCluster.cpus1: Entering KVM...
333200000: system.littleCluster.cpus1: KVM: Executing for 166800000 ticks
96909000: system.littleCluster.cpus0: Entering KVM...
96909000: system.littleCluster.cpus0: KVM: Executing for 403091000 ticks
96909000: system.littleCluster.cpus0: KVM: Executed 4384 instructions in 15257
cycles (7628500 ticks, sim cycles: 15257).
104538000: system.littleCluster.cpus0: handleKvmExit (exit_reason: 6)
104538000: system.littleCluster.cpus0: KVM: Handling MMIO (w: 1, addr:
0x1c0100a0, len: 4)
333200000: system.littleCluster.cpus1: KVM: Executed 47544 instructions in
200820 cycles (100410000 ticks, sim cycles: 200820).
433610000: system.littleCluster.cpus1: Entering KVM...
433610000: system.littleCluster.cpus1: KVM: Executing for 66390000 ticks
104688000: system.littleCluster.cpus0: Entering KVM...
104688000: system.littleCluster.cpus0: KVM: Executing for 395312000 ticks
104688000: system.littleCluster.cpus0: KVM: Executed 4382 instructions in 14942
cycles (7471000 ticks, sim cycles: 14942).
Comparing this trace to yours, I'd say that there the frequent KVM exits look a
bit suspicious. I would expect secondary CPUs to make very little process while
the main CPU initializes the system and starts the early boot code.
There area couple of possibilities that might be causing issues:
1) There is some CPU ID weirdness that confuses the boot code and puts both
CPUs in the holding pen. This seems unlikely since there are some writes to the
UART.
2) Some device is incorrectly mapped to the CPU event queues and causes
frequent KVM exits. Have a look at _build_kvm in fs_bigLITTLE.py, it doesn't
use configs/common, so no need to tear your eyes out. ;) Do you map event
queues in the same way? It's mapping all simulated devices to one event queue
and the CPUs to private event queues. It's important to remap CPU child devices
to the device queue instead of the CPU queue. Failing to do this will cause
chaos, madness, and quite possibly result in Armageddon.
3) You're using DTB autogeneration. This doesn't work for KVM guests due to
issues with the timer interrupt specification. We have a patch for the timer
that we are testing internally. Sorry. :(
Regards,
Andreas
On 16/03/2018 23:20, Gabe Black wrote:
Ok, diving into this a little deeper, it looks like execution is progressing but is
making very slow progress for some reason. I added a call to "dump()" before
each ioctl invocation which enters the VM and looked at the PC to get an idea of what it
was up to. I made sure to put that before the timers to avoid taking up VM time with
printing debug stuff. In any case, I see that neither CPU gets off of PC 0 for about 2ms
simulated time (~500Hz), and that's EXTREMELY slow for a CPU which is supposed to be
running in the ballpark of 2GHz. It's not clear to me why it's making such slow progress,
but that would explain why I'm getting very little out on the simulated console. It's
just taking forever to make it that far.
Any idea why it's going so slow, or how to debug further?
Gabe
On Wed, Mar 14, 2018 at 7:42 PM, Gabe Black
<[email protected]<mailto:[email protected]>> wrote:
Some output which I think is suspicious:
55462000: system.cpus0: Entering KVM...
55462000: system.cpus0: KVM: Executing for 1506000 ticks
55462000: system.cpus0: KVM: Executed 5159 instructions in 13646 cycles
(6823000 ticks, sim cycles: 13646).
56968000: system.cpus1: Entering KVM...
56968000: system.cpus1: KVM: Executing for 5317000 ticks
56968000: system.cpus1: KVM: Executed 7229 instructions in 14379 cycles
(7189500 ticks, sim cycles: 14379).
62285000: system.cpus0: Entering KVM...
62285000: system.cpus0: KVM: Executing for 1872500 ticks
62285000: system.cpus0: KVM: Executed 5159 instructions in 13496 cycles
(6748000 ticks, sim cycles: 13496).
64157500: system.cpus1: Entering KVM...
64157500: system.cpus1: KVM: Executing for 4875500 ticks
64157500: system.cpus1: KVM: Executed 6950 instructions in 13863 cycles
(6931500 ticks, sim cycles: 13863).
69033000: system.cpus0: Entering KVM...
69033000: system.cpus0: KVM: Executing for 2056000 ticks
69033000: system.cpus0: KVM: Executed 5159 instructions in 13454 cycles
(6727000 ticks, sim cycles: 13454).
71089000: system.cpus1: Entering KVM...
71089000: system.cpus1: KVM: Executing for 4671000 ticks
71089000: system.cpus1: KVM: Executed 6950 instructions in 13861 cycles
(6930500 ticks, sim cycles: 13861).
75760000: system.cpus0: Entering KVM...
75760000: system.cpus0: KVM: Executing for 2259500 ticks
75760000: system.cpus0: KVM: Executed 5159 instructions in 13688 cycles
(6844000 ticks, sim cycles: 13688).
[...]
126512000: system.cpus0: handleKvmExit (exit_reason: 6)
126512000: system.cpus0: KVM: Handling MMIO (w: 1, addr: 0x1c090024, len: 4)
126512000: system.cpus0: In updateThreadContext():
[...]
126512000: system.cpus0: PC := 0xd8 (t: 0, a64: 1)
On Wed, Mar 14, 2018 at 7:37 PM, Gabe Black
<[email protected]<mailto:[email protected]>> wrote:
I tried it just now, and I still don't see anything on the console. I switched
back to using my own script since it's a bit simpler (it doesn't use all the
configs/common stuff), and started looking at the KVM debug output. I see that
both cpus claim to execute instructions, although cpu1 didn't take an exit in
the output I was looking at. cpu0 took four exits, two which touched some UART
registers, and two which touched RealView registes, the V2M_SYS_CFGDATA and
V2M_SYS_CFGCTRL registers judging by the comments in the bootloader assembly
file.
After that they claim to be doing stuff, although I see no further console
output or KVM exits. The accesses themselves and their PCs are from the
bootloader blob, and so I'm pretty confident that it's starting that and
executing some of those instructions. One thing that looks very odd now that I
think about it, is that the KVM messages about entering and executing
instructions (like those below) seem to say that cpu0 has executed thousands of
instructions, but the exits I see seem to correspond to the first maybe 50
instructions it should be seeing in the bootloader blob. Are those values bogus
for some reason? Is there some existing debug output which would let me see
where KVM thinks it is periodically to see if it's in the kernel or if it went
bananas and is executing random memory somewhere? Or if it just got stuck
waiting for some event that's not going to show up?
Are there any important CLs which haven't made their way into upstream somehow?
Gabe
On Wed, Mar 14, 2018 at 4:28 AM, Andreas Sandberg
<[email protected]<mailto:[email protected]>> wrote:
Have you tried using the fs_bigLITTLE script in configs/examples/arm?
That's the script I have been using for testing.
I just tested the script with 8 little CPUs and 0 big CPUs and it seems
to work. Timing is a bit temperamental though, so you might need to
override the simulation quantum. The default is 1ms, you might need to
decrease it to something slightly smaller (I'm currently using 0.5ms).
Another caveat is that there seem to be some issues related to dtb
auto-generation that affect KVM guests. We are currently testing a
solution for this issue.
Cheers,
Andreas
On 12/03/2018 22:26, Gabe Black wrote:
I'm trying to run in FS mode, to boot android/linux.
Gabe
On Mon, Mar 12, 2018 at 3:26 PM, Dutu, Alexandru
<[email protected]<mailto:[email protected]>>
wrote:
Hi Gabe,
Are you running SE or FS mode?
Thanks,
Alex
-----Original Message-----
From: gem5-dev
[mailto:[email protected]<mailto:[email protected]>] On Behalf
Of Gabe Black
Sent: Friday, March 9, 2018 5:46 PM
To: gem5 Developer List <[email protected]<mailto:[email protected]>>
Subject: [gem5-dev] Multicore ARM v8 KVM based simulation
Hi folks. I have a config script set up where I can run a KVM based ARM v8
simulation just fine when I have a single CPU in it, but when I try running
with more than one CPU, it just seems to get lost and not do anything. Is
this a configuration that's supported? If so, are there any caveats to how
it's set up? I may be missing something simple, but it's not apparent to me
at the moment.
Gabe
_______________________________________________
gem5-dev mailing list
[email protected]<mailto:[email protected]>
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]<mailto:[email protected]>
http://m5sim.org/mailman/listinfo/gem5-dev
_______________________________________________
gem5-dev mailing list
[email protected]<mailto:[email protected]>
http://m5sim.org/mailman/listinfo/gem5-dev
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the contents to any
other person, use it for any purpose, or store or copy the information in any
medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are
confidential and may also be privileged. If you are not the intended recipient,
please notify the sender immediately and do not disclose the contents to any
other person, use it for any purpose, or store or copy the information in any
medium. Thank you.
_______________________________________________
gem5-dev mailing list
[email protected]
http://m5sim.org/mailman/listinfo/gem5-dev
