Hi Wenqi > -----Original Message----- > From: wq...@utexas.edu <wq...@utexas.edu> > Sent: 09 April 2021 23:07 > To: Giacomo Travaglini <giacomo.travagl...@arm.com>; gem5 users mailing > list <gem5-users@gem5.org> > Subject: Re: [gem5-users] Fail to Boot Multicore Arm System with KVM CPU > > Hi Giacomo, > > Thanks for letting me know what the problem is, I am able to verify the > solution on my end as well :-) > > Just curious, as I am not familiar with the arm system, do you always need to > provide a separate bootloader to boot arm FS simulation in gem5?
You don't if you run a baremetal workload in an arm FS simulation. If you boot an OS like Linux, then the answer is: it depends, but you need to initialize the system somehow. The default behaviour in fs_bigLITTLE, fs and fs_starter is to use the gem5 bootloader (very minimal) to boot the OS. Nothing stops you from using different kinds of bootloaders though. For example in the past I have used the aarch64 bootwrapper which is generating a single image with kernel + bootloader (external) + dtb. Lately I have been using GRUB as well. You could in theory avoid having the extra bootloader binary; one option is to embed it in gem5 (I might be wrong but I believe this is what QEMU does), and the other is to initialize the platform on the host side (replicating what the guest bootloader is supposed to do before starting executing the kernel). Those two options are currently not supported. > Will the one packed with the kernel work? (I assume when sim x86, it uses > the bootloader from the kernel). I believe x86 is implementing one of the two options above > > Best, > > Wenqi Kind Regards Giacomo > > On 4/9/21 5:09 AM, Giacomo Travaglini wrote: > > Hi Wenqi, > > > > I have found the problem; in order to fix it you should recompile the > > bootloaders from system/arm/bootloader/arm64 and replace all of them > with the ones in M5_PATH/binaries That solves the problem on my side. > > > > I am gonna update the guest binaries tarball in gem5.org (I was going > > to do that anyway after the 21.0 release) > > > > Kind Regards > > > > Giacomo > > > > > >> -----Original Message----- > >> From: Giacomo Travaglini > >> Sent: 07 April 2021 09:13 > >> To: Wenqi Yin <wq...@utexas.edu>; gem5 users mailing list <gem5- > >> us...@gem5.org> > >> Subject: RE: [gem5-users] Fail to Boot Multicore Arm System with KVM > >> CPU > >> > >> Thanks Wenqi, > >> > >> I was about to suggest the same; please feel free to open a JIRA ticket at: > >> > >> https://gem5.atlassian.net/jira/software/c/projects/GEM5/issues/ > >> > >> Kind Regards > >> > >> Giacomo > >> > >>> -----Original Message----- > >>> From: Wenqi Yin <wq...@utexas.edu> > >>> Sent: 07 April 2021 06:40 > >>> To: gem5 users mailing list <gem5-users@gem5.org> > >>> Cc: Giacomo Travaglini <giacomo.travagl...@arm.com> > >>> Subject: Re: [gem5-users] Fail to Boot Multicore Arm System with KVM > >>> CPU > >>> > >>> Hi All, > >>> > >>> The problem turns out to be with the recent release v21.0. I rolled > >>> back to v20.2, it can boot 8+ KVM cpu after applying all the changes > >> discusses before. > >>> Shall I report this issue anywhere? If so is there any instructions to do > >>> so? > >>> Thanks > >>> > >>> Best, > >>> Wenqi > >>> > >>>> On Apr 6, 2021, at 11:54, wq...@utexas.edu wrote: > >>>> > >>>> Hi Giacomo, > >>>> > >>>> I pasted the perhaps most relevant dmesg output of guest kernel and > >>>> the > >>> disassembled dtb file snippet (In this case, I use fs_bigLittle.py > >>> and boot 4 > >>> vCPU) . Also attached the full dts and dmesg output in case I left > anything. > >>>> [ 0.000000] CPU features: GIC system register CPU interface present > but > >>> disab > >>>> led by higher exception level > >>>> > >>>> [ 0.052000] smp: Bringing up secondary CPUs ... > >>>> [ 2.004021] random: fast init done > >>>> [ 2.132023] CPU1: failed to come online > >>>> [ 2.132023] CPU1: failed in unknown state : 0x0 > >>>> [ 4.244045] CPU2: failed to come online > >>>> [ 4.244045] CPU2: failed in unknown state : 0x0 > >>>> [ 6.356068] CPU3: failed to come online > >>>> [ 6.356068] CPU3: failed in unknown state : 0x0 > >>>> [ 8.468091] CPU4: failed to come online > >>>> [ 8.468091] CPU4: failed in unknown state : 0x0 > >>>> [ 8.468091] smp: Brought up 1 node, 1 CPU > >>>> > >>>> > >>>> cpus { > >>>> #address-cells = <0x1>; > >>>> #size-cells = <0x0>; > >>>> > >>>> cpu@0 { > >>>> device_type = "cpu"; > >>>> compatible = "gem5,arm-cpu"; > >>>> reg = <0x0>; > >>>> enable-method = "spin-table"; > >>>> cpu-release-addr = <0x0 0x87fffff8>; > >>>> clock-frequency = <0x773593ff>; > >>>> phandle = <0xf>; > >>>> }; > >>>> > >>>> cpu@1 { > >>>> device_type = "cpu"; > >>>> compatible = "gem5,arm-cpu"; > >>>> reg = <0x1>; > >>>> enable-method = "spin-table"; > >>>> cpu-release-addr = <0x0 0x87fffff8>; > >>>> clock-frequency = <0x773593ff>; > >>>> phandle = <0x10>; > >>>> }; > >>>> > >>>> cpu@2 { > >>>> device_type = "cpu"; > >>>> compatible = "gem5,arm-cpu"; > >>>> reg = <0x2>; > >>>> enable-method = "spin-table"; > >>>> cpu-release-addr = <0x0 0x87fffff8>; > >>>> clock-frequency = <0x773593ff>; > >>>> phandle = <0x11>; > >>>> }; > >>>> > >>>> cpu@3 { > >>>> device_type = "cpu"; > >>>> compatible = "gem5,arm-cpu"; > >>>> reg = <0x3>; > >>>> enable-method = "spin-table"; > >>>> cpu-release-addr = <0x0 0x87fffff8>; > >>>> clock-frequency = <0x773593ff>; > >>>> phandle = <0x12>; > >>>> }; > >>>> > >>>> cpu@104 { > >>>> device_type = "cpu"; > >>>> compatible = "gem5,arm-cpu"; > >>>> reg = <0x104>; > >>>> enable-method = "spin-table"; > >>>> cpu-release-addr = <0x0 0x87fffff8>; > >>>> clock-frequency = <0x3b9ac9ff>; > >>>> phandle = <0x13>; > >>>> }; > >>>> }; > >>>> > >>>> best, > >>>> > >>>> Wenqi > >>>> > >>>> On 4/6/21 7:35 AM, Giacomo Travaglini wrote: > >>>>> Hi Wenqi, > >>>>> > >>>>> Could you provide us with the booting log of the guest kernel? > >>>>> > >>>>> Kind Regards > >>>>> > >>>>> Giacomo > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Wenqi Yin <wq...@utexas.edu> > >>>>>> Sent: 06 April 2021 05:48 > >>>>>> To: Giacomo Travaglini <giacomo.travagl...@arm.com> > >>>>>> Cc: gem5 users mailing list <gem5-users@gem5.org>; Wenqi Yin > >>>>>> <wq...@utexas.edu> > >>>>>> Subject: Re: [gem5-users] Fail to Boot Multicore Arm System with > >>>>>> KVM CPU > >>>>>> > >>>>>> Hi Giacomo, > >>>>>> > >>>>>> Just want to follow up a bit on this, any suggestions on what > >>>>>> could be causing the kernel failing to bring up all those secondary > CPUs? > >>>>>> > >>>>>> Best, > >>>>>> Wenqi > >>>>>>> On Apr 1, 2021, at 16:53, Wenqi Yin <wq...@utexas.edu> wrote: > >>>>>>> > >>>>>>> HI Giacomo, > >>>>>>> > >>>>>>> I add that line of code you pointed me to to the > >>>>>>> fs_bigLittle.py, now I can > >>>>>> boot into the terminal, however, the kernel cannot bring all but > >>>>>> one cpu online. It’s also a bit strange in the sense that I > >>>>>> specify ‘-smp 12’, and inside the guest when I check lscpu, it > >>>>>> shows total of 13 CPUs and only cpu0 is online, cpu1-12 are offline. > >>>>>>> I tried both the stock kernel as well as compiling kernel myself > >>>>>>> following > >>>>>> your suggestions from earlier email, both have the same issue. Do > >>>>>> you have any suggestions on what could be causing the problem? > >>>>>>> Best, > >>>>>>> Wenqi > >>>>>>> > >>>>>>>> On Apr 1, 2021, at 03:28, Giacomo Travaglini > >>>>>> <giacomo.travagl...@arm.com> wrote: > >>>>>>>> Yes, apologies, I forgot to mention that. > >>>>>>>> As your host is likely using a GICv3 interrupt controller, you > >>>>>>>> need to > >>>>>> entirely emulate the GICv2 and Generic Timer in userspace (gem5). > >>>>>>>> This is done via the simulate_gic (as you have already done) > >>>>>>>> and by removing the system register interface info from the DTB > >>>>>>>> node > >>>>>>>> > >>>>>>>> Here's an example: > >>>>>>>> > >>>>>>>> > >> > https://github.com/gem5/gem5/blob/stable/tests/gem5/configs/arm_gene > >>>>>> r > >>>>>>>> ic.py#L99 > >>>>>>>> > >>>>>>>> Let me know if this works > >>>>>>>> > >>>>>>>> Kind Regards > >>>>>>>> > >>>>>>>> Giacomo > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Original Message----- > >>>>>>>>> From: wq...@utexas.edu <wq...@utexas.edu> > >>>>>>>>> Sent: 31 March 2021 19:18 > >>>>>>>>> To: Giacomo Travaglini <giacomo.travagl...@arm.com>; gem5 > >> users > >>>>>>>>> mailing list <gem5-users@gem5.org>; wq...@utexas.edu > >>>>>>>>> Subject: Re: [gem5-users] Fail to Boot Multicore Arm System > >>>>>>>>> with KVM CPU > >>>>>>>>> > >>>>>>>>> Hi Giacomo, > >>>>>>>>> > >>>>>>>>> Thanks for your reply. I tried the solution you suggested, but > >>>>>>>>> seems there are still problems. Just make sure I understood > >>>>>>>>> correctly, I specified the 'machine-type' as 'VExpress_GEM5_V1' > >>>>>>>>> and in the VExpress_GEM5_V1_Base class's definition > >>>>>>>>> (src/dev/arm/RealView.py), > >>>>>> when instantiating the gic, I use: > >>>>>>>>> gic = kvm_gicv2_class(dist_addr=0x2c001000, > >> cpu_addr=0x2c002000, > >>>>>>>>> it_lines=512, gem5_extensions=True) > >>>>>>>>> > >>>>>>>>> The kvm_gicv2_class will be resolved as MuxingKvmGic, which > >>>>>>>>> inherent from > >>>>>>>>> GicV2 class. To make sure the change applied, I also change > >>>>>>>>> the default value of gem5_extensions from False to True in the > >>>>>>>>> GicV2 class's definition (src/dev/arm/Gic.py). After rebuild, > >>>>>>>>> when starting gem5 there will be a panic says "KVM: failed to > >>>>>>>>> create virtual CPU" (from /src/cpu/kvm/vm.cc, when using ioctl > >>>>>>>>> to create > >>>>>>>>> vcpu) > >>>>>>>>> > >>>>>>>>> Another thing I tried is to set the "simulate_gic" of the > >>>>>>>>> MuxingKvmGic class from False to True, this seems help me get > >>>>>>>>> around the vCPU creation failure, however, at some point it > >>>>>>>>> faces an assertion > >>>>>> failure of: > >>>>>>>>> gem5.opt: build/ARM/sim/eventq.hh:763: void > >>>>>>>>> EventQueue::schedule(Event*, Tick, bool): Assertion `when >= > >>>>>> getCurTick()' > >>>>>>>>> failed. > >>>>>>>>> Program aborted at tick 112565277000 > >>>>>>>>> > >>>>>>>>> Best, > >>>>>>>>> > >>>>>>>>> Wenqi > >>>>>>>>> > >>>>>>>>> On 3/31/21 7:57 AM, Giacomo Travaglini wrote: > >>>>>>>>>> Hi Wenqi, > >>>>>>>>>> > >>>>>>>>>> First of all thanks for the detailed explanation of your problem. > >>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: wqyin--- via gem5-users <gem5-users@gem5.org> > >>>>>>>>>>> Sent: 30 March 2021 22:47 > >>>>>>>>>>> To: gem5-users@gem5.org; wq...@utexas.edu > >>>>>>>>>>> Cc: wq...@utexas.edu > >>>>>>>>>>> Subject: [gem5-users] Fail to Boot Multicore Arm System with > >>>>>>>>>>> KVM CPU > >>>>>>>>>>> > >>>>>>>>>>> Hello all, > >>>>>>>>>>> > >>>>>>>>>>> I am trying to model a multicore arm64 system with the > >>>>>>>>>>> example/arm/fs_bigLittle.py using kvm cpu. However, when I > >>>>>>>>>>> specify > >>>>>>>>>>> "- machine-type VExpress_Gem5_V2_XXX", the kernel panic > >>> during > >>>>>>>>>>> booting, giving an error message of: > >>>>>>>>>>> > >>>>>>>>>>> [ 0.000000] GICv3: Distributor has Range Selector support > >>>>>> [ 0.000000] > >>>>>>>>>>> GICv3: no VLPI support, direct LPI support [ 0.000000] ITS > >> [mem > >>>>>>>>>>> 0x2e010000-0x2e02ffff] [ 0.000000] > ITS@0x000000002e010000: > >>>>>> allocated > >>>>>>>>>>> 262144 Devices > >>>>>>>>>>> @fc600000 (flat, esz 8, psz 64K, shr 1) > >>>>>>>>>>> [ 0.000000] ITS@0x000000002e010000: allocated 8192 > Interrupt > >>>>>>>>> Collections > >>>>>>>>>>> @fc460000 (flat, esz 8, psz 64K, shr 1) [ 0.000000] GIC: using > LPI > >>>>>> property > >>>>>>>>>>> table @0x00000000fc470000 [ 0.000000] ITS: Allocated 1792 > >>> chunks > >>>>>> for > >>>>>>>>> LPIs > >>>>>>>>>>> [ 0.000000] GICv3: CPU0: found redistributor 0 region > >>>>>>>>>>> 0:0x000000002c010000 > >>>>>>>>>>> [ 0.000000] CPU0: using LPI pending table > >> @0x00000000fc480000 > >>>>>>>>> [ 0.000000] > >>>>>>>>>>> GIC: using cache flushing for LPI property table [ 0.000000] > >> GICv3: > >>>>>> GIC: > >>>>>>>>>>> unable to set SRE (disabled at EL2), panic ahead [ 0.000000] -- > -- > >> --- > >>> ---- > >>>>>> - > >>>>>>>>> [ cut > >>>>>>>>>>> here ]------------ [ 0.000000] kernel BUG at /work/gem5- > >>>>>>>>>>> scripts/submodules/linux/arch/arm64/kernel/entry.S:602! > >>>>>>>>>>> [ 0.000000] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP > >>>>>> [ 0.000000] > >>>>>>>>>>> Modules linked in: > >>>>>>>>>>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted > 4.18.0+ > >>> #1 > >>>>>>>>>>> [ 0.000000] Hardware name: V2P-CA15 (DT) [ 0.000000] > pstate: > >>>>>>>>> 80000085 > >>>>>>>>>>> (Nzcv daIf -PAN -UAO) > >>>>>>>>>> Unfortunately KVM with GICv3 is not currently supported. > >>>>>>>>>> > >>>>>>>>>>> I saw this error message regardless of how many vCPU I try > >>>>>>>>>>> to > >>> model. > >>>>>>>>>>> However, when I switch to machine-type of "VExpress- > Gem5", > >>> the > >>>>>>>>>>> guest can boot, but it only allows up to 8 vCPUs. I suppose > >>>>>>>>>>> this is because in this config it doesn't support GICv3. > >>>>>>>>>>> > >>>>>>>>>>> For my use case, I do need more than 8 cpus in the modeled > >>>>>>>>>>> system, can anyone help me here? > >>>>>>>>>> We provide a workaround for that, allowing you to use more > >>>>>>>>>> than > >>>>>>>>>> 8 CPUs > >>>>>>>>> with GICv2. > >>>>>>>>>> That requires you to set the Gicv2.gem5_extensions parameter > >>>>>>>>>> to > >>> True. > >>>>>>>>>> A change in the Linux GIC driver is also needed. IIRC the > >>>>>>>>>> prebuilt binary > >>>>>>>>> should already contain that. > >>>>>>>>>> If not, you would have to rebuild the kernel yourself from > >>>>>>>>>> the following > >>>>>>>>> Linux fork: > >>> https://gem5.googlesource.com/arm/linux/+/refs/heads/gem5/v4.15 > >>>>>>>>>> (Use the gem5_defconfig) > >>>>>>>>>> > >>>>>>>>>> Let me know if it works > >>>>>>>>>> > >>>>>>>>>>> Here is the environment I used for the experiment: > >>>>>>>>>>> > >>>>>>>>>>> Host: aarch64 platform > >>>>>>>>>>> > >>>>>>>>>>> Gem5 version: v21.0 both stable and develop branch > >>>>>>>>>>> > >>>>>>>>>>> kernel and disk image: both downloaded from gem5 > homepage: > >>>>>>>>>>> > >>>>>>>>>>> kernel: http://dist.gem5.org/dist/current/arm/aarch-system- > >>>>>>>>>>> 201901106.tar.bz2 > >>>>>>>>>>> > >>>>>>>>>>> disk: > >>>>>>>>>>> http://dist.gem5.org/dist/current/arm/disks/ubuntu-18.04- > >>> arm64- > >>>>>>>>>>> docker.img.bz2 > >>>>>>>>>>> > >>>>>>>>>>> launch cmd: build/ARM/gem5.opt > >>>>>> configs/example/arm/fs_bigLITTLE.py > >>>>>>>>>>> --kernel ../gem5_image/binaries/vmlinux.arm64 --machine- > >> type > >>>>>>>>>>> VExpress_GEM5_V2 --disk ../gem5_image/ubuntu-18.04- > >> arm64- > >>>>>>>>> docker.img > >>>>>>>>>>> --cpu-type kvm --big-cpus X > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> > >>>>>>>>>>> Wenqi > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> gem5-users mailing list -- gem5-users@gem5.org To > >>>>>>>>>>> unsubscribe send an email to gem5-users- le...@gem5.org > >>>>>>>>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > >>>>>>>>>> Kind Regards > >>>>>>>>>> > >>>>>>>>>> Giacomo > >>>>>>>>>> IMPORTANT NOTICE: The contents of this email and any > >>> attachments > >>>>>>>>>> are > >>>>>>>>> confidential and may also be privileged. If you are not the > >>>>>>>>> intended recipient, please notify the sender immediately and > >>>>>>>>> do not disclose the contents to any other person, use it for > >>>>>>>>> any purpose, or store or copy the information in any medium. > >>>>>>>>> Thank > >> you. > >>>>>>>> IMPORTANT NOTICE: The contents of this email and any > >> attachments > >>>>>>>> are > >>>>>> confidential and may also be privileged. If you are not the > >>>>>> intended recipient, please notify the sender immediately and do > >>>>>> not disclose the contents to any other person, use it for any > >>>>>> purpose, or store or copy the information in any medium. Thank you. > >>>>> IMPORTANT NOTICE: The contents of this email and any attachments > >>>>> are > >>> confidential and may also be privileged. If you are not the intended > >>> recipient, please notify the sender immediately and do not disclose > >>> the contents to any other person, use it for any purpose, or store > >>> or copy the information in any medium. Thank you. > >>>> <dmsg.txt><system.dts> > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, > please notify the sender immediately and do not disclose the contents to any > other person, use it for any purpose, or store or copy the information in any > medium. Thank you. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
