Hi Wenqi, I have found the problem; in order to fix it you should recompile the bootloaders from system/arm/bootloader/arm64 and replace all of them with the ones in M5_PATH/binaries That solves the problem on my side.
I am gonna update the guest binaries tarball in gem5.org (I was going to do that anyway after the 21.0 release) Kind Regards Giacomo > -----Original Message----- > From: Giacomo Travaglini > Sent: 07 April 2021 09:13 > To: Wenqi Yin <wq...@utexas.edu>; gem5 users mailing list <gem5- > us...@gem5.org> > Subject: RE: [gem5-users] Fail to Boot Multicore Arm System with KVM CPU > > Thanks Wenqi, > > I was about to suggest the same; please feel free to open a JIRA ticket at: > > https://gem5.atlassian.net/jira/software/c/projects/GEM5/issues/ > > Kind Regards > > Giacomo > > > -----Original Message----- > > From: Wenqi Yin <wq...@utexas.edu> > > Sent: 07 April 2021 06:40 > > To: gem5 users mailing list <gem5-users@gem5.org> > > Cc: Giacomo Travaglini <giacomo.travagl...@arm.com> > > Subject: Re: [gem5-users] Fail to Boot Multicore Arm System with KVM > > CPU > > > > Hi All, > > > > The problem turns out to be with the recent release v21.0. I rolled > > back to v20.2, it can boot 8+ KVM cpu after applying all the changes > discusses before. > > > > Shall I report this issue anywhere? If so is there any instructions to do > > so? > > Thanks > > > > Best, > > Wenqi > > > > > On Apr 6, 2021, at 11:54, wq...@utexas.edu wrote: > > > > > > Hi Giacomo, > > > > > > I pasted the perhaps most relevant dmesg output of guest kernel and > > > the > > disassembled dtb file snippet (In this case, I use fs_bigLittle.py and > > boot 4 > > vCPU) . Also attached the full dts and dmesg output in case I left anything. > > > > > > [ 0.000000] CPU features: GIC system register CPU interface present but > > disab > > > led by higher exception level > > > > > > [ 0.052000] smp: Bringing up secondary CPUs ... > > > [ 2.004021] random: fast init done > > > [ 2.132023] CPU1: failed to come online > > > [ 2.132023] CPU1: failed in unknown state : 0x0 > > > [ 4.244045] CPU2: failed to come online > > > [ 4.244045] CPU2: failed in unknown state : 0x0 > > > [ 6.356068] CPU3: failed to come online > > > [ 6.356068] CPU3: failed in unknown state : 0x0 > > > [ 8.468091] CPU4: failed to come online > > > [ 8.468091] CPU4: failed in unknown state : 0x0 > > > [ 8.468091] smp: Brought up 1 node, 1 CPU > > > > > > > > > cpus { > > > #address-cells = <0x1>; > > > #size-cells = <0x0>; > > > > > > cpu@0 { > > > device_type = "cpu"; > > > compatible = "gem5,arm-cpu"; > > > reg = <0x0>; > > > enable-method = "spin-table"; > > > cpu-release-addr = <0x0 0x87fffff8>; > > > clock-frequency = <0x773593ff>; > > > phandle = <0xf>; > > > }; > > > > > > cpu@1 { > > > device_type = "cpu"; > > > compatible = "gem5,arm-cpu"; > > > reg = <0x1>; > > > enable-method = "spin-table"; > > > cpu-release-addr = <0x0 0x87fffff8>; > > > clock-frequency = <0x773593ff>; > > > phandle = <0x10>; > > > }; > > > > > > cpu@2 { > > > device_type = "cpu"; > > > compatible = "gem5,arm-cpu"; > > > reg = <0x2>; > > > enable-method = "spin-table"; > > > cpu-release-addr = <0x0 0x87fffff8>; > > > clock-frequency = <0x773593ff>; > > > phandle = <0x11>; > > > }; > > > > > > cpu@3 { > > > device_type = "cpu"; > > > compatible = "gem5,arm-cpu"; > > > reg = <0x3>; > > > enable-method = "spin-table"; > > > cpu-release-addr = <0x0 0x87fffff8>; > > > clock-frequency = <0x773593ff>; > > > phandle = <0x12>; > > > }; > > > > > > cpu@104 { > > > device_type = "cpu"; > > > compatible = "gem5,arm-cpu"; > > > reg = <0x104>; > > > enable-method = "spin-table"; > > > cpu-release-addr = <0x0 0x87fffff8>; > > > clock-frequency = <0x3b9ac9ff>; > > > phandle = <0x13>; > > > }; > > > }; > > > > > > best, > > > > > > Wenqi > > > > > > On 4/6/21 7:35 AM, Giacomo Travaglini wrote: > > >> Hi Wenqi, > > >> > > >> Could you provide us with the booting log of the guest kernel? > > >> > > >> Kind Regards > > >> > > >> Giacomo > > >> > > >>> -----Original Message----- > > >>> From: Wenqi Yin <wq...@utexas.edu> > > >>> Sent: 06 April 2021 05:48 > > >>> To: Giacomo Travaglini <giacomo.travagl...@arm.com> > > >>> Cc: gem5 users mailing list <gem5-users@gem5.org>; Wenqi Yin > > >>> <wq...@utexas.edu> > > >>> Subject: Re: [gem5-users] Fail to Boot Multicore Arm System with > > >>> KVM CPU > > >>> > > >>> Hi Giacomo, > > >>> > > >>> Just want to follow up a bit on this, any suggestions on what > > >>> could be causing the kernel failing to bring up all those secondary > > >>> CPUs? > > >>> > > >>> Best, > > >>> Wenqi > > >>>> On Apr 1, 2021, at 16:53, Wenqi Yin <wq...@utexas.edu> wrote: > > >>>> > > >>>> HI Giacomo, > > >>>> > > >>>> I add that line of code you pointed me to to the fs_bigLittle.py, > > >>>> now I can > > >>> boot into the terminal, however, the kernel cannot bring all but > > >>> one cpu online. It’s also a bit strange in the sense that I > > >>> specify ‘-smp 12’, and inside the guest when I check lscpu, it > > >>> shows total of 13 CPUs and only cpu0 is online, cpu1-12 are offline. > > >>>> I tried both the stock kernel as well as compiling kernel myself > > >>>> following > > >>> your suggestions from earlier email, both have the same issue. Do > > >>> you have any suggestions on what could be causing the problem? > > >>>> Best, > > >>>> Wenqi > > >>>> > > >>>>> On Apr 1, 2021, at 03:28, Giacomo Travaglini > > >>> <giacomo.travagl...@arm.com> wrote: > > >>>>> Yes, apologies, I forgot to mention that. > > >>>>> As your host is likely using a GICv3 interrupt controller, you > > >>>>> need to > > >>> entirely emulate the GICv2 and Generic Timer in userspace (gem5). > > >>>>> This is done via the simulate_gic (as you have already done) and > > >>>>> by removing the system register interface info from the DTB node > > >>>>> > > >>>>> Here's an example: > > >>>>> > > >>>>> > > >>> > > > https://github.com/gem5/gem5/blob/stable/tests/gem5/configs/arm_gene > > >>> r > > >>>>> ic.py#L99 > > >>>>> > > >>>>> Let me know if this works > > >>>>> > > >>>>> Kind Regards > > >>>>> > > >>>>> Giacomo > > >>>>> > > >>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: wq...@utexas.edu <wq...@utexas.edu> > > >>>>>> Sent: 31 March 2021 19:18 > > >>>>>> To: Giacomo Travaglini <giacomo.travagl...@arm.com>; gem5 > users > > >>>>>> mailing list <gem5-users@gem5.org>; wq...@utexas.edu > > >>>>>> Subject: Re: [gem5-users] Fail to Boot Multicore Arm System > > >>>>>> with KVM CPU > > >>>>>> > > >>>>>> Hi Giacomo, > > >>>>>> > > >>>>>> Thanks for your reply. I tried the solution you suggested, but > > >>>>>> seems there are still problems. Just make sure I understood > > >>>>>> correctly, I specified the 'machine-type' as 'VExpress_GEM5_V1' > > >>>>>> and in the VExpress_GEM5_V1_Base class's definition > > >>>>>> (src/dev/arm/RealView.py), > > >>> when instantiating the gic, I use: > > >>>>>> gic = kvm_gicv2_class(dist_addr=0x2c001000, > cpu_addr=0x2c002000, > > >>>>>> it_lines=512, gem5_extensions=True) > > >>>>>> > > >>>>>> The kvm_gicv2_class will be resolved as MuxingKvmGic, which > > >>>>>> inherent from > > >>>>>> GicV2 class. To make sure the change applied, I also change the > > >>>>>> default value of gem5_extensions from False to True in the > > >>>>>> GicV2 class's definition (src/dev/arm/Gic.py). After rebuild, > > >>>>>> when starting gem5 there will be a panic says "KVM: failed to > > >>>>>> create virtual CPU" (from /src/cpu/kvm/vm.cc, when using ioctl > > >>>>>> to create > > >>>>>> vcpu) > > >>>>>> > > >>>>>> Another thing I tried is to set the "simulate_gic" of the > > >>>>>> MuxingKvmGic class from False to True, this seems help me get > > >>>>>> around the vCPU creation failure, however, at some point it > > >>>>>> faces an assertion > > >>> failure of: > > >>>>>> gem5.opt: build/ARM/sim/eventq.hh:763: void > > >>>>>> EventQueue::schedule(Event*, Tick, bool): Assertion `when >= > > >>> getCurTick()' > > >>>>>> failed. > > >>>>>> Program aborted at tick 112565277000 > > >>>>>> > > >>>>>> Best, > > >>>>>> > > >>>>>> Wenqi > > >>>>>> > > >>>>>> On 3/31/21 7:57 AM, Giacomo Travaglini wrote: > > >>>>>>> Hi Wenqi, > > >>>>>>> > > >>>>>>> First of all thanks for the detailed explanation of your problem. > > >>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: wqyin--- via gem5-users <gem5-users@gem5.org> > > >>>>>>>> Sent: 30 March 2021 22:47 > > >>>>>>>> To: gem5-users@gem5.org; wq...@utexas.edu > > >>>>>>>> Cc: wq...@utexas.edu > > >>>>>>>> Subject: [gem5-users] Fail to Boot Multicore Arm System with > > >>>>>>>> KVM CPU > > >>>>>>>> > > >>>>>>>> Hello all, > > >>>>>>>> > > >>>>>>>> I am trying to model a multicore arm64 system with the > > >>>>>>>> example/arm/fs_bigLittle.py using kvm cpu. However, when I > > >>>>>>>> specify > > >>>>>>>> "- machine-type VExpress_Gem5_V2_XXX", the kernel panic > > during > > >>>>>>>> booting, giving an error message of: > > >>>>>>>> > > >>>>>>>> [ 0.000000] GICv3: Distributor has Range Selector support > > >>> [ 0.000000] > > >>>>>>>> GICv3: no VLPI support, direct LPI support [ 0.000000] ITS > [mem > > >>>>>>>> 0x2e010000-0x2e02ffff] [ 0.000000] ITS@0x000000002e010000: > > >>> allocated > > >>>>>>>> 262144 Devices > > >>>>>>>> @fc600000 (flat, esz 8, psz 64K, shr 1) > > >>>>>>>> [ 0.000000] ITS@0x000000002e010000: allocated 8192 Interrupt > > >>>>>> Collections > > >>>>>>>> @fc460000 (flat, esz 8, psz 64K, shr 1) [ 0.000000] GIC: using > > >>>>>>>> LPI > > >>> property > > >>>>>>>> table @0x00000000fc470000 [ 0.000000] ITS: Allocated 1792 > > chunks > > >>> for > > >>>>>> LPIs > > >>>>>>>> [ 0.000000] GICv3: CPU0: found redistributor 0 region > > >>>>>>>> 0:0x000000002c010000 > > >>>>>>>> [ 0.000000] CPU0: using LPI pending table > @0x00000000fc480000 > > >>>>>> [ 0.000000] > > >>>>>>>> GIC: using cache flushing for LPI property table [ 0.000000] > GICv3: > > >>> GIC: > > >>>>>>>> unable to set SRE (disabled at EL2), panic ahead [ 0.000000] > > >>>>>>>> ---- > --- > > ---- > > >>> - > > >>>>>> [ cut > > >>>>>>>> here ]------------ [ 0.000000] kernel BUG at /work/gem5- > > >>>>>>>> scripts/submodules/linux/arch/arm64/kernel/entry.S:602! > > >>>>>>>> [ 0.000000] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP > > >>> [ 0.000000] > > >>>>>>>> Modules linked in: > > >>>>>>>> [ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0+ > > #1 > > >>>>>>>> [ 0.000000] Hardware name: V2P-CA15 (DT) [ 0.000000] pstate: > > >>>>>> 80000085 > > >>>>>>>> (Nzcv daIf -PAN -UAO) > > >>>>>>> Unfortunately KVM with GICv3 is not currently supported. > > >>>>>>> > > >>>>>>>> I saw this error message regardless of how many vCPU I try to > > model. > > >>>>>>>> However, when I switch to machine-type of "VExpress-Gem5", > > the > > >>>>>>>> guest can boot, but it only allows up to 8 vCPUs. I suppose > > >>>>>>>> this is because in this config it doesn't support GICv3. > > >>>>>>>> > > >>>>>>>> For my use case, I do need more than 8 cpus in the modeled > > >>>>>>>> system, can anyone help me here? > > >>>>>>> We provide a workaround for that, allowing you to use more > > >>>>>>> than > > >>>>>>> 8 CPUs > > >>>>>> with GICv2. > > >>>>>>> That requires you to set the Gicv2.gem5_extensions parameter > > >>>>>>> to > > True. > > >>>>>>> A change in the Linux GIC driver is also needed. IIRC the > > >>>>>>> prebuilt binary > > >>>>>> should already contain that. > > >>>>>>> If not, you would have to rebuild the kernel yourself from the > > >>>>>>> following > > >>>>>> Linux fork: > > >>>>>>> > > https://gem5.googlesource.com/arm/linux/+/refs/heads/gem5/v4.15 > > >>>>>>> > > >>>>>>> (Use the gem5_defconfig) > > >>>>>>> > > >>>>>>> Let me know if it works > > >>>>>>> > > >>>>>>>> Here is the environment I used for the experiment: > > >>>>>>>> > > >>>>>>>> Host: aarch64 platform > > >>>>>>>> > > >>>>>>>> Gem5 version: v21.0 both stable and develop branch > > >>>>>>>> > > >>>>>>>> kernel and disk image: both downloaded from gem5 homepage: > > >>>>>>>> > > >>>>>>>> kernel: http://dist.gem5.org/dist/current/arm/aarch-system- > > >>>>>>>> 201901106.tar.bz2 > > >>>>>>>> > > >>>>>>>> disk: > > >>>>>>>> http://dist.gem5.org/dist/current/arm/disks/ubuntu-18.04- > > arm64- > > >>>>>>>> docker.img.bz2 > > >>>>>>>> > > >>>>>>>> launch cmd: build/ARM/gem5.opt > > >>> configs/example/arm/fs_bigLITTLE.py > > >>>>>>>> --kernel ../gem5_image/binaries/vmlinux.arm64 --machine- > type > > >>>>>>>> VExpress_GEM5_V2 --disk ../gem5_image/ubuntu-18.04- > arm64- > > >>>>>> docker.img > > >>>>>>>> --cpu-type kvm --big-cpus X > > >>>>>>>> > > >>>>>>>> Best, > > >>>>>>>> > > >>>>>>>> Wenqi > > >>>>>>>> _______________________________________________ > > >>>>>>>> gem5-users mailing list -- gem5-users@gem5.org To unsubscribe > > >>>>>>>> send an email to gem5-users- le...@gem5.org > > >>>>>>>> %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > >>>>>>> Kind Regards > > >>>>>>> > > >>>>>>> Giacomo > > >>>>>>> IMPORTANT NOTICE: The contents of this email and any > > attachments > > >>>>>>> are > > >>>>>> confidential and may also be privileged. If you are not the > > >>>>>> intended recipient, please notify the sender immediately and do > > >>>>>> not disclose the contents to any other person, use it for any > > >>>>>> purpose, or store or copy the information in any medium. Thank > you. > > >>>>> IMPORTANT NOTICE: The contents of this email and any > attachments > > >>>>> are > > >>> confidential and may also be privileged. If you are not the > > >>> intended recipient, please notify the sender immediately and do > > >>> not disclose the contents to any other person, use it for any > > >>> purpose, or store or copy the information in any medium. Thank you. > > >> IMPORTANT NOTICE: The contents of this email and any attachments > > >> are > > confidential and may also be privileged. If you are not the intended > > recipient, please notify the sender immediately and do not disclose > > the contents to any other person, use it for any purpose, or store or > > copy the information in any medium. Thank you. > > > <dmsg.txt><system.dts> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. _______________________________________________ gem5-users mailing list -- gem5-users@gem5.org To unsubscribe send an email to gem5-users-le...@gem5.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
