Re: [gem5-dev] Review Request 2557: x86: kvm: Fix the KVM CPU in SE and FS on Intel CPUs.
From AMD's system programming manual: SYSCALL New selectors are loaded, without permission checking (see above), as follows: Bits 47:32 of the STAR register specify the selector that is copied into the CS register. Bits 47:32 of the STAR register + 8 specify the selector that is copied into the SS register. The CS_base and the SS_base are both forced to zero. The CS_limit and the SS_limit are both forced to 4 Gbyte. The CS segment attributes are set to execute/read 64-bit code with a CPL of zero. The SS segment attributes are set to read/write and expand-up with a 64-bit stack referenced by RSP. SYSRET When a system procedure performs a SYSRET back to application software, the CS selector is updated from bits 63:50 of the STAR register (STAR.SYSRET_CS) as follows: If the return is to 32-bit mode (legacy or compatibility), CS is updated with the value of STAR.SYSRET_CS. If the return is to 64-bit mode, CS is updated with the value of STAR.SYSRET_CS + 16. In both cases, the CPL is forced to 3, effectively ignoring STAR bits 49:48. The SS selector is updated to point to the next descriptor-table entry after the CS descriptor (STAR.SYSRET_CS + 8), and its RPL is not forced to 3. I am wondering if we could detect the CPU vendor with CPUID and have different setup of the GDT based on the platform you are running on. Could scons actually detect this at build time? Best, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Gabe Black via gem5-dev [gem5-dev@gem5.org] Sent: Wednesday, January 21, 2015 3:48 PM To: mike upton; Default; Gabe Black Subject: Re: [gem5-dev] Review Request 2557: x86: kvm: Fix the KVM CPU in SE and FS on Intel CPUs. On Jan. 21, 2015, 9:22 p.m., mike upton wrote: src/arch/x86/process.cc, lines 218-237 http://reviews.gem5.org/r/2557/diff/2/?file=42948#file42948line218 For AMD systems, the sys descriptors need to come first. On intel systems they need to come second. I do not know how to resolve... mike upton wrote: I have been debugging why patch rb2557 breaks AMD KVM functionality. I was hoping to get to code that would work on both intel and AMD platforms, but am not there yet. This patch is to be applied on top of rb2557.patch. There are 2 main issues, neither of which I understand well enough to take much further. The first issue is that the order that the segment descriptors get instantiated in the GDT table seems to matter between AMD and Intel, and they seem to be mutually incompatible. AMD wants: csSys dsSys ds cs Intel wants: ds cs dsSys csSys I am not sure the relative ordering of ds and cs within a class matters, only that AMD wants the Sys ones first, and Intel wants them second. There is also an issue with how 'star' gets defined. I can not make the Intel code work for AMD. Both issues are addressed in this patch. The patch makes the AMD system work, but breaks Intel functionality. I am also not sure how to upload this into review board. Do I create a separate patch from TOT, or can I somehow attach this to rb2557. Hopefully Gabe or Alexandru can weigh in. I am happy to help, but I am at my 'Peter Principal Limit' as far as my understanding goes. I think it would be really ugly to have a machine-type test to version the code... You should grab a copy of the architecture manual. From there: STAR—The STAR register has the following fields (unless otherwise noted, all bits are read/write): - SYSRET CS and SS Selectors—Bits 63:48. This field is used to specify both the CS and SS selectors loaded into CS and SS during SYSRET. If SYSRET is returning to 32-bit mode (either legacy or compatibility), this field is copied directly into the CS selector field. If SYSRET is returning to 64-bit mode, the CS selector is set to this field + 16. SS.Sel is set to this field + 8, regardless of the target mode. Because SYSRET always returns to CPL 3, the RPL bits 49:48 should be initialized to 11b. - SYSCALL CS and SS Selectors—Bits 47:32. This field is used to specify both the CS and SS selectors loaded into CS and SS during SYSCALL. This field is copied directly into CS.Sel. SS.Sel is set to this field + 8. Because SYSCALL always switches to CPL 0, the RPL bits 33:32 should be initialized to 00b. That's why the order matters and is what it is. - Gabe --- This is an automatically generated e-mail. To reply, visit: http://reviews.gem5.org/r/2557/#review5782 --- On Dec. 10, 2014, 10:11 a.m., Gabe Black wrote: --- This is an automatically generated e-mail. To reply, visit: http://reviews.gem5.org/r/2557/
[gem5-dev] SMT with X86
Hi everyone, I have been trying to run with SMT enabled in SE mode, using x86. It seems ZeroRegister gets mapped to index 16 for the first hardware thread, which is also register t0 that is used by the microcode. For the following hardware threads the ZeroRegister will get mapped to 54, 92, 130 and so on, as there are 16 architectural registers and 22 microcode and additional registers which leads to 38 total registers per hardware thread. All these registers are renamed and because of all this the following assert(prev_reg == zeroReg) fails in SimpleRenameMap::rename. Mapping the ZeroReg to some huge index into the register file introduces new issues as t0 is not maintained 0 any more and effective address computations which include t0 will be wrong. On the other hand if the rename logic is modified to keep the initial mapping to the physical register file for register index 16 of all hardware threads (mapped to 16, 54, 92 etc.) the program counter gets corrupted to 0 and a fault is produced as it tries to execute from address 0. From the observed behavior it seems that register t0 needs to play the role of a zero register with the current microcode implementation. Can somebody please comment if this is required? Thank you, Alex ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Andreas, Your patches solve the issue, I don't see spurious kvm exits anymore. Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas Hansson via gem5-dev Sent: Friday, December 19, 2014 2:43 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Hi Alex, You are absolutely right (and mercurial is not that great in tracking change sets :-). You need the patch before it: http://reviews.gem5.org/r/2572/ I am hoping to get all these pushed before Christmas. Andreas On 17/12/2014 16:38, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Andreas, I've tried applying this patch on top of revision 8fc6e7a835d1 and I get bunch of rejects. It seems dram_ctrl.cc is a bit different in this patch it has all sorts of extra code to deal with ranks. So I wondering this patch requires another one to apply properly. Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas Hansson via gem5-dev Sent: Saturday, December 13, 2014 2:12 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) This patch should hopefully solve the issue with the refresh event: http://reviews.gem5.org/r/2573/ Andreas On 11/12/2014 15:52, Andreas Hansson via gem5-dev gem5-dev@gem5.org wrote: Hi Alex, Mike, I¹ll try and fix this whole issue of the refresh event once and for all. SimpleMemory should only really be used for fast-forwarding and high-level sweeps, and I would like to ensure there are really no reasons to move away from the DRAM controller. It seems the sensible thing to do is to use startup and drainResume as the points where we check the mode of the memory system and either disable/enable the refresh event of the DRAM controller. Hopefully I will have something working before the weekend. Andreas On 11/12/2014 15:32, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, Are you running with SimpleMemory, SE or FS? On my AMD platform, for SE, I get very similar execution times with old implementation, for SimpleMemory and classic memory with detailed memory controller. Also what linux kernel are you using? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of mike upton via gem5-dev Sent: Wednesday, December 10, 2014 3:59 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I was testing this on both Intel and AMD platforms. The new code does seem to work for Intel platforms. The new code also seems to clean up a bunch of runtime warnings I was getting on AMD platforms. However the new code on AMD is either much slower, or it is stuck in a loop. A test that runs for 30 sec with the old code is running for more than 10 mins, and still has a long way to go. On Wed, Dec 10, 2014 at 12:27 PM, Gabe Black via gem5-dev gem5-dev@gem5.org wrote: That's not actually extending the TSS limit, that's what it works out to be with the granularity bit set. The AM and WP bits were set to the wrong thing according to the comments next to them I'm pretty sure. If we wanted the other behavior, we might be able to change them back and have it work. The _BASE registers hold the base of segments as they're specified by the ISA. The _EFF_BASE registers hold the base that will actually be used in address calculations based on the mode of the CPU. For instance, if you're in 64 bit mode, the _BASE of DS might still be 0xFFF from when you were in another mode. The _EFF_BASE would be 0 though, since the DS base is ignored in that case. _EFF_BASE may not be used by the KVM CPU, but it should be set up anyway in case we switch back to a regular CPU. Gabe On Wed, Dec 10, 2014 at 10:20 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Thank you for all the clarifications. I see that for SE to work on vmx the TSS limit had to be extended, am and wp bits in CR0 had to be reset and *_EFF_BASE registers had be set in addition to *_BASE registers for TR TSG IDTR. I wonder what is TR_EFF_BASE. It seems that the contents of TR register (which gets passed to kvm) are the same if with or without *_EFF_BASE registers set. Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 1:21 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Ok, I got SE working too. I'll clean up my patch and send that out in a bit. Gabe On Tue, Dec 9, 2014 at 9:41 PM, Gabe Black gabebl...@google.com wrote: I figured out what the other problem was, so here's the review. http://reviews.gem5.org/r/2557/ Gabe On Tue, Dec 9, 2014 at 5:00 PM, Gabe Black gabebl...@google.com wrote: It was attached
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Andreas, I've tried applying this patch on top of revision 8fc6e7a835d1 and I get bunch of rejects. It seems dram_ctrl.cc is a bit different in this patch it has all sorts of extra code to deal with ranks. So I wondering this patch requires another one to apply properly. Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas Hansson via gem5-dev Sent: Saturday, December 13, 2014 2:12 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) This patch should hopefully solve the issue with the refresh event: http://reviews.gem5.org/r/2573/ Andreas On 11/12/2014 15:52, Andreas Hansson via gem5-dev gem5-dev@gem5.org wrote: Hi Alex, Mike, I¹ll try and fix this whole issue of the refresh event once and for all. SimpleMemory should only really be used for fast-forwarding and high-level sweeps, and I would like to ensure there are really no reasons to move away from the DRAM controller. It seems the sensible thing to do is to use startup and drainResume as the points where we check the mode of the memory system and either disable/enable the refresh event of the DRAM controller. Hopefully I will have something working before the weekend. Andreas On 11/12/2014 15:32, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, Are you running with SimpleMemory, SE or FS? On my AMD platform, for SE, I get very similar execution times with old implementation, for SimpleMemory and classic memory with detailed memory controller. Also what linux kernel are you using? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of mike upton via gem5-dev Sent: Wednesday, December 10, 2014 3:59 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I was testing this on both Intel and AMD platforms. The new code does seem to work for Intel platforms. The new code also seems to clean up a bunch of runtime warnings I was getting on AMD platforms. However the new code on AMD is either much slower, or it is stuck in a loop. A test that runs for 30 sec with the old code is running for more than 10 mins, and still has a long way to go. On Wed, Dec 10, 2014 at 12:27 PM, Gabe Black via gem5-dev gem5-dev@gem5.org wrote: That's not actually extending the TSS limit, that's what it works out to be with the granularity bit set. The AM and WP bits were set to the wrong thing according to the comments next to them I'm pretty sure. If we wanted the other behavior, we might be able to change them back and have it work. The _BASE registers hold the base of segments as they're specified by the ISA. The _EFF_BASE registers hold the base that will actually be used in address calculations based on the mode of the CPU. For instance, if you're in 64 bit mode, the _BASE of DS might still be 0xFFF from when you were in another mode. The _EFF_BASE would be 0 though, since the DS base is ignored in that case. _EFF_BASE may not be used by the KVM CPU, but it should be set up anyway in case we switch back to a regular CPU. Gabe On Wed, Dec 10, 2014 at 10:20 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Thank you for all the clarifications. I see that for SE to work on vmx the TSS limit had to be extended, am and wp bits in CR0 had to be reset and *_EFF_BASE registers had be set in addition to *_BASE registers for TR TSG IDTR. I wonder what is TR_EFF_BASE. It seems that the contents of TR register (which gets passed to kvm) are the same if with or without *_EFF_BASE registers set. Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 1:21 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Ok, I got SE working too. I'll clean up my patch and send that out in a bit. Gabe On Tue, Dec 9, 2014 at 9:41 PM, Gabe Black gabebl...@google.com wrote: I figured out what the other problem was, so here's the review. http://reviews.gem5.org/r/2557/ Gabe On Tue, Dec 9, 2014 at 5:00 PM, Gabe Black gabebl...@google.com wrote: It was attached in my sent mail. Maybe it's being blocked by something? I'm hunting down another problem so I don't want to move my tree around too much, but once that's done I'll post it as a review. Gabe On Tue, Dec 9, 2014 at 4:51 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: I haven't received any attachment to your email. So I don't have your patch. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:42 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality
Re: [gem5-dev] Kernel panic caused by changeset 10552
I don't remember of the top of my head exactly the reason to enable all those features in the CPUID. I do remember trying not to enable things that gem5 does not support like SSEx, SSSE and AVX. I also remember encountering problems with libc that uses x87 instructions and pxor, so I had to enable MMX and x87. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Monday, December 15, 2014 12:35 AM To: gem5 Developer List Subject: Re: [gem5-dev] Kernel panic caused by changeset 10552 I tried reverting this change and it fixes the undefined instruction exceptions I was seeing. It does break KVM in SE mode though, so we probably shouldn't yank it out entirely. We need to find a minimal version of that change which makes KVM in SE work without breaking other things. Gabe On Wed, Dec 10, 2014 at 1:40 AM, Gabe Black gabebl...@google.com wrote: Yeah, I was going to say something about that. CPUID shouldn't advertise features that we don't support. For instance, that change tells CPUID to say we support AVX, but the decoder has no idea what to do with those instructions and will trigger an exception if one is executed. I noticed a bunch of undefined instruction exceptions in my own workload that weren't happening before, and I wonder if that's the cause. I'm not sure how that change helps support KVM in SE mode. Perhaps it should be reverted? Can you explain why it's necessary Alex? If it is, maybe we can reshape it a bit to avoid these side effects. Gabe On Wed, Dec 10, 2014 at 12:43 AM, 马久跃 via gem5-dev gem5-dev@gem5.org wrote: Hi everyone, I found x86_64-vmlinux-2.6.28.4 panic when apply changeset 10552: cpuid, x86: Enabling more features in CPUid. (2.6.22.9 works fine) The gem5 also report warn: x86 cpuid: unimplemented function 13, and kernel report BUG at arch/x86/kernel/xsave.c:323 as following. Can anybody check/fix this bug? Thanks. KERNEL OUTPUT -- MPTABLE: APIC at: 0xFEE0 Processor #0 (Bootup-CPU) I/O APIC #1 Version 17 at 0xFEC0. Processors: 1 Allocating PCI resources starting at c400 (gap: c000:3fff) Built 1 zonelists in Zone order, mobility grouping on. Total pages: 127500 Kernel command line: earlyprintk=ttyS0 console=ttyS0 lpj=723 root=/dev/hda1 Initializing CPU#0 FP/SSE not shown under xsave features 0x80050033000d [ cut here ] kernel BUG at arch/x86/kernel/xsave.c:323! invalid opcode: [#1] last sysfs file: CPU 0 Modules linked in: Pid: 0, comm: swapper Tainted: GW 2.6.28.4 #2 RIP: 0010:[80577b45] [80577b45] xsave_cntxt_init+0x35/0x130 RSP: 0018:80769f48 EFLAGS: 00b8 RAX: 003c RBX: RCX: 807cd460 RDX: RSI: 0d5c RDI: 80702180 RBP: R08: R09: 03fd R10: R11: R12: R13: R14: R15: FS: () GS:807bf020() knlGS: CS: 0010 DS: 0018 ES: 0018 CR0: 80050033 CR2: CR3: 00201000 CR4: 06a0 DR0: DR1: DR2: DR3: DR6: DR7: Process swapper (pid: 0, threadinfo 80768000, task 806f5380) Stack: 8078f133 80795000 8078fe72 8000896f59002087 8022ee104a50 80794000 80795000 8076a98d 80795000 Call Trace: [8078f133] fpu_init+0x3e/0x8e [8078fe72] cpu_init+0x222/0x240 [8076a98d] start_kernel+0x16f/0x2d9 [8076a407] x86_64_start_kernel+0xd9/0xdfCode: 48 c1 e2 20 89 c0 48 8d 34 02 48 89 f0 48 89 35 12 a6 24 00 83 e0 03 48 83 f8 03 74 12 48 c7 c7 50 74 67 80 31 c0 e8 9c 30 01 00 0f 0b eb fe f6 05 57 b7 1e 00 04 48 c7 05 e5 a5 24 00 03 00 00 RIP [80577b45] xsave_cntxt_init+0x35/0x130 RSP 80769f48 ---[ end trace 4eaa2a86a8e2da22 ]--- Kernel panic - not syncing: Attempted to kill the idle task! Jiuyue ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Mike, Are you running with SimpleMemory, SE or FS? On my AMD platform, for SE, I get very similar execution times with old implementation, for SimpleMemory and classic memory with detailed memory controller. Also what linux kernel are you using? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of mike upton via gem5-dev Sent: Wednesday, December 10, 2014 3:59 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I was testing this on both Intel and AMD platforms. The new code does seem to work for Intel platforms. The new code also seems to clean up a bunch of runtime warnings I was getting on AMD platforms. However the new code on AMD is either much slower, or it is stuck in a loop. A test that runs for 30 sec with the old code is running for more than 10 mins, and still has a long way to go. On Wed, Dec 10, 2014 at 12:27 PM, Gabe Black via gem5-dev gem5-dev@gem5.org wrote: That's not actually extending the TSS limit, that's what it works out to be with the granularity bit set. The AM and WP bits were set to the wrong thing according to the comments next to them I'm pretty sure. If we wanted the other behavior, we might be able to change them back and have it work. The _BASE registers hold the base of segments as they're specified by the ISA. The _EFF_BASE registers hold the base that will actually be used in address calculations based on the mode of the CPU. For instance, if you're in 64 bit mode, the _BASE of DS might still be 0xFFF from when you were in another mode. The _EFF_BASE would be 0 though, since the DS base is ignored in that case. _EFF_BASE may not be used by the KVM CPU, but it should be set up anyway in case we switch back to a regular CPU. Gabe On Wed, Dec 10, 2014 at 10:20 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Thank you for all the clarifications. I see that for SE to work on vmx the TSS limit had to be extended, am and wp bits in CR0 had to be reset and *_EFF_BASE registers had be set in addition to *_BASE registers for TR TSG IDTR. I wonder what is TR_EFF_BASE. It seems that the contents of TR register (which gets passed to kvm) are the same if with or without *_EFF_BASE registers set. Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 1:21 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Ok, I got SE working too. I'll clean up my patch and send that out in a bit. Gabe On Tue, Dec 9, 2014 at 9:41 PM, Gabe Black gabebl...@google.com wrote: I figured out what the other problem was, so here's the review. http://reviews.gem5.org/r/2557/ Gabe On Tue, Dec 9, 2014 at 5:00 PM, Gabe Black gabebl...@google.com wrote: It was attached in my sent mail. Maybe it's being blocked by something? I'm hunting down another problem so I don't want to move my tree around too much, but once that's done I'll post it as a review. Gabe On Tue, Dec 9, 2014 at 4:51 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: I haven't received any attachment to your email. So I don't have your patch. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:42 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) And... it turns out the KVM change wasn't necessary. If you're working from my patch, get rid of where the segment limit is divided by PageBytes. That was only necessary because I wasn't adding 0xFFF to the limit when the granularity bit was set. Gabe On Tue, Dec 9, 2014 at 4:31 PM, Gabe Black gabebl...@google.com wrote: Oh, also segment limits weren't being computed correctly in the installSegDesc function, although I don't think that was from the KVM stuff. Once it was fixed it required adjusting the KVM stuff a little, though. Gabe On Tue, Dec 9, 2014 at 4:29 PM, Gabe Black gabebl...@google.com wrote: Here is my patch so far. There were a few things wrong, although I didn't really keep notes. The limits were mixed up, the long mode bit was set on all descriptors when it's only valid for the code segment, privilege level 0 is the OS and 3 is for applications and not the other way around, and I think the type was being set wrong for one of the segments. Also, the syscall and sysenter registers (star and friends) require the segments in the GDT to be in a particular order which I don't think they were. Gabe On Tue, Dec 9, 2014 at 4:22 PM, Dutu, Alexandru via
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Thank you for all the clarifications. I see that for SE to work on vmx the TSS limit had to be extended, am and wp bits in CR0 had to be reset and *_EFF_BASE registers had be set in addition to *_BASE registers for TR TSG IDTR. I wonder what is TR_EFF_BASE. It seems that the contents of TR register (which gets passed to kvm) are the same if with or without *_EFF_BASE registers set. Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 1:21 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Ok, I got SE working too. I'll clean up my patch and send that out in a bit. Gabe On Tue, Dec 9, 2014 at 9:41 PM, Gabe Black gabebl...@google.com wrote: I figured out what the other problem was, so here's the review. http://reviews.gem5.org/r/2557/ Gabe On Tue, Dec 9, 2014 at 5:00 PM, Gabe Black gabebl...@google.com wrote: It was attached in my sent mail. Maybe it's being blocked by something? I'm hunting down another problem so I don't want to move my tree around too much, but once that's done I'll post it as a review. Gabe On Tue, Dec 9, 2014 at 4:51 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: I haven't received any attachment to your email. So I don't have your patch. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:42 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) And... it turns out the KVM change wasn't necessary. If you're working from my patch, get rid of where the segment limit is divided by PageBytes. That was only necessary because I wasn't adding 0xFFF to the limit when the granularity bit was set. Gabe On Tue, Dec 9, 2014 at 4:31 PM, Gabe Black gabebl...@google.com wrote: Oh, also segment limits weren't being computed correctly in the installSegDesc function, although I don't think that was from the KVM stuff. Once it was fixed it required adjusting the KVM stuff a little, though. Gabe On Tue, Dec 9, 2014 at 4:29 PM, Gabe Black gabebl...@google.com wrote: Here is my patch so far. There were a few things wrong, although I didn't really keep notes. The limits were mixed up, the long mode bit was set on all descriptors when it's only valid for the code segment, privilege level 0 is the OS and 3 is for applications and not the other way around, and I think the type was being set wrong for one of the segments. Also, the syscall and sysenter registers (star and friends) require the segments in the GDT to be in a particular order which I don't think they were. Gabe On Tue, Dec 9, 2014 at 4:22 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: So, I am doing this on an AMD system and I have SE working and am able to get FS entering into virtualized mode. However, in FS I get an early exception while the kernel is booting. This seems a bit different from what Nilay and Adrian observed for FS. Could you please share the diffs that got FS working? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:07 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Oh, I see you have FS working again and not SE. NM, I'll keep looking. Gabe On Tue, Dec 9, 2014 at 4:04 PM, Gabe Black gabebl...@google.com wrote: I have FS working again which is good, but I'm still having problems with SE. If you could let me know what you did to get things going that would be very helpful. Gabe On Tue, Dec 9, 2014 at 10:06 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Adrian, Sorry for missing your first email. I do see the interchanged segment limits for full system mode, though I get a different behaviour on my system. The simulation seems to hang in the following manner: Processor #0 (Bootup-CPU) I/O APIC #1 at 0xFEC0. Setting APIC routing to flat Processors: 1 PANIC: early exception rip 807909a9 error 9 cr2 ff5fd020 Can please provide a patch with all the modifications that fixed the issue on your system? Thank you, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Adrián Colaso Diego via gem5-dev [gem5-dev@gem5.org] Sent: Tuesday, December 09, 2014 2:09 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) You are right Nilay. I sent an email last week but nobody has replied. It seems that descriptors (cdDesc, dsDesc and tssDesc) located in src/arch/x86/system.cc
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Thanks for the clarification, actually WP needs to be reset for CPU switching to work properly. I think it is better to leave alignment check off for SE mode as there is no handler in place for unaligned access exception. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 2:28 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) That's not actually extending the TSS limit, that's what it works out to be with the granularity bit set. The AM and WP bits were set to the wrong thing according to the comments next to them I'm pretty sure. If we wanted the other behavior, we might be able to change them back and have it work. The _BASE registers hold the base of segments as they're specified by the ISA. The _EFF_BASE registers hold the base that will actually be used in address calculations based on the mode of the CPU. For instance, if you're in 64 bit mode, the _BASE of DS might still be 0xFFF from when you were in another mode. The _EFF_BASE would be 0 though, since the DS base is ignored in that case. _EFF_BASE may not be used by the KVM CPU, but it should be set up anyway in case we switch back to a regular CPU. Gabe On Wed, Dec 10, 2014 at 10:20 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Thank you for all the clarifications. I see that for SE to work on vmx the TSS limit had to be extended, am and wp bits in CR0 had to be reset and *_EFF_BASE registers had be set in addition to *_BASE registers for TR TSG IDTR. I wonder what is TR_EFF_BASE. It seems that the contents of TR register (which gets passed to kvm) are the same if with or without *_EFF_BASE registers set. Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Wednesday, December 10, 2014 1:21 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Ok, I got SE working too. I'll clean up my patch and send that out in a bit. Gabe On Tue, Dec 9, 2014 at 9:41 PM, Gabe Black gabebl...@google.com wrote: I figured out what the other problem was, so here's the review. http://reviews.gem5.org/r/2557/ Gabe On Tue, Dec 9, 2014 at 5:00 PM, Gabe Black gabebl...@google.com wrote: It was attached in my sent mail. Maybe it's being blocked by something? I'm hunting down another problem so I don't want to move my tree around too much, but once that's done I'll post it as a review. Gabe On Tue, Dec 9, 2014 at 4:51 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: I haven't received any attachment to your email. So I don't have your patch. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:42 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) And... it turns out the KVM change wasn't necessary. If you're working from my patch, get rid of where the segment limit is divided by PageBytes. That was only necessary because I wasn't adding 0xFFF to the limit when the granularity bit was set. Gabe On Tue, Dec 9, 2014 at 4:31 PM, Gabe Black gabebl...@google.com wrote: Oh, also segment limits weren't being computed correctly in the installSegDesc function, although I don't think that was from the KVM stuff. Once it was fixed it required adjusting the KVM stuff a little, though. Gabe On Tue, Dec 9, 2014 at 4:29 PM, Gabe Black gabebl...@google.com wrote: Here is my patch so far. There were a few things wrong, although I didn't really keep notes. The limits were mixed up, the long mode bit was set on all descriptors when it's only valid for the code segment, privilege level 0 is the OS and 3 is for applications and not the other way around, and I think the type was being set wrong for one of the segments. Also, the syscall and sysenter registers (star and friends) require the segments in the GDT to be in a particular order which I don't think they were. Gabe On Tue, Dec 9, 2014 at 4:22 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: So, I am doing this on an AMD system and I have SE working and am able to get FS entering into virtualized mode. However, in FS I get an early exception while the kernel is booting. This seems a bit different from what Nilay and Adrian observed for FS. Could you please share the diffs that got FS working? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:07 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Adrian, Sorry for missing your first email. I have solved the interchanged segment limits and other bits in segment descriptors for full system mode, though I get a different behavior on my system. The simulation seems to hang in the following manner: Processor #0 (Bootup-CPU) I/O APIC #1 at 0xFEC0. Setting APIC routing to flat Processors: 1 PANIC: early exception rip 807909a9 error 9 cr2 ff5fd020 Can you please provide a patch with all the modifications that fixed the issue on your system? Thank you, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of mike upton via gem5-dev Sent: Tuesday, December 09, 2014 11:52 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Will someone be providing a patch for this? I am happy to test it. From Adrian's description it seems there are a bunch of issues. On Tue, Dec 9, 2014 at 12:09 AM, Adrián Colaso Diego via gem5-dev gem5-dev@gem5.org wrote: You are right Nilay. I sent an email last week but nobody has replied. It seems that descriptors (cdDesc, dsDesc and tssDesc) located in src/arch/x86/system.cc file are not well-initialized and as a consequence kvm does not work when running in full-system mode. Segment limits values (limitHigh and limitLow) are interchanged and several segment descriptor values are wrong too. If these values are corrected kvm works again as before. Adrian El lun, 08-12-2014 a las 22:50 -0600, Nilay Vaish via gem5-dev escribió: I also faced problem in getting KVM CPU to run in FS mode. I figured that the following changeset causes problems: authorAlexandru Dutu alexandru.d...@amd.com Sun Nov 23 18:01:08 2014 -0800 (2 weeks ago) changeset 10554 fe2e2f06a7c8 I saw the hardware reason 0x8021, but did not try to figure what was going on wrong. -- Nilay On Mon, 8 Dec 2014, Gabe Black via gem5-dev wrote: I'm pretty sure entering 64 bit mode is the same between AMD and Intel CPUs. I vaguely remember there being some subtle page table difference though, and gem5 is building the page tables in SE mode instead of the kernel. Gabe On Mon, Dec 8, 2014 at 7:44 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, trace-cmd is a very handy tool to get an overview of what the kvm kernel module is doing before going into gdb. In extreme cases ftrace can be useful as well. What is the error that you are seeing? Is it still failing to enter virtualized mode? If that is the case and the hardware reason is 0x8021, that seems to be an unrecoverable exception (drivers/hv/hyperv_vmbus.h in linux kernel source code). When running in SE mode, we are trying to bring the machine state to full 64bit mode without going through legacy modes. It might be that Intel machines have a different way of going to 64bit mode than AMD machines (different CR4, different way of enabling 64bit mode page tables etc.). I remember dealing with these issue for AMD platforms by going through System Programming manual and making sure gem5 gets all the bits right as there is not much the KVM kernel model will tell about the cause of failure. Best regards, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Gabe Black via gem5-dev [gem5-dev@gem5.org] Sent: Monday, December 08, 2014 7:08 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I'm not an expert either, but I did have problems running KVM in SE mode on an Intel CPU. I didn't look into it that much, but I think things failed in the kernel somewhere. What might be happening is that the different vendors hardware virtualization mechanisms are more or less picky about various things. Something might be set up incorrectly, and one implementation gets more upset about it than the other. I believe there are tools which will help you determine whether your VM state is legal. Perhaps Andreas can tell you more about those? Gabe On Mon, Dec 8, 2014 at 4:29 PM, mike upton via gem5-dev gem5-dev@gem5.org wrote: I have verified that x86 kvm works fine on AMD platforms, but fails on Intel platforms. Any hints about how to narrow down the cause (other than diving into gdb, which I will do). I am not an expert in KVM or how gem5 hooks up to libkvm. ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Adrian, Sorry for missing your first email. I do see the interchanged segment limits for full system mode, though I get a different behaviour on my system. The simulation seems to hang in the following manner: Processor #0 (Bootup-CPU) I/O APIC #1 at 0xFEC0. Setting APIC routing to flat Processors: 1 PANIC: early exception rip 807909a9 error 9 cr2 ff5fd020 Can please provide a patch with all the modifications that fixed the issue on your system? Thank you, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Adrián Colaso Diego via gem5-dev [gem5-dev@gem5.org] Sent: Tuesday, December 09, 2014 2:09 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) You are right Nilay. I sent an email last week but nobody has replied. It seems that descriptors (cdDesc, dsDesc and tssDesc) located in src/arch/x86/system.cc file are not well-initialized and as a consequence kvm does not work when running in full-system mode. Segment limits values (limitHigh and limitLow) are interchanged and several segment descriptor values are wrong too. If these values are corrected kvm works again as before. Adrian El lun, 08-12-2014 a las 22:50 -0600, Nilay Vaish via gem5-dev escribió: I also faced problem in getting KVM CPU to run in FS mode. I figured that the following changeset causes problems: authorAlexandru Dutu alexandru.d...@amd.com Sun Nov 23 18:01:08 2014 -0800 (2 weeks ago) changeset 10554 fe2e2f06a7c8 I saw the hardware reason 0x8021, but did not try to figure what was going on wrong. -- Nilay On Mon, 8 Dec 2014, Gabe Black via gem5-dev wrote: I'm pretty sure entering 64 bit mode is the same between AMD and Intel CPUs. I vaguely remember there being some subtle page table difference though, and gem5 is building the page tables in SE mode instead of the kernel. Gabe On Mon, Dec 8, 2014 at 7:44 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, trace-cmd is a very handy tool to get an overview of what the kvm kernel module is doing before going into gdb. In extreme cases ftrace can be useful as well. What is the error that you are seeing? Is it still failing to enter virtualized mode? If that is the case and the hardware reason is 0x8021, that seems to be an unrecoverable exception (drivers/hv/hyperv_vmbus.h in linux kernel source code). When running in SE mode, we are trying to bring the machine state to full 64bit mode without going through legacy modes. It might be that Intel machines have a different way of going to 64bit mode than AMD machines (different CR4, different way of enabling 64bit mode page tables etc.). I remember dealing with these issue for AMD platforms by going through System Programming manual and making sure gem5 gets all the bits right as there is not much the KVM kernel model will tell about the cause of failure. Best regards, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Gabe Black via gem5-dev [gem5-dev@gem5.org] Sent: Monday, December 08, 2014 7:08 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I'm not an expert either, but I did have problems running KVM in SE mode on an Intel CPU. I didn't look into it that much, but I think things failed in the kernel somewhere. What might be happening is that the different vendors hardware virtualization mechanisms are more or less picky about various things. Something might be set up incorrectly, and one implementation gets more upset about it than the other. I believe there are tools which will help you determine whether your VM state is legal. Perhaps Andreas can tell you more about those? Gabe On Mon, Dec 8, 2014 at 4:29 PM, mike upton via gem5-dev gem5-dev@gem5.org wrote: I have verified that x86 kvm works fine on AMD platforms, but fails on Intel platforms. Any hints about how to narrow down the cause (other than diving into gdb, which I will do). I am not an expert in KVM or how gem5 hooks up to libkvm. ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
So, I am doing this on an AMD system and I have SE working and am able to get FS entering into virtualized mode. However, in FS I get an early exception while the kernel is booting. This seems a bit different from what Nilay and Adrian observed for FS. Could you please share the diffs that got FS working? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:07 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Oh, I see you have FS working again and not SE. NM, I'll keep looking. Gabe On Tue, Dec 9, 2014 at 4:04 PM, Gabe Black gabebl...@google.com wrote: I have FS working again which is good, but I'm still having problems with SE. If you could let me know what you did to get things going that would be very helpful. Gabe On Tue, Dec 9, 2014 at 10:06 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Adrian, Sorry for missing your first email. I do see the interchanged segment limits for full system mode, though I get a different behaviour on my system. The simulation seems to hang in the following manner: Processor #0 (Bootup-CPU) I/O APIC #1 at 0xFEC0. Setting APIC routing to flat Processors: 1 PANIC: early exception rip 807909a9 error 9 cr2 ff5fd020 Can please provide a patch with all the modifications that fixed the issue on your system? Thank you, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Adrián Colaso Diego via gem5-dev [gem5-dev@gem5.org] Sent: Tuesday, December 09, 2014 2:09 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) You are right Nilay. I sent an email last week but nobody has replied. It seems that descriptors (cdDesc, dsDesc and tssDesc) located in src/arch/x86/system.cc file are not well-initialized and as a consequence kvm does not work when running in full-system mode. Segment limits values (limitHigh and limitLow) are interchanged and several segment descriptor values are wrong too. If these values are corrected kvm works again as before. Adrian El lun, 08-12-2014 a las 22:50 -0600, Nilay Vaish via gem5-dev escribió: I also faced problem in getting KVM CPU to run in FS mode. I figured that the following changeset causes problems: authorAlexandru Dutu alexandru.d...@amd.com Sun Nov 23 18:01:08 2014 -0800 (2 weeks ago) changeset 10554 fe2e2f06a7c8 I saw the hardware reason 0x8021, but did not try to figure what was going on wrong. -- Nilay On Mon, 8 Dec 2014, Gabe Black via gem5-dev wrote: I'm pretty sure entering 64 bit mode is the same between AMD and Intel CPUs. I vaguely remember there being some subtle page table difference though, and gem5 is building the page tables in SE mode instead of the kernel. Gabe On Mon, Dec 8, 2014 at 7:44 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, trace-cmd is a very handy tool to get an overview of what the kvm kernel module is doing before going into gdb. In extreme cases ftrace can be useful as well. What is the error that you are seeing? Is it still failing to enter virtualized mode? If that is the case and the hardware reason is 0x8021, that seems to be an unrecoverable exception (drivers/hv/hyperv_vmbus.h in linux kernel source code). When running in SE mode, we are trying to bring the machine state to full 64bit mode without going through legacy modes. It might be that Intel machines have a different way of going to 64bit mode than AMD machines (different CR4, different way of enabling 64bit mode page tables etc.). I remember dealing with these issue for AMD platforms by going through System Programming manual and making sure gem5 gets all the bits right as there is not much the KVM kernel model will tell about the cause of failure. Best regards, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Gabe Black via gem5-dev [gem5-dev@gem5.org] Sent: Monday, December 08, 2014 7:08 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I'm not an expert either, but I did have problems running KVM in SE mode on an Intel CPU. I didn't look into it that much, but I think things failed in the kernel somewhere. What might be happening is that the different vendors hardware virtualization mechanisms are more or less picky about various things. Something might be set up incorrectly, and one implementation gets more upset about it than the other. I believe there are tools which will help you determine whether your VM state is legal. Perhaps
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
I haven't received any attachment to your email. So I don't have your patch. Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:42 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) And... it turns out the KVM change wasn't necessary. If you're working from my patch, get rid of where the segment limit is divided by PageBytes. That was only necessary because I wasn't adding 0xFFF to the limit when the granularity bit was set. Gabe On Tue, Dec 9, 2014 at 4:31 PM, Gabe Black gabebl...@google.com wrote: Oh, also segment limits weren't being computed correctly in the installSegDesc function, although I don't think that was from the KVM stuff. Once it was fixed it required adjusting the KVM stuff a little, though. Gabe On Tue, Dec 9, 2014 at 4:29 PM, Gabe Black gabebl...@google.com wrote: Here is my patch so far. There were a few things wrong, although I didn't really keep notes. The limits were mixed up, the long mode bit was set on all descriptors when it's only valid for the code segment, privilege level 0 is the OS and 3 is for applications and not the other way around, and I think the type was being set wrong for one of the segments. Also, the syscall and sysenter registers (star and friends) require the segments in the GDT to be in a particular order which I don't think they were. Gabe On Tue, Dec 9, 2014 at 4:22 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: So, I am doing this on an AMD system and I have SE working and am able to get FS entering into virtualized mode. However, in FS I get an early exception while the kernel is booting. This seems a bit different from what Nilay and Adrian observed for FS. Could you please share the diffs that got FS working? Thanks, Alex -Original Message- From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black via gem5-dev Sent: Tuesday, December 09, 2014 6:07 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) Oh, I see you have FS working again and not SE. NM, I'll keep looking. Gabe On Tue, Dec 9, 2014 at 4:04 PM, Gabe Black gabebl...@google.com wrote: I have FS working again which is good, but I'm still having problems with SE. If you could let me know what you did to get things going that would be very helpful. Gabe On Tue, Dec 9, 2014 at 10:06 AM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Adrian, Sorry for missing your first email. I do see the interchanged segment limits for full system mode, though I get a different behaviour on my system. The simulation seems to hang in the following manner: Processor #0 (Bootup-CPU) I/O APIC #1 at 0xFEC0. Setting APIC routing to flat Processors: 1 PANIC: early exception rip 807909a9 error 9 cr2 ff5fd020 Can please provide a patch with all the modifications that fixed the issue on your system? Thank you, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Adrián Colaso Diego via gem5-dev [gem5-dev@gem5.org] Sent: Tuesday, December 09, 2014 2:09 AM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) You are right Nilay. I sent an email last week but nobody has replied. It seems that descriptors (cdDesc, dsDesc and tssDesc) located in src/arch/x86/system.cc file are not well-initialized and as a consequence kvm does not work when running in full-system mode. Segment limits values (limitHigh and limitLow) are interchanged and several segment descriptor values are wrong too. If these values are corrected kvm works again as before. Adrian El lun, 08-12-2014 a las 22:50 -0600, Nilay Vaish via gem5-dev escribió: I also faced problem in getting KVM CPU to run in FS mode. I figured that the following changeset causes problems: authorAlexandru Dutu alexandru.d...@amd.com Sun Nov 23 18:01:08 2014 -0800 (2 weeks ago) changeset 10554 fe2e2f06a7c8 I saw the hardware reason 0x8021, but did not try to figure what was going on wrong. -- Nilay On Mon, 8 Dec 2014, Gabe Black via gem5-dev wrote: I'm pretty sure entering 64 bit mode is the same between AMD and Intel CPUs. I vaguely remember there being some subtle page table difference though, and gem5 is building the page tables in SE mode instead of the kernel. Gabe On Mon, Dec 8, 2014 at 7:44 PM, Dutu, Alexandru via gem5-dev gem5-dev@gem5.org wrote: Hi Mike, trace-cmd is a very handy tool to get an overview of what the kvm kernel module is doing before going into gdb. In extreme cases ftrace can be useful as well. What
Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel)
Hi Mike, trace-cmd is a very handy tool to get an overview of what the kvm kernel module is doing before going into gdb. In extreme cases ftrace can be useful as well. What is the error that you are seeing? Is it still failing to enter virtualized mode? If that is the case and the hardware reason is 0x8021, that seems to be an unrecoverable exception (drivers/hv/hyperv_vmbus.h in linux kernel source code). When running in SE mode, we are trying to bring the machine state to full 64bit mode without going through legacy modes. It might be that Intel machines have a different way of going to 64bit mode than AMD machines (different CR4, different way of enabling 64bit mode page tables etc.). I remember dealing with these issue for AMD platforms by going through System Programming manual and making sure gem5 gets all the bits right as there is not much the KVM kernel model will tell about the cause of failure. Best regards, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Gabe Black via gem5-dev [gem5-dev@gem5.org] Sent: Monday, December 08, 2014 7:08 PM To: gem5 Developer List Subject: Re: [gem5-dev] x86 SE kvm functionality (AMD vs Intel) I'm not an expert either, but I did have problems running KVM in SE mode on an Intel CPU. I didn't look into it that much, but I think things failed in the kernel somewhere. What might be happening is that the different vendors hardware virtualization mechanisms are more or less picky about various things. Something might be set up incorrectly, and one implementation gets more upset about it than the other. I believe there are tools which will help you determine whether your VM state is legal. Perhaps Andreas can tell you more about those? Gabe On Mon, Dec 8, 2014 at 4:29 PM, mike upton via gem5-dev gem5-dev@gem5.org wrote: I have verified that x86 kvm works fine on AMD platforms, but fails on Intel platforms. Any hints about how to narrow down the cause (other than diving into gdb, which I will do). I am not an expert in KVM or how gem5 hooks up to libkvm. ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] KvmCPU Behaviour
Hi Andreas, Sorry for the late reply, I have missed your email. I will investigate more the issues with memory controller refresh events and let you know what I find out. Best regards, Alex ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] [gem5-users] x86 SE mode kvm support?
Hi Mike, There are some patches ready to be committed for this to work. I expect them to be committed by the end of this week. Best regards, Alex -- Forwarded message -- From: Mike Upton via gem5-users gem5-us...@gem5.orgmailto:gem5-us...@gem5.org Date: Mon, Nov 17, 2014 at 3:15 PM Subject: [gem5-users] x86 SE mode kvm support? To: gem5-us...@gem5.orgmailto:gem5-us...@gem5.org gem5-us...@gem5.orgmailto:gem5-us...@gem5.org How do I do a gem5 run in SE with KVM enabled? I tried: ./build/X86/gem5.opt configs/example/se.py –c hello_world_path/hello --cpu-type=kvm I get an error : Can’t resolve proxy ‘any’ of type ‘KvmVM’ from ‘system.cpu’ Looking in Options.py, I think I have the right type setting… I see a few other folks have asked the same question, but I could not find an answer. Thanks, Mike ___ gem5-users mailing list gem5-us...@gem5.orgmailto:gem5-us...@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] Ethernet device doesn't work with O3 cpu model in X86 ISA
From what I see the MTRRs are not used by gem5. On a real system I would imagine the hardware would check MTRRs and PATs to determine if an access is uncacheable or not. It seems that memory accesses to the range of physical addresses [0xC000-0x] are not marked as uncacheable by the TLB with either atomic or O3 model. I am curios to why the same problem does not manifest with the atomic model. In addition, to get rid of all the hardcoded physical address range checks that the memory subsystem has to do, it would be a bit cleaner to set the memory request flags based on the flags in page table entries. In this way the flags for cacheable/uncacheable, read-only/read-write etc. will be set once in the page table, the TLB will be updated according to the page table and TLB::finalizePhysical can actually set memory access request flags in the same way the TlbEntry has them set. I am preparing to post a patch that adds these flags in the TlbEntry and changes the PageTable mapping interface to account for the extra flags. Best regards, Alex From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Nilay Vaish via gem5-dev [gem5-dev@gem5.org] Sent: Wednesday, October 01, 2014 9:41 AM To: Mohammad Alian via gem5-dev Subject: Re: [gem5-dev] Ethernet device doesn't work with O3 cpu model in X86 ISA How did you do this? My current understanding is that we need to support MTRRs to mark the PCI address ranges as uncacheable. -- Nilay On Mon, 8 Sep 2014, Mohammad Alian via gem5-dev wrote: Hi, For some reasons gem5 doesn't recognize that PCI device address ranges are uncacheable. I set UNCACHEABLE flag for all accesses in the PCI physical address range and it fixed the problem. Thanks, Mohammad On Friday, September 5, 2014 10:48 AM, Steve Reinhardt via gem5-dev gem5-dev@gem5.org wrote: I don't know exactly what the problem is, but it's very suspicious that the device would receive a cache-block-sized access when it's in uncacheable memory space. My guess is that O3 is for some reason issuing a misspeculated cacheable access to the physical address where the device lives. Normally these would be suppressed when the TLB lookup indicates that the address is uncacheable, but if you're in kernel mode, you could end up with speculative accesses to the translation bypass range. To be honest, I don't know off the top of my head how this is dealt with in actual x86 systems... it would certainly be safest to suppress all speculative accesses to bypassed addresses, but I suspect that we can be smarter than that using the MTRRs. I could dig into it but don't have time at the moment, so if someone else knows these details, please speak up. Steve On Thu, Sep 4, 2014 at 2:30 PM, Mohammad Alian via gem5-dev gem5-dev@gem5.org wrote: Hi all, Ethernet device work perfectly with atomic cpu in X86 ISA but I get an error when I switch cpu to detailed O3 cpu. The error is: gem5.opt: build/X86/dev/i8254xGBe.cc:182: virtual Tick IGbE::read(PacketPtr): Assertion `pkt-getSize() == 4' failed. Program aborted at tick 131516086224500 Aborted I noticed that the size of packet that Ethenet device receive with O3 cpu is same as cache line size(64Bytes) however the Ethernet device just support 32bit accesses. Any Idea how to fix this problem? Thank you, Mohammad Alian ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev ___ gem5-dev mailing list gem5-dev@gem5.org http://m5sim.org/mailman/listinfo/gem5-dev
Re: [gem5-dev] changeset in gem5: mem: adding architectural page table support ...
Hi Andreas,
You are right, grep -ERsonH '.{80,}$' reveales 7 lines being longer.
Sorry about this, next time I will double check. Thanks for pointing this out!
Alex
From: gem5-dev [gem5-dev-boun...@gem5.org] on behalf of Andreas Hansson via
gem5-dev [gem5-dev@gem5.org]
Sent: Thursday, August 28, 2014 11:31 AM
To: gem5 Developer List
Subject: Re: [gem5-dev] changeset in gem5: mem: adding architectural page table
support ...
Hi Alexandru,
I merely wanted to point out that there are quite some places where the
new changes violate the 80 char limit. It might not be worth fixing on its
own, but for any future changes...
Thanks,
Andreas
On 28/08/2014 16:31, Alexandru via gem5-dev gem5-dev@gem5.org wrote:
changeset bec0c5ffc323 in /z/repo/gem5
details: http://repo.gem5.org/gem5?cmd=changeset;node=bec0c5ffc323
description:
mem: adding architectural page table support for SE mode
This patch enables the use of page tables that are stored in system
memory
and respect x86 specification, in SE mode. It defines an architectural
page table for x86 as a MultiLevelPageTable class and puts a placeholder
class for other ISAs page tables, giving the possibility for future
implementation.
diffstat:
src/arch/alpha/process.hh| 3 +
src/arch/arm/process.hh | 3 +
src/arch/mips/process.hh | 3 +
src/arch/power/process.hh| 3 +
src/arch/sparc/process.hh| 3 +
src/arch/x86/pagetable.hh| 85
src/arch/x86/pagetable_walker.cc | 17
src/arch/x86/process.hh | 9
src/arch/x86/system.hh | 14 ++
src/mem/SConscript | 2 +
src/sim/Process.py | 2 +
src/sim/process.cc | 5 +-
src/sim/process.hh | 2 +
13 files changed, 133 insertions(+), 18 deletions(-)
diffs (truncated from 305 to 300 lines):
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/alpha/process.hh
--- a/src/arch/alpha/process.hhTue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/alpha/process.hhThu Aug 28 10:11:44 2014 -0500
@@ -55,4 +55,7 @@
void setSyscallReturn(ThreadContext *tc, SyscallReturn return_value);
};
+/* No architectural page table defined for this ISA */
+typedef NoArchPageTable ArchPageTable;
+
#endif // __ARCH_ALPHA_PROCESS_HH__
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/arm/process.hh
--- a/src/arch/arm/process.hh Tue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/arm/process.hh Thu Aug 28 10:11:44 2014 -0500
@@ -98,5 +98,8 @@
void setSyscallReturn(ThreadContext *tc, SyscallReturn return_value);
};
+/* No architectural page table defined for this ISA */
+typedef NoArchPageTable ArchPageTable;
+
#endif // __ARM_PROCESS_HH__
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/mips/process.hh
--- a/src/arch/mips/process.hh Tue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/mips/process.hh Thu Aug 28 10:11:44 2014 -0500
@@ -59,5 +59,8 @@
void setSyscallReturn(ThreadContext *tc, SyscallReturn return_value);
};
+/* No architectural page table defined for this ISA */
+typedef NoArchPageTable ArchPageTable;
+
#endif // __MIPS_PROCESS_HH__
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/power/process.hh
--- a/src/arch/power/process.hhTue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/power/process.hhThu Aug 28 10:11:44 2014 -0500
@@ -58,5 +58,8 @@
void setSyscallReturn(ThreadContext *tc, SyscallReturn return_value);
};
+/* No architectural page table defined for this ISA */
+typedef NoArchPageTable ArchPageTable;
+
#endif // __POWER_PROCESS_HH__
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/sparc/process.hh
--- a/src/arch/sparc/process.hhTue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/sparc/process.hhThu Aug 28 10:11:44 2014 -0500
@@ -131,4 +131,7 @@
void setSyscallArg(ThreadContext *tc, int i, SparcISA::IntReg val);
};
+/* No architectural page table defined for this ISA */
+typedef NoArchPageTable ArchPageTable;
+
#endif // __SPARC_PROCESS_HH__
diff -r 77af86f37337 -r bec0c5ffc323 src/arch/x86/pagetable.hh
--- a/src/arch/x86/pagetable.hhTue Apr 01 12:18:12 2014 -0500
+++ b/src/arch/x86/pagetable.hhThu Aug 28 10:11:44 2014 -0500
@@ -1,4 +1,5 @@
/*
+ * Copyright (c) 2014 Advanced Micro Devices, Inc.
* Copyright (c) 2007 The Hewlett-Packard Development Company
* All rights reserved.
*
@@ -42,11 +43,15 @@
#include iostream
#include string
+#include vector
#include base/bitunion.hh
#include base/misc.hh
#include base/types.hh
#include base/trie.hh
+#include cpu/thread_context.hh
+#include arch/x86/system.hh
+#include debug/MMU.hh
class Checkpoint;
@@ -73,6 +78,25 @@
Bitfield31, 22 norml2;
EndBitUnion(VAddr)
+// Unfortunately, the placement of the base field in a page table
entry is
+// very erratic and would make a mess here. It might be moved here
at some
+
