Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 30/01/12 22:46, Peter Maydell wrote: On 20 January 2012 02:59, Christoffer Dall c.d...@virtualopensystems.com wrote: There's a new list of issues available at: https://github.com/virtualopensystems/linux-kvm-arm/issues Thanks for putting this up. Here's a couple more for you :-) * Support guest kernels configured for LPAE At the moment (well, if you have Marc's 3.3rc1 tree with the A15 L2 cache control register bodge in it) you can boot an A15 kernel configured without LPAE as a KVM guest, but an LPAE kernel with LPAE enabled will not boot. This probably mostly requires supporting the 64 bit wide cp15 registers that LPAE implies. So I had a look at that one. LPAE is not the problem (an LPAE kernel runs fine as a guest), but a KVM-enabled kernel dies trying to execute an SMC instruction. The obvious fix would be to test for the virt extensions in ID_PFR1 before enabling KVM, and mask out the virt extensions from the same register in the guest by trapping CP15-c0. But c0 is used by cache maintenance operations, and trapping would lead to unacceptable performance degradation. Ideas anyone? M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 1 February 2012 12:11, Marc Zyngier marc.zyng...@arm.com wrote: The obvious fix would be to test for the virt extensions in ID_PFR1 before enabling KVM, and mask out the virt extensions from the same register in the guest by trapping CP15-c0. But c0 is used by cache maintenance operations, and trapping would lead to unacceptable performance degradation. You can just set HCR.TID3 to get hyp traps on only the feature ID registers, right? (traps access to only ID_PFR0, ID_PFR1, ID_DFR0, ID_AFR0, ID_MMFR0, ID_MMFR1, ID_MMFR2, ID_MMFR3, ID_ISAR0, ID_ISAR1, ID_ISAR2, ID_ISAR3, ID_ISAR4, ID_ISAR5, MVFR0, and MVFR1, none of which should be on performance-critical paths I hope.) -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 01/02/12 12:20, Peter Maydell wrote: On 1 February 2012 12:11, Marc Zyngier marc.zyng...@arm.com wrote: The obvious fix would be to test for the virt extensions in ID_PFR1 before enabling KVM, and mask out the virt extensions from the same register in the guest by trapping CP15-c0. But c0 is used by cache maintenance operations, and trapping would lead to unacceptable performance degradation. You can just set HCR.TID3 to get hyp traps on only the feature ID registers, right? (traps access to only ID_PFR0, ID_PFR1, ID_DFR0, ID_AFR0, ID_MMFR0, ID_MMFR1, ID_MMFR2, ID_MMFR3, ID_ISAR0, ID_ISAR1, ID_ISAR2, ID_ISAR3, ID_ISAR4, ID_ISAR5, MVFR0, and MVFR1, none of which should be on performance-critical paths I hope.) Good catch. It looks much better, except that ID_MMFR3 is used in cpu_v7_set_pte_ext(), which is called each time you map a page. Ouch. I'm not even sure we can the cache ID_MMFR3 in the kernel (multi-cluster systems may have different features). M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 1 February 2012 13:40, Marc Zyngier marc.zyng...@arm.com wrote: On 01/02/12 12:20, Peter Maydell wrote: On 1 February 2012 12:11, Marc Zyngier marc.zyng...@arm.com wrote: The obvious fix would be to test for the virt extensions in ID_PFR1 before enabling KVM, and mask out the virt extensions from the same register in the guest by trapping CP15-c0. But c0 is used by cache maintenance operations, and trapping would lead to unacceptable performance degradation. You can just set HCR.TID3 to get hyp traps on only the feature ID registers, right? (traps access to only ID_PFR0, ID_PFR1, ID_DFR0, ID_AFR0, ID_MMFR0, ID_MMFR1, ID_MMFR2, ID_MMFR3, ID_ISAR0, ID_ISAR1, ID_ISAR2, ID_ISAR3, ID_ISAR4, ID_ISAR5, MVFR0, and MVFR1, none of which should be on performance-critical paths I hope.) Good catch. It looks much better, except that ID_MMFR3 is used in cpu_v7_set_pte_ext(), which is called each time you map a page. Ouch. I'm not even sure we can the cache ID_MMFR3 in the kernel (multi-cluster systems may have different features). We only look at the coherent-walk bits there, right? We can just cache a single does any core require clean-to-PoU after translation table update? [ie does any core have 0 in this field?] flag, I guess. The chances that you actually have a cluster with different cores differing here seem quite low[*], and if we do encounter one then cleaning the dcache line even on the cores that don't need it would be a safe if slightly less than optimal approach... [*] particularly since every multicore Cortex A-class processor supports coherent walk... -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
If the final implementation is going to be as per this thread: http://lists.linaro.org/pipermail/boot-architecture/2011-August/58.html then this shouldn't actually be an issue right? On Wed, Feb 1, 2012 at 8:40 AM, Marc Zyngier marc.zyng...@arm.com wrote: On 01/02/12 12:20, Peter Maydell wrote: On 1 February 2012 12:11, Marc Zyngier marc.zyng...@arm.com wrote: The obvious fix would be to test for the virt extensions in ID_PFR1 before enabling KVM, and mask out the virt extensions from the same register in the guest by trapping CP15-c0. But c0 is used by cache maintenance operations, and trapping would lead to unacceptable performance degradation. You can just set HCR.TID3 to get hyp traps on only the feature ID registers, right? (traps access to only ID_PFR0, ID_PFR1, ID_DFR0, ID_AFR0, ID_MMFR0, ID_MMFR1, ID_MMFR2, ID_MMFR3, ID_ISAR0, ID_ISAR1, ID_ISAR2, ID_ISAR3, ID_ISAR4, ID_ISAR5, MVFR0, and MVFR1, none of which should be on performance-critical paths I hope.) Good catch. It looks much better, except that ID_MMFR3 is used in cpu_v7_set_pte_ext(), which is called each time you map a page. Ouch. I'm not even sure we can the cache ID_MMFR3 in the kernel (multi-cluster systems may have different features). M. -- Jazz is not dead. It just smells funny... -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 01/30/2012 11:46 PM, Peter Maydell wrote: On 20 January 2012 02:59, Christoffer Dall c.d...@virtualopensystems.com wrote: There's a new list of issues available at: https://github.com/virtualopensystems/linux-kvm-arm/issues Thanks for putting this up. Here's a couple more for you :-) * Support guest kernels configured for LPAE At the moment (well, if you have Marc's 3.3rc1 tree with the A15 L2 cache control register bodge in it) you can boot an A15 kernel configured without LPAE as a KVM guest, but an LPAE kernel with LPAE enabled will not boot. This probably mostly requires supporting the 64 bit wide cp15 registers that LPAE implies. * handle QEMU being ^C'd Currently if you ^C the qemu process then instead of a nice clean exit things go a bit pear shaped, with apparently part of the qemu/kvm combo having quit and the rest not and lots of error messages being emitted. (I admit to not having looked at this one enough to be certain tht it's a kernel side thing rather than a QEMU one; I'm just guessing.) -- PMM I took the initiative to add them to that list, thanks. -Antonios -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 20 January 2012 02:59, Christoffer Dall c.d...@virtualopensystems.com wrote: There's a new list of issues available at: https://github.com/virtualopensystems/linux-kvm-arm/issues Thanks for putting this up. Here's a couple more for you :-) * Support guest kernels configured for LPAE At the moment (well, if you have Marc's 3.3rc1 tree with the A15 L2 cache control register bodge in it) you can boot an A15 kernel configured without LPAE as a KVM guest, but an LPAE kernel with LPAE enabled will not boot. This probably mostly requires supporting the 64 bit wide cp15 registers that LPAE implies. * handle QEMU being ^C'd Currently if you ^C the qemu process then instead of a nice clean exit things go a bit pear shaped, with apparently part of the qemu/kvm combo having quit and the rest not and lots of error messages being emitted. (I admit to not having looked at this one enough to be certain tht it's a kernel side thing rather than a QEMU one; I'm just guessing.) -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On 30.01.2012, at 23:46, Peter Maydell wrote: On 20 January 2012 02:59, Christoffer Dall c.d...@virtualopensystems.com wrote: There's a new list of issues available at: https://github.com/virtualopensystems/linux-kvm-arm/issues Thanks for putting this up. Here's a couple more for you :-) * Support guest kernels configured for LPAE At the moment (well, if you have Marc's 3.3rc1 tree with the A15 L2 cache control register bodge in it) you can boot an A15 kernel configured without LPAE as a KVM guest, but an LPAE kernel with LPAE enabled will not boot. This probably mostly requires supporting the 64 bit wide cp15 registers that LPAE implies. * handle QEMU being ^C'd Currently if you ^C the qemu process then instead of a nice clean exit things go a bit pear shaped, with apparently part of the qemu/kvm combo having quit and the rest not and lots of error messages being emitted. (I admit to not having looked at this one enough to be certain tht it's a kernel side thing rather than a QEMU one; I'm just guessing.) That one sounds like missing signal detection in the kvm code. I remember having had that in ppc kvm too during early development. Alex -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
There's a new list of issues available at: https://github.com/virtualopensystems/linux-kvm-arm/issues -Christoffer On Thu, Jan 12, 2012 at 11:15 AM, Christoffer Dall c.d...@virtualopensystems.com wrote: On Thu, Jan 12, 2012 at 12:19 AM, Peter Maydell peter.mayd...@linaro.org wrote: On 12 January 2012 03:29, Christoffer Dall c.d...@virtualopensystems.com wrote: On Jan 11, 2012, at 8:48 AM, Peter Maydell wrote: (It would be quite useful to be able to boot a reasonably modern [read, ARMv7, Thumb2, VFPv3] guest userspace; does anybody plan to work on this part soon?) We have booted the linaro init environment and recent Angstrom distributions. Android is being actively tested. What specifically did you have in mind? I meant fixes for these todo list items: - Support guest Thumb mode for MMIO emulation - Support Neon/VFP in guests (the fpu regs struct is empty ATM) Have I missed patches which added these? Last time I tried booting a stock linaro userspace it died very early because of the missing Thumb2 guest support. no, you're not missing anything. I am in fact working on these. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/13] KVM/ARM Implementation
On 12 January 2012 03:29, Christoffer Dall c.d...@virtualopensystems.com wrote: On Jan 11, 2012, at 8:48 AM, Peter Maydell wrote: (It would be quite useful to be able to boot a reasonably modern [read, ARMv7, Thumb2, VFPv3] guest userspace; does anybody plan to work on this part soon?) We have booted the linaro init environment and recent Angstrom distributions. Android is being actively tested. What specifically did you have in mind? I meant fixes for these todo list items: - Support guest Thumb mode for MMIO emulation - Support Neon/VFP in guests (the fpu regs struct is empty ATM) Have I missed patches which added these? Last time I tried booting a stock linaro userspace it died very early because of the missing Thumb2 guest support. -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Android-virt] [PATCH v5 00/13] KVM/ARM Implementation
On Thu, Jan 12, 2012 at 12:19 AM, Peter Maydell peter.mayd...@linaro.org wrote: On 12 January 2012 03:29, Christoffer Dall c.d...@virtualopensystems.com wrote: On Jan 11, 2012, at 8:48 AM, Peter Maydell wrote: (It would be quite useful to be able to boot a reasonably modern [read, ARMv7, Thumb2, VFPv3] guest userspace; does anybody plan to work on this part soon?) We have booted the linaro init environment and recent Angstrom distributions. Android is being actively tested. What specifically did you have in mind? I meant fixes for these todo list items: - Support guest Thumb mode for MMIO emulation - Support Neon/VFP in guests (the fpu regs struct is empty ATM) Have I missed patches which added these? Last time I tried booting a stock linaro userspace it died very early because of the missing Thumb2 guest support. no, you're not missing anything. I am in fact working on these. -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/13] KVM/ARM Implementation
On 11 December 2011 19:23, Christoffer Dall c.d...@virtualopensystems.com wrote: On Sun, Dec 11, 2011 at 6:32 AM, Peter Maydell peter.mayd...@linaro.org wrote: On 11 December 2011 10:24, Christoffer Dall c.d...@virtualopensystems.com wrote: Still on the to-do list: - Reuse VMIDs - Fix SMP host support - Fix SMP guest support - Support guest Thumb mode for MMIO emulation - Further testing - Performance improvements Other items for this list: - Support Neon/VFP in guests (the fpu regs struct is empty ATM) - Support guest debugging ok, thanks, will add these to the list. I have a feeling it will keep growing for a while :) Do you have a kernel-side TODO list somewhere public (wiki page?) (It would be quite useful to be able to boot a reasonably modern [read, ARMv7, Thumb2, VFPv3] guest userspace; does anybody plan to work on this part soon?) thanks -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/13] KVM/ARM Implementation
On Jan 11, 2012, at 8:48 AM, Peter Maydell wrote: On 11 December 2011 19:23, Christoffer Dall c.d...@virtualopensystems.com wrote: On Sun, Dec 11, 2011 at 6:32 AM, Peter Maydell peter.mayd...@linaro.org wrote: On 11 December 2011 10:24, Christoffer Dall c.d...@virtualopensystems.com wrote: Still on the to-do list: - Reuse VMIDs - Fix SMP host support - Fix SMP guest support - Support guest Thumb mode for MMIO emulation - Further testing - Performance improvements Other items for this list: - Support Neon/VFP in guests (the fpu regs struct is empty ATM) - Support guest debugging ok, thanks, will add these to the list. I have a feeling it will keep growing for a while :) Do you have a kernel-side TODO list somewhere public (wiki page?) I wanted to create this as issues on the github repos... (It would be quite useful to be able to boot a reasonably modern [read, ARMv7, Thumb2, VFPv3] guest userspace; does anybody plan to work on this part soon?) We have booted the linaro init environment and recent Angstrom distributions. Android is being actively tested. What specifically did you have in mind? -Christoffer-- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v5 00/13] KVM/ARM Implementation
The following series implements KVM support for ARM processors, specifically on the Cortex A-15 platform. The patch series applies to commit 0ec4044a029b5ba9ed6dc7c52390c25da717e184 on Catalin Marinas' linux-arm-arch tree. This is Version 5 of the patch series, but the first two versions were reviewed outside of the KVM mailing list. Changes can also be pulled from: git://github.com/virtualopensystems/linux-kvm-arm.git kvm-a15-v5 The implementation is broken up into a logical set of patches, the first one containing a skeleton of files, makefile changes, the basic user space interface and KVM architecture specific stubs. Subsequent patches implement parts of the system as listed: 1. Skeleton 2. Identity Mapping for Hyp mode 3. Hypervisor initialization 4. Memory virtualization setup (hyp mode mappings and 2nd stage) 5. Inject IRQs and FIQs from userspace 6. World-switch implementation and Hyp exception vectors 7. Emulation framework and CP15 emulation 8. Handle guest user memory aborts 9. Handle guest MMIO aborts 10. Support guest wait-for-interrupt instructions. 11. Initial SMP host support (incomplete!) 12. Fix guest view of MPIDR 13. Initial SMP guest support (incomplete!) Testing: Limited testing, but have run GCC inside guest, which compiled a small hello-world program, which was successfully run. Hardware still unavailable, so all testing has been done on ARM Fast Models. For a guide on how to set up a testing environment and try out these patches, see: http://www.virtualopensystems.com/media/pdf/kvm-arm-guide.pdf https://wiki.linaro.org/PeterMaydell/A15OnFastModels Still on the to-do list: - Reuse VMIDs - Fix SMP host support - Fix SMP guest support - Support guest Thumb mode for MMIO emulation - Further testing - Performance improvements Changes since v4: - Addressed reviewer comments from v4 * cleanup debug and trace code * remove printks * fixup kvm_arch_vcpu_ioctl_run * add trace details to mmio emulation - Fix from Marc Zyngier: Move kvm_guest_enter/exit into non-preemptible section (squashed into world-switch patch) - Cleanup create_hyp_mappings/remove_hyp_mappings from Marc Zyngier (squashed into hypervisor initialization patch) - Removed the remove_hyp_mappings feature. Removing hypervisor mappings could potentially unmap other important data shared in the same page. - Removed the arm_ prefix from the arch-specific files. - Initial SMP host/guest support Changes since v3: - v4 actually works, fully boots a guest - Support compiling as a module - Use static inlines instead of macros for vcpu_reg and friends - Optimize kvm_vcpu_reg function - Use Ftrace for trace capabilities - Updated documentation and commenting - Use KVM_IRQ_LINE instead of KVM_INTERRUPT - Emulates load/store instructions not supported through HSR syndrome information. - Frees 2nd stage translation tables on VM teardown - Handles IRQ/FIQ instructions - Handles more CP15 accesses - Support guest WFI calls - Uses debugfs instead of /proc - Support compiling in Thumb mode Changes since v2: - Performs world-switch code - Maps guest memory using 2nd stage translation - Emulates co-processor 15 instructions - Forwards I/O faults to QEMU. --- Christoffer Dall (12): ARM: KVM: Initial skeleton to compile KVM support ARM: KVM: Hypervisor identity mapping ARM: KVM: Add hypervisor inititalization ARM: KVM: Memory virtualization setup ARM: KVM: Inject IRQs and FIQs from userspace ARM: KVM: World-switch implementation ARM: KVM: Emulation framework and CP15 emulation ARM: KVM: Handle guest faults in KVM ARM: KVM: Handle I/O aborts ARM: KVM: Guest wait-for-interrupts (WFI) support ARM: KVM: Support SMP hosts ARM: KVM: Support SMP guests Marc Zyngier (1): ARM: KVM: Fix guest view of MPIDR Documentation/virtual/kvm/api.txt | 10 arch/arm/Kconfig|2 arch/arm/Makefile |1 arch/arm/include/asm/kvm.h | 75 +++ arch/arm/include/asm/kvm_arm.h | 130 + arch/arm/include/asm/kvm_asm.h | 51 ++ arch/arm/include/asm/kvm_emulate.h | 100 arch/arm/include/asm/kvm_host.h | 112 arch/arm/include/asm/kvm_mmu.h | 42 ++ arch/arm/include/asm/kvm_para.h |9 arch/arm/include/asm/pgtable-3level-hwdef.h |5 arch/arm/include/asm/pgtable-3level.h | 12 arch/arm/include/asm/pgtable.h | 11 arch/arm/include/asm/unified.h | 12 arch/arm/kernel/armksyms.c |7 arch/arm/kernel/asm-offsets.c | 34 + arch/arm/kernel/entry-armv.S|1 arch/arm/kvm/Kconfig| 44 ++ arch/arm/kvm/Makefile | 17 + arch/arm/kvm/arm.c | 716
Re: [PATCH v5 00/13] KVM/ARM Implementation
On 11 December 2011 10:24, Christoffer Dall c.d...@virtualopensystems.com wrote: The following series implements KVM support for ARM processors, specifically on the Cortex A-15 platform. Still on the to-do list: - Reuse VMIDs - Fix SMP host support - Fix SMP guest support - Support guest Thumb mode for MMIO emulation - Further testing - Performance improvements Other items for this list: - Support Neon/VFP in guests (the fpu regs struct is empty ATM) - Support guest debugging I couldn't see any support for the TLS registers in your cp15 emulation: did I miss it, or do we handle it without needing to trap? -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/13] KVM/ARM Implementation
On Sun, Dec 11, 2011 at 6:32 AM, Peter Maydell peter.mayd...@linaro.org wrote: On 11 December 2011 10:24, Christoffer Dall c.d...@virtualopensystems.com wrote: The following series implements KVM support for ARM processors, specifically on the Cortex A-15 platform. Still on the to-do list: - Reuse VMIDs - Fix SMP host support - Fix SMP guest support - Support guest Thumb mode for MMIO emulation - Further testing - Performance improvements Other items for this list: - Support Neon/VFP in guests (the fpu regs struct is empty ATM) - Support guest debugging ok, thanks, will add these to the list. I have a feeling it will keep growing for a while :) I couldn't see any support for the TLS registers in your cp15 emulation: did I miss it, or do we handle it without needing to trap? by TLS you mean the cp15, c13 registers (tid and friends?) If so, I handle these in the world-switch code (look at read_cp15_state and write_cp15_state). otherwise, help me out on the acronym... -Christoffer -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v5 00/13] KVM/ARM Implementation
On 11 December 2011 19:23, Christoffer Dall c.d...@virtualopensystems.com wrote: by TLS you mean the cp15, c13 registers (tid and friends?) If so, I handle these in the world-switch code (look at read_cp15_state and write_cp15_state). otherwise, help me out on the acronym... Yes, those are the ones (TLS == thread local storage). Thanks for the pointer. -- PMM -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
