Re: [RFC PATCH 00/16] KVM RISC-V Support
On 30/07/19 15:50, Anup Patel wrote: >> BTW, since IPIs are handled in the SBI I wouldn't bother with in-kernel >> PLIC emulation unless you can demonstrate performance improvements (for >> example due to irqfd). In fact, it may be more interesting to add > > I thought VHOST requires irqfd and we would certainly endup providing > in-kernel PLIC emulation to support VHOST. vhost only needs an eventfd, userspace can poll the eventfd and inject the irq as usual with KVM_INTERRUPT. Of course that can be slower, but you can benchmark it and see if it's indeed a good reason for in-kernel PLIC. >> plumbing for userspace handling of selected SBI calls (in addition to >> get/putchar, sbi_system_reset and sbi_hart_down look like good >> candidates in SBI v0.2). > > The get/putchar SBI v0.1 calls won't be encouraged going forward because > we already have earlycon implmentation in-place and Guest kernel can directly > write to UART registers for earlyprints. > If we still wanted to implement get/putchar calls then we would need a RISC-V > specific exit reason in KVM. We have tried to avoid RISC-V specific IOCTLs > or exit reason in this series. Sounds good. Paolo >> >>> We were thinking to keep KVM RISC-V disabled by default (i.e. keep it >>> experimental) until we have validated it on some FPGA or real HW. For now, >>> users can explicitly enable it and play-around on QEMU emulation. I hope >>> this is fine with most people ? >> >> That's certainly okay with me. >> > > Thanks, > Anup >
Re: [RFC PATCH 00/16] KVM RISC-V Support
On Tue, Jul 30, 2019 at 5:03 PM Paolo Bonzini wrote: > > On 30/07/19 07:26, Anup Patel wrote: > > Here's a brief TODO list which we want to immediately work upon after this > > series: > > 1. Handle trap from unpriv access in SBI v0.1 emulation > > 2. In-kernel PLIC emulation > > 3. SBI v0.2 emulation in-kernel > > 4. SBI v0.2 hart hotplug emulation in-kernel > > 5. . and so on . > > > > We will include above TODO list in v2 series cover letter as well. > > I guess I gave you a bunch of extra items in today's more thorough > review. :) Thanks, your review comments are very useful. We will address all of them. > > BTW, since IPIs are handled in the SBI I wouldn't bother with in-kernel > PLIC emulation unless you can demonstrate performance improvements (for > example due to irqfd). In fact, it may be more interesting to add I thought VHOST requires irqfd and we would certainly endup providing in-kernel PLIC emulation to support VHOST. > plumbing for userspace handling of selected SBI calls (in addition to > get/putchar, sbi_system_reset and sbi_hart_down look like good > candidates in SBI v0.2). The get/putchar SBI v0.1 calls won't be encouraged going forward because we already have earlycon implmentation in-place and Guest kernel can directly write to UART registers for earlyprints. If we still wanted to implement get/putchar calls then we would need a RISC-V specific exit reason in KVM. We have tried to avoid RISC-V specific IOCTLs or exit reason in this series. > > > We were thinking to keep KVM RISC-V disabled by default (i.e. keep it > > experimental) until we have validated it on some FPGA or real HW. For now, > > users can explicitly enable it and play-around on QEMU emulation. I hope > > this is fine with most people ? > > That's certainly okay with me. > Thanks, Anup
Re: [RFC PATCH 00/16] KVM RISC-V Support
On 30/07/19 07:26, Anup Patel wrote: > Here's a brief TODO list which we want to immediately work upon after this > series: > 1. Handle trap from unpriv access in SBI v0.1 emulation > 2. In-kernel PLIC emulation > 3. SBI v0.2 emulation in-kernel > 4. SBI v0.2 hart hotplug emulation in-kernel > 5. . and so on . > > We will include above TODO list in v2 series cover letter as well. I guess I gave you a bunch of extra items in today's more thorough review. :) BTW, since IPIs are handled in the SBI I wouldn't bother with in-kernel PLIC emulation unless you can demonstrate performance improvements (for example due to irqfd). In fact, it may be more interesting to add plumbing for userspace handling of selected SBI calls (in addition to get/putchar, sbi_system_reset and sbi_hart_down look like good candidates in SBI v0.2). > We were thinking to keep KVM RISC-V disabled by default (i.e. keep it > experimental) until we have validated it on some FPGA or real HW. For now, > users can explicitly enable it and play-around on QEMU emulation. I hope > this is fine with most people ? That's certainly okay with me. Paolo
Re: [RFC PATCH 00/16] KVM RISC-V Support
On Jul 30 2019, Anup Patel wrote: > On Tue, Jul 30, 2019 at 12:23 PM Andreas Schwab wrote: >> >> ERROR: "riscv_cs_get_mult_shift" [arch/riscv/kvm/kvm.ko] undefined! >> ERROR: "riscv_isa" [arch/riscv/kvm/kvm.ko] undefined! >> ERROR: "smp_send_reschedule" [arch/riscv/kvm/kvm.ko] undefined! >> ERROR: "riscv_timebase" [arch/riscv/kvm/kvm.ko] undefined! > > Strange, we are not seeing these compile errors. None of these symbols are exported. > Anyway, please ensure that you apply Atish's KVM prep patches > (https://lkml.org/lkml/2019/7/26/1271) on Linux-5.3-rcX before applying > this series. None of these patches contain EXPORT_SYMBOL declarations. Andreas. -- Andreas Schwab, SUSE Labs, sch...@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Re: [RFC PATCH 00/16] KVM RISC-V Support
On Tue, Jul 30, 2019 at 12:23 PM Andreas Schwab wrote: > > ERROR: "riscv_cs_get_mult_shift" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "riscv_isa" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "smp_send_reschedule" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "riscv_timebase" [arch/riscv/kvm/kvm.ko] undefined! Found the issue. These symbols are not exported and you are building KVM RISC-V as module. Thanks for reporting. We will fix it. Regards, Anup
Re: [RFC PATCH 00/16] KVM RISC-V Support
On Tue, Jul 30, 2019 at 12:23 PM Andreas Schwab wrote: > > ERROR: "riscv_cs_get_mult_shift" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "riscv_isa" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "smp_send_reschedule" [arch/riscv/kvm/kvm.ko] undefined! > ERROR: "riscv_timebase" [arch/riscv/kvm/kvm.ko] undefined! Strange, we are not seeing these compile errors. Anyway, please ensure that you apply Atish's KVM prep patches (https://lkml.org/lkml/2019/7/26/1271) on Linux-5.3-rcX before applying this series. Regards, Anup
Re: [RFC PATCH 00/16] KVM RISC-V Support
ERROR: "riscv_cs_get_mult_shift" [arch/riscv/kvm/kvm.ko] undefined! ERROR: "riscv_isa" [arch/riscv/kvm/kvm.ko] undefined! ERROR: "smp_send_reschedule" [arch/riscv/kvm/kvm.ko] undefined! ERROR: "riscv_timebase" [arch/riscv/kvm/kvm.ko] undefined! Andreas. -- Andreas Schwab, SUSE Labs, sch...@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Re: [RFC PATCH 00/16] KVM RISC-V Support
On Tue, Jul 30, 2019 at 3:17 AM Paolo Bonzini wrote: > > On 29/07/19 13:56, Anup Patel wrote: > > This series adds initial KVM RISC-V support. Currently, we are able to boot > > RISC-V 64bit Linux Guests with multiple VCPUs. > > > > Few key aspects of KVM RISC-V added by this series are: > > 1. Minimal possible KVM world-switch which touches only GPRs and few CSRs. > > 2. Full Guest/VM switch is done via vcpu_get/vcpu_put infrastructure. > > 3. KVM ONE_REG interface for VCPU register access from user-space. > > 4. PLIC emulation is done in user-space. In-kernel PLIC emulation, will > >be added in future. > > 5. Timer and IPI emuation is done in-kernel. > > 6. MMU notifiers supported. > > 7. FP lazy save/restore supported. > > 8. SBI v0.1 emulation for KVM Guest available. > > > > More feature additions and enhancments will follow after this series and > > eventually KVM RISC-V will be at-par with other architectures. > > This looks clean and it shouldn't take long to have it merged. Please > sort out the MAINTAINERS additions. It would also be nice if > tools/testing/selftests/kvm/ worked with RISC-V from the beginning; > there have been recent ARM and s390 ports that you can take some > inspiration from. Thanks Paolo. We will certainly include a patch in v2 series for MAINTAINERS entry. We referred existing implementation of KVM ARM/ARM64, KVM powerpc and KVM mips when we started KVM RISC-V port. Here's a brief TODO list which we want to immediately work upon after this series: 1. Handle trap from unpriv access in SBI v0.1 emulation 2. In-kernel PLIC emulation 3. SBI v0.2 emulation in-kernel 4. SBI v0.2 hart hotplug emulation in-kernel 5. . and so on . We will include above TODO list in v2 series cover letter as well. Apart from above, we also have a more exhaustive TODO list based on study of other KVM ports which we want to discuss at upcoming LPC 2019. We were thinking to keep KVM RISC-V disabled by default (i.e. keep it experimental) until we have validated it on some FPGA or real HW. For now, users can explicitly enable it and play-around on QEMU emulation. I hope this is fine with most people ? Regards, Anup
Re: [RFC PATCH 00/16] KVM RISC-V Support
On 29/07/19 13:56, Anup Patel wrote: > This series adds initial KVM RISC-V support. Currently, we are able to boot > RISC-V 64bit Linux Guests with multiple VCPUs. > > Few key aspects of KVM RISC-V added by this series are: > 1. Minimal possible KVM world-switch which touches only GPRs and few CSRs. > 2. Full Guest/VM switch is done via vcpu_get/vcpu_put infrastructure. > 3. KVM ONE_REG interface for VCPU register access from user-space. > 4. PLIC emulation is done in user-space. In-kernel PLIC emulation, will >be added in future. > 5. Timer and IPI emuation is done in-kernel. > 6. MMU notifiers supported. > 7. FP lazy save/restore supported. > 8. SBI v0.1 emulation for KVM Guest available. > > More feature additions and enhancments will follow after this series and > eventually KVM RISC-V will be at-par with other architectures. This looks clean and it shouldn't take long to have it merged. Please sort out the MAINTAINERS additions. It would also be nice if tools/testing/selftests/kvm/ worked with RISC-V from the beginning; there have been recent ARM and s390 ports that you can take some inspiration from. Paolo > This series is based upon KVM pre-patches sent by Atish earlier > (https://lkml.org/lkml/2019/7/26/1271) and it can be found in > riscv_kvm_v1 branch at: > https//github.com/avpatel/linux.git > > Our work-in-progress KVMTOOL RISC-V port can be found in riscv_v1 branch at: > https//github.com/avpatel/kvmtool.git > > We need OpenSBI with RISC-V hypervisor extension support which can be > found in hyp_ext_changes_v1 branch at: > https://github.com/riscv/opensbi.git > > The QEMU RISC-V hypervisor emulation is done by Alistair and is available > in riscv-hyp-work.next branch at: > https://github.com/alistair23/qemu.git > > To play around with KVM RISC-V, here are few reference commands: > 1) To cross-compile KVMTOOL: >$ make lkvm-static > 2) To launch RISC-V Host Linux: >$ qemu-system-riscv64 -monitor null -cpu rv64,h=true -M virt \ >-m 512M -display none -serial mon:stdio \ >-kernel opensbi/build/platform/qemu/virt/firmware/fw_jump.elf \ >-device loader,file=build-riscv64/arch/riscv/boot/Image,addr=0x8020 \ >-initrd ./rootfs_kvm_riscv64.img \ >-append "root=/dev/ram rw console=ttyS0 earlycon=sbi" > 3) To launch RISC-V Guest Linux with 9P rootfs: >$ ./apps/lkvm-static run -m 128 -c2 --console serial \ >-p "console=ttyS0 earlycon=uart8250,mmio,0x3f8" -k ./apps/Image --debug > 4) To launch RISC-V Guest Linux with initrd: >$ ./apps/lkvm-static run -m 128 -c2 --console serial \ >-p "console=ttyS0 earlycon=uart8250,mmio,0x3f8" -k ./apps/Image \ >-i ./apps/rootfs.img --debug > > Anup Patel (13): > KVM: RISC-V: Add KVM_REG_RISCV for ONE_REG interface > RISC-V: Add hypervisor extension related CSR defines > RISC-V: Add initial skeletal KVM support > RISC-V: KVM: Implement VCPU create, init and destroy functions > RISC-V: KVM: Implement VCPU interrupts and requests handling > RISC-V: KVM: Implement KVM_GET_ONE_REG/KVM_SET_ONE_REG ioctls > RISC-V: KVM: Implement VCPU world-switch > RISC-V: KVM: Handle MMIO exits for VCPU > RISC-V: KVM: Handle WFI exits for VCPU > RISC-V: KVM: Implement VMID allocator > RISC-V: KVM: Implement stage2 page table programming > RISC-V: KVM: Implement MMU notifiers > RISC-V: Enable VIRTIO drivers in RV64 and RV32 defconfig > > Atish Patra (3): > RISC-V: KVM: Add timer functionality > RISC-V: KVM: FP lazy save/restore > RISC-V: KVM: Add SBI v0.1 support > > arch/riscv/Kconfig | 2 + > arch/riscv/Makefile | 2 + > arch/riscv/configs/defconfig| 23 +- > arch/riscv/configs/rv32_defconfig | 13 + > arch/riscv/include/asm/csr.h| 58 ++ > arch/riscv/include/asm/kvm_host.h | 232 ++ > arch/riscv/include/asm/kvm_vcpu_timer.h | 32 + > arch/riscv/include/asm/pgtable-bits.h | 1 + > arch/riscv/include/uapi/asm/kvm.h | 74 ++ > arch/riscv/kernel/asm-offsets.c | 148 > arch/riscv/kvm/Kconfig | 34 + > arch/riscv/kvm/Makefile | 14 + > arch/riscv/kvm/main.c | 64 ++ > arch/riscv/kvm/mmu.c| 904 > arch/riscv/kvm/tlb.S| 42 ++ > arch/riscv/kvm/vcpu.c | 817 + > arch/riscv/kvm/vcpu_exit.c | 553 +++ > arch/riscv/kvm/vcpu_sbi.c | 118 > arch/riscv/kvm/vcpu_switch.S| 367 ++ > arch/riscv/kvm/vcpu_timer.c | 106 +++ > arch/riscv/kvm/vm.c | 107 +++ > arch/riscv/kvm/vmid.c | 130 > drivers/clocksource/timer-riscv.c | 6 + > include/clocksource/timer-riscv.h | 14 + > include/uapi/linux/kvm.h| 1 + > 25 files changed, 3857 insertions(+), 5 deletions(-) > create mode 100644
[RFC PATCH 00/16] KVM RISC-V Support
This series adds initial KVM RISC-V support. Currently, we are able to boot RISC-V 64bit Linux Guests with multiple VCPUs. Few key aspects of KVM RISC-V added by this series are: 1. Minimal possible KVM world-switch which touches only GPRs and few CSRs. 2. Full Guest/VM switch is done via vcpu_get/vcpu_put infrastructure. 3. KVM ONE_REG interface for VCPU register access from user-space. 4. PLIC emulation is done in user-space. In-kernel PLIC emulation, will be added in future. 5. Timer and IPI emuation is done in-kernel. 6. MMU notifiers supported. 7. FP lazy save/restore supported. 8. SBI v0.1 emulation for KVM Guest available. More feature additions and enhancments will follow after this series and eventually KVM RISC-V will be at-par with other architectures. This series is based upon KVM pre-patches sent by Atish earlier (https://lkml.org/lkml/2019/7/26/1271) and it can be found in riscv_kvm_v1 branch at: https//github.com/avpatel/linux.git Our work-in-progress KVMTOOL RISC-V port can be found in riscv_v1 branch at: https//github.com/avpatel/kvmtool.git We need OpenSBI with RISC-V hypervisor extension support which can be found in hyp_ext_changes_v1 branch at: https://github.com/riscv/opensbi.git The QEMU RISC-V hypervisor emulation is done by Alistair and is available in riscv-hyp-work.next branch at: https://github.com/alistair23/qemu.git To play around with KVM RISC-V, here are few reference commands: 1) To cross-compile KVMTOOL: $ make lkvm-static 2) To launch RISC-V Host Linux: $ qemu-system-riscv64 -monitor null -cpu rv64,h=true -M virt \ -m 512M -display none -serial mon:stdio \ -kernel opensbi/build/platform/qemu/virt/firmware/fw_jump.elf \ -device loader,file=build-riscv64/arch/riscv/boot/Image,addr=0x8020 \ -initrd ./rootfs_kvm_riscv64.img \ -append "root=/dev/ram rw console=ttyS0 earlycon=sbi" 3) To launch RISC-V Guest Linux with 9P rootfs: $ ./apps/lkvm-static run -m 128 -c2 --console serial \ -p "console=ttyS0 earlycon=uart8250,mmio,0x3f8" -k ./apps/Image --debug 4) To launch RISC-V Guest Linux with initrd: $ ./apps/lkvm-static run -m 128 -c2 --console serial \ -p "console=ttyS0 earlycon=uart8250,mmio,0x3f8" -k ./apps/Image \ -i ./apps/rootfs.img --debug Anup Patel (13): KVM: RISC-V: Add KVM_REG_RISCV for ONE_REG interface RISC-V: Add hypervisor extension related CSR defines RISC-V: Add initial skeletal KVM support RISC-V: KVM: Implement VCPU create, init and destroy functions RISC-V: KVM: Implement VCPU interrupts and requests handling RISC-V: KVM: Implement KVM_GET_ONE_REG/KVM_SET_ONE_REG ioctls RISC-V: KVM: Implement VCPU world-switch RISC-V: KVM: Handle MMIO exits for VCPU RISC-V: KVM: Handle WFI exits for VCPU RISC-V: KVM: Implement VMID allocator RISC-V: KVM: Implement stage2 page table programming RISC-V: KVM: Implement MMU notifiers RISC-V: Enable VIRTIO drivers in RV64 and RV32 defconfig Atish Patra (3): RISC-V: KVM: Add timer functionality RISC-V: KVM: FP lazy save/restore RISC-V: KVM: Add SBI v0.1 support arch/riscv/Kconfig | 2 + arch/riscv/Makefile | 2 + arch/riscv/configs/defconfig| 23 +- arch/riscv/configs/rv32_defconfig | 13 + arch/riscv/include/asm/csr.h| 58 ++ arch/riscv/include/asm/kvm_host.h | 232 ++ arch/riscv/include/asm/kvm_vcpu_timer.h | 32 + arch/riscv/include/asm/pgtable-bits.h | 1 + arch/riscv/include/uapi/asm/kvm.h | 74 ++ arch/riscv/kernel/asm-offsets.c | 148 arch/riscv/kvm/Kconfig | 34 + arch/riscv/kvm/Makefile | 14 + arch/riscv/kvm/main.c | 64 ++ arch/riscv/kvm/mmu.c| 904 arch/riscv/kvm/tlb.S| 42 ++ arch/riscv/kvm/vcpu.c | 817 + arch/riscv/kvm/vcpu_exit.c | 553 +++ arch/riscv/kvm/vcpu_sbi.c | 118 arch/riscv/kvm/vcpu_switch.S| 367 ++ arch/riscv/kvm/vcpu_timer.c | 106 +++ arch/riscv/kvm/vm.c | 107 +++ arch/riscv/kvm/vmid.c | 130 drivers/clocksource/timer-riscv.c | 6 + include/clocksource/timer-riscv.h | 14 + include/uapi/linux/kvm.h| 1 + 25 files changed, 3857 insertions(+), 5 deletions(-) create mode 100644 arch/riscv/include/asm/kvm_host.h create mode 100644 arch/riscv/include/asm/kvm_vcpu_timer.h create mode 100644 arch/riscv/include/uapi/asm/kvm.h create mode 100644 arch/riscv/kvm/Kconfig create mode 100644 arch/riscv/kvm/Makefile create mode 100644 arch/riscv/kvm/main.c create mode 100644 arch/riscv/kvm/mmu.c create mode 100644 arch/riscv/kvm/tlb.S create mode 100644 arch/riscv/kvm/vcpu.c create mode 100644 arch/riscv/kvm/vcpu_exit.c create mode 100644 arch/riscv/kvm/vcpu_sbi.c create mode 100644
