Re: [PATCH] KVM: PPC: Book3S HV: fix handling for interrupted H_ENTER_NESTED
On Thu, 2018-11-08 at 21:27 -0600, Michael Roth wrote: > While running a nested guest VCPU on L0 via H_ENTER_NESTED hcall, a > pending signal in the L0 QEMU process can generate the following > sequence: > > ret0 = kvmppc_pseries_do_hcall() > ret1 = kvmhv_enter_nested_guest() > ret2 = kvmhv_run_single_vcpu() > if (ret2 == -EINTR) > return H_INTERRUPT > if (ret1 == H_INTERRUPT) > kvmppc_set_gpr(vcpu, 3, 0) > return -EINTR > /* skipped: */ > kvmppc_set_gpr(vcpu, 3, ret) > vcpu->arch.hcall_needed = 0 > return RESUME_GUEST > > which causes an exit to L0 userspace with ret0 == -EINTR. > > The intention seems to be to set the hcall return value to 0 (via > VCPU r3) so that L1 will see a successful return from H_ENTER_NESTED > once we resume executing the VCPU. However, because we don't set > vcpu->arch.hcall_needed = 0, we do the following once userspace > resumes execution via kvm_arch_vcpu_ioctl_run(): > > ... > } else if (vcpu->arch.hcall_needed) { > int i > > kvmppc_set_gpr(vcpu, 3, run->papr_hcall.ret); > for (i = 0; i < 9; ++i) >kvmppc_set_gpr(vcpu, 4 + i, run->papr_hcall.args[i]); > vcpu->arch.hcall_needed = 0; > > since vcpu->arch.hcall_needed == 1 indicates that userspace should > have handled the hcall and stored the return value in > run->papr_hcall.ret. Since that's not the case here, we can get an > unexpected value in VCPU r3, which can result in > kvmhv_p9_guest_entry() reporting an unexpected trap value when it > returns from H_ENTER_NESTED, causing the following register dump to > console via subsequent call to kvmppc_handle_exit_hv() in L1: > > [ 350.612854] vcpu f9564cf8 (0): > [ 350.612915] pc = c013eb98 msr = 80009033 trap > = 1 > [ 350.613020] r 0 = c04b9044 r16 = > [ 350.613075] r 1 = c0007cffba30 r17 = > [ 350.613120] r 2 = c178c100 r18 = 7fffc24f3b50 > [ 350.613166] r 3 = c0007ef52480 r19 = 7fffc24fff58 > [ 350.613212] r 4 = r20 = 0a1e96ece9d0 > [ 350.613253] r 5 = 70616d00746f6f72 r21 = 0a1ea117c9b0 > [ 350.613295] r 6 = 0020 r22 = 0a1ea1184360 > [ 350.613338] r 7 = c000783be440 r23 = 0003 > [ 350.613380] r 8 = fffc r24 = 0a1e96e9e124 > [ 350.613423] r 9 = c0007ef52490 r25 = 07ff > [ 350.613469] r10 = 0004 r26 = c0007eb2f7a0 > [ 350.613513] r11 = b0616d0009eccdb2 r27 = c0007cffbb10 > [ 350.613556] r12 = c04b9000 r28 = c0007d83a2c0 > [ 350.613597] r13 = c1b0 r29 = c000783cdf68 > [ 350.613639] r14 = r30 = > [ 350.613681] r15 = r31 = c0007cffbbf0 > [ 350.613723] ctr = c04b9000 lr = c04b9044 > [ 350.613765] srr0 = 772f954dd48c srr1 = 8280f033 > [ 350.613808] sprg0 = sprg1 = c1b0 > [ 350.613859] sprg2 = 772f9565a280 sprg3 = > [ 350.613911] cr = 88002848 xer = 2004 dsisr = > 4200 > [ 350.613962] dar = 772f9539 > [ 350.614031] fault dar = c00244b278c0 dsisr = > [ 350.614073] SLB (0 entries): > [ 350.614157] lpcr = 004003d40413 sdr1 = > last_inst = > [ 350.614252] trap=0x1 | pc=0xc013eb98 | > msr=0x80009033 > > followed by L1's QEMU reporting the following before stopping > execution > of the nested guest: > > KVM: unknown exit, hardware reason 1 > NIP c013eb98 LR c04b9044 CTR c04b9000 XER > 2004 CPU#0 > MSR 80009033 HID0 HF 8000 > iidx 3 didx 3 > TB DECR > GPR00 c04b9044 c0007cffba30 c178c100 > c0007ef52480 > GPR04 70616d00746f6f72 0020 > c000783be440 > GPR08 fffc c0007ef52490 0004 > b0616d0009eccdb2 > GPR12 c04b9000 c1b0 > > GPR16 7fffc24f3b50 > 7fffc24fff58 > GPR20 0a1e96ece9d0 0a1ea117c9b0 0a1ea1184360 > 0003 > GPR24 0a1e96e9e124 07ff c0007eb2f7a0 > c0007cffbb10 > GPR28 c0007d83a2c0 c000783cdf68 > c0007cffbbf0 > CR 88002848 [ L L - - E L G L ] RES > >SRR0 772f954dd48c SRR1 8280f033PVR > 004e1202 VRSAVE > SPRG0 SPRG1 c1b0 SPRG2 > 772f9565a280 SPRG3 > SPRG4 SPRG5 SPRG6 > SPRG7 > HSRR0 HSRR1 >CFAR >LPCR
[PATCH] KVM: PPC: Book3S HV: fix handling for interrupted H_ENTER_NESTED
While running a nested guest VCPU on L0 via H_ENTER_NESTED hcall, a pending signal in the L0 QEMU process can generate the following sequence: ret0 = kvmppc_pseries_do_hcall() ret1 = kvmhv_enter_nested_guest() ret2 = kvmhv_run_single_vcpu() if (ret2 == -EINTR) return H_INTERRUPT if (ret1 == H_INTERRUPT) kvmppc_set_gpr(vcpu, 3, 0) return -EINTR /* skipped: */ kvmppc_set_gpr(vcpu, 3, ret) vcpu->arch.hcall_needed = 0 return RESUME_GUEST which causes an exit to L0 userspace with ret0 == -EINTR. The intention seems to be to set the hcall return value to 0 (via VCPU r3) so that L1 will see a successful return from H_ENTER_NESTED once we resume executing the VCPU. However, because we don't set vcpu->arch.hcall_needed = 0, we do the following once userspace resumes execution via kvm_arch_vcpu_ioctl_run(): ... } else if (vcpu->arch.hcall_needed) { int i kvmppc_set_gpr(vcpu, 3, run->papr_hcall.ret); for (i = 0; i < 9; ++i) kvmppc_set_gpr(vcpu, 4 + i, run->papr_hcall.args[i]); vcpu->arch.hcall_needed = 0; since vcpu->arch.hcall_needed == 1 indicates that userspace should have handled the hcall and stored the return value in run->papr_hcall.ret. Since that's not the case here, we can get an unexpected value in VCPU r3, which can result in kvmhv_p9_guest_entry() reporting an unexpected trap value when it returns from H_ENTER_NESTED, causing the following register dump to console via subsequent call to kvmppc_handle_exit_hv() in L1: [ 350.612854] vcpu f9564cf8 (0): [ 350.612915] pc = c013eb98 msr = 80009033 trap = 1 [ 350.613020] r 0 = c04b9044 r16 = [ 350.613075] r 1 = c0007cffba30 r17 = [ 350.613120] r 2 = c178c100 r18 = 7fffc24f3b50 [ 350.613166] r 3 = c0007ef52480 r19 = 7fffc24fff58 [ 350.613212] r 4 = r20 = 0a1e96ece9d0 [ 350.613253] r 5 = 70616d00746f6f72 r21 = 0a1ea117c9b0 [ 350.613295] r 6 = 0020 r22 = 0a1ea1184360 [ 350.613338] r 7 = c000783be440 r23 = 0003 [ 350.613380] r 8 = fffc r24 = 0a1e96e9e124 [ 350.613423] r 9 = c0007ef52490 r25 = 07ff [ 350.613469] r10 = 0004 r26 = c0007eb2f7a0 [ 350.613513] r11 = b0616d0009eccdb2 r27 = c0007cffbb10 [ 350.613556] r12 = c04b9000 r28 = c0007d83a2c0 [ 350.613597] r13 = c1b0 r29 = c000783cdf68 [ 350.613639] r14 = r30 = [ 350.613681] r15 = r31 = c0007cffbbf0 [ 350.613723] ctr = c04b9000 lr = c04b9044 [ 350.613765] srr0 = 772f954dd48c srr1 = 8280f033 [ 350.613808] sprg0 = sprg1 = c1b0 [ 350.613859] sprg2 = 772f9565a280 sprg3 = [ 350.613911] cr = 88002848 xer = 2004 dsisr = 4200 [ 350.613962] dar = 772f9539 [ 350.614031] fault dar = c00244b278c0 dsisr = [ 350.614073] SLB (0 entries): [ 350.614157] lpcr = 004003d40413 sdr1 = last_inst = [ 350.614252] trap=0x1 | pc=0xc013eb98 | msr=0x80009033 followed by L1's QEMU reporting the following before stopping execution of the nested guest: KVM: unknown exit, hardware reason 1 NIP c013eb98 LR c04b9044 CTR c04b9000 XER 2004 CPU#0 MSR 80009033 HID0 HF 8000 iidx 3 didx 3 TB DECR GPR00 c04b9044 c0007cffba30 c178c100 c0007ef52480 GPR04 70616d00746f6f72 0020 c000783be440 GPR08 fffc c0007ef52490 0004 b0616d0009eccdb2 GPR12 c04b9000 c1b0 GPR16 7fffc24f3b50 7fffc24fff58 GPR20 0a1e96ece9d0 0a1ea117c9b0 0a1ea1184360 0003 GPR24 0a1e96e9e124 07ff c0007eb2f7a0 c0007cffbb10 GPR28 c0007d83a2c0 c000783cdf68 c0007cffbbf0 CR 88002848 [ L L - - E L G L ] RES SRR0 772f954dd48c SRR1 8280f033PVR 004e1202 VRSAVE SPRG0 SPRG1 c1b0 SPRG2 772f9565a280 SPRG3 SPRG4 SPRG5 SPRG6 SPRG7 HSRR0 HSRR1 CFAR LPCR 03d40413 PTCR DAR 772f9539 DSISR 4200 Fix this by setting vcpu->arch.hcall_needed = 0 to indicate completion of H_ENTER_NESTED before we exit to L0 userspace. Cc: linuxppc-...@ozlabs.org Cc: David Gibson Cc: Paul Mackerras Cc: