Re: [PATCH] KVM: PPC: Book3S HV: fix handling for interrupted H_ENTER_NESTED
On Thu, 2018-11-08 at 21:27 -0600, Michael Roth wrote:
> While running a nested guest VCPU on L0 via H_ENTER_NESTED hcall, a
> pending signal in the L0 QEMU process can generate the following
> sequence:
>
> ret0 = kvmppc_pseries_do_hcall()
> ret1 = kvmhv_enter_nested_guest()
> ret2 = kvmhv_run_single_vcpu()
> if (ret2 == -EINTR)
> return H_INTERRUPT
> if (ret1 == H_INTERRUPT)
> kvmppc_set_gpr(vcpu, 3, 0)
> return -EINTR
> /* skipped: */
> kvmppc_set_gpr(vcpu, 3, ret)
> vcpu->arch.hcall_needed = 0
> return RESUME_GUEST
>
> which causes an exit to L0 userspace with ret0 == -EINTR.
>
> The intention seems to be to set the hcall return value to 0 (via
> VCPU r3) so that L1 will see a successful return from H_ENTER_NESTED
> once we resume executing the VCPU. However, because we don't set
> vcpu->arch.hcall_needed = 0, we do the following once userspace
> resumes execution via kvm_arch_vcpu_ioctl_run():
>
> ...
> } else if (vcpu->arch.hcall_needed) {
> int i
>
> kvmppc_set_gpr(vcpu, 3, run->papr_hcall.ret);
> for (i = 0; i < 9; ++i)
>kvmppc_set_gpr(vcpu, 4 + i, run->papr_hcall.args[i]);
> vcpu->arch.hcall_needed = 0;
>
> since vcpu->arch.hcall_needed == 1 indicates that userspace should
> have handled the hcall and stored the return value in
> run->papr_hcall.ret. Since that's not the case here, we can get an
> unexpected value in VCPU r3, which can result in
> kvmhv_p9_guest_entry() reporting an unexpected trap value when it
> returns from H_ENTER_NESTED, causing the following register dump to
> console via subsequent call to kvmppc_handle_exit_hv() in L1:
>
> [ 350.612854] vcpu f9564cf8 (0):
> [ 350.612915] pc = c013eb98 msr = 80009033 trap
> = 1
> [ 350.613020] r 0 = c04b9044 r16 =
> [ 350.613075] r 1 = c0007cffba30 r17 =
> [ 350.613120] r 2 = c178c100 r18 = 7fffc24f3b50
> [ 350.613166] r 3 = c0007ef52480 r19 = 7fffc24fff58
> [ 350.613212] r 4 = r20 = 0a1e96ece9d0
> [ 350.613253] r 5 = 70616d00746f6f72 r21 = 0a1ea117c9b0
> [ 350.613295] r 6 = 0020 r22 = 0a1ea1184360
> [ 350.613338] r 7 = c000783be440 r23 = 0003
> [ 350.613380] r 8 = fffc r24 = 0a1e96e9e124
> [ 350.613423] r 9 = c0007ef52490 r25 = 07ff
> [ 350.613469] r10 = 0004 r26 = c0007eb2f7a0
> [ 350.613513] r11 = b0616d0009eccdb2 r27 = c0007cffbb10
> [ 350.613556] r12 = c04b9000 r28 = c0007d83a2c0
> [ 350.613597] r13 = c1b0 r29 = c000783cdf68
> [ 350.613639] r14 = r30 =
> [ 350.613681] r15 = r31 = c0007cffbbf0
> [ 350.613723] ctr = c04b9000 lr = c04b9044
> [ 350.613765] srr0 = 772f954dd48c srr1 = 8280f033
> [ 350.613808] sprg0 = sprg1 = c1b0
> [ 350.613859] sprg2 = 772f9565a280 sprg3 =
> [ 350.613911] cr = 88002848 xer = 2004 dsisr =
> 4200
> [ 350.613962] dar = 772f9539
> [ 350.614031] fault dar = c00244b278c0 dsisr =
> [ 350.614073] SLB (0 entries):
> [ 350.614157] lpcr = 004003d40413 sdr1 =
> last_inst =
> [ 350.614252] trap=0x1 | pc=0xc013eb98 |
> msr=0x80009033
>
> followed by L1's QEMU reporting the following before stopping
> execution
> of the nested guest:
>
> KVM: unknown exit, hardware reason 1
> NIP c013eb98 LR c04b9044 CTR c04b9000 XER
> 2004 CPU#0
> MSR 80009033 HID0 HF 8000
> iidx 3 didx 3
> TB DECR
> GPR00 c04b9044 c0007cffba30 c178c100
> c0007ef52480
> GPR04 70616d00746f6f72 0020
> c000783be440
> GPR08 fffc c0007ef52490 0004
> b0616d0009eccdb2
> GPR12 c04b9000 c1b0
>
> GPR16 7fffc24f3b50
> 7fffc24fff58
> GPR20 0a1e96ece9d0 0a1ea117c9b0 0a1ea1184360
> 0003
> GPR24 0a1e96e9e124 07ff c0007eb2f7a0
> c0007cffbb10
> GPR28 c0007d83a2c0 c000783cdf68
> c0007cffbbf0
> CR 88002848 [ L L - - E L G L ] RES
>
>SRR0 772f954dd48c SRR1 8280f033PVR
> 004e1202 VRSAVE
> SPRG0 SPRG1 c1b0 SPRG2
> 772f9565a280 SPRG3
> SPRG4 SPRG5 SPRG6
> SPRG7
> HSRR0 HSRR1
>CFAR
>LPCR
[PATCH] KVM: PPC: Book3S HV: fix handling for interrupted H_ENTER_NESTED
While running a nested guest VCPU on L0 via H_ENTER_NESTED hcall, a
pending signal in the L0 QEMU process can generate the following
sequence:
ret0 = kvmppc_pseries_do_hcall()
ret1 = kvmhv_enter_nested_guest()
ret2 = kvmhv_run_single_vcpu()
if (ret2 == -EINTR)
return H_INTERRUPT
if (ret1 == H_INTERRUPT)
kvmppc_set_gpr(vcpu, 3, 0)
return -EINTR
/* skipped: */
kvmppc_set_gpr(vcpu, 3, ret)
vcpu->arch.hcall_needed = 0
return RESUME_GUEST
which causes an exit to L0 userspace with ret0 == -EINTR.
The intention seems to be to set the hcall return value to 0 (via
VCPU r3) so that L1 will see a successful return from H_ENTER_NESTED
once we resume executing the VCPU. However, because we don't set
vcpu->arch.hcall_needed = 0, we do the following once userspace
resumes execution via kvm_arch_vcpu_ioctl_run():
...
} else if (vcpu->arch.hcall_needed) {
int i
kvmppc_set_gpr(vcpu, 3, run->papr_hcall.ret);
for (i = 0; i < 9; ++i)
kvmppc_set_gpr(vcpu, 4 + i, run->papr_hcall.args[i]);
vcpu->arch.hcall_needed = 0;
since vcpu->arch.hcall_needed == 1 indicates that userspace should
have handled the hcall and stored the return value in
run->papr_hcall.ret. Since that's not the case here, we can get an
unexpected value in VCPU r3, which can result in
kvmhv_p9_guest_entry() reporting an unexpected trap value when it
returns from H_ENTER_NESTED, causing the following register dump to
console via subsequent call to kvmppc_handle_exit_hv() in L1:
[ 350.612854] vcpu f9564cf8 (0):
[ 350.612915] pc = c013eb98 msr = 80009033 trap = 1
[ 350.613020] r 0 = c04b9044 r16 =
[ 350.613075] r 1 = c0007cffba30 r17 =
[ 350.613120] r 2 = c178c100 r18 = 7fffc24f3b50
[ 350.613166] r 3 = c0007ef52480 r19 = 7fffc24fff58
[ 350.613212] r 4 = r20 = 0a1e96ece9d0
[ 350.613253] r 5 = 70616d00746f6f72 r21 = 0a1ea117c9b0
[ 350.613295] r 6 = 0020 r22 = 0a1ea1184360
[ 350.613338] r 7 = c000783be440 r23 = 0003
[ 350.613380] r 8 = fffc r24 = 0a1e96e9e124
[ 350.613423] r 9 = c0007ef52490 r25 = 07ff
[ 350.613469] r10 = 0004 r26 = c0007eb2f7a0
[ 350.613513] r11 = b0616d0009eccdb2 r27 = c0007cffbb10
[ 350.613556] r12 = c04b9000 r28 = c0007d83a2c0
[ 350.613597] r13 = c1b0 r29 = c000783cdf68
[ 350.613639] r14 = r30 =
[ 350.613681] r15 = r31 = c0007cffbbf0
[ 350.613723] ctr = c04b9000 lr = c04b9044
[ 350.613765] srr0 = 772f954dd48c srr1 = 8280f033
[ 350.613808] sprg0 = sprg1 = c1b0
[ 350.613859] sprg2 = 772f9565a280 sprg3 =
[ 350.613911] cr = 88002848 xer = 2004 dsisr = 4200
[ 350.613962] dar = 772f9539
[ 350.614031] fault dar = c00244b278c0 dsisr =
[ 350.614073] SLB (0 entries):
[ 350.614157] lpcr = 004003d40413 sdr1 = last_inst =
[ 350.614252] trap=0x1 | pc=0xc013eb98 | msr=0x80009033
followed by L1's QEMU reporting the following before stopping execution
of the nested guest:
KVM: unknown exit, hardware reason 1
NIP c013eb98 LR c04b9044 CTR c04b9000 XER
2004 CPU#0
MSR 80009033 HID0 HF 8000 iidx 3 didx 3
TB DECR
GPR00 c04b9044 c0007cffba30 c178c100 c0007ef52480
GPR04 70616d00746f6f72 0020 c000783be440
GPR08 fffc c0007ef52490 0004 b0616d0009eccdb2
GPR12 c04b9000 c1b0
GPR16 7fffc24f3b50 7fffc24fff58
GPR20 0a1e96ece9d0 0a1ea117c9b0 0a1ea1184360 0003
GPR24 0a1e96e9e124 07ff c0007eb2f7a0 c0007cffbb10
GPR28 c0007d83a2c0 c000783cdf68 c0007cffbbf0
CR 88002848 [ L L - - E L G L ] RES
SRR0 772f954dd48c SRR1 8280f033PVR 004e1202 VRSAVE
SPRG0 SPRG1 c1b0 SPRG2 772f9565a280 SPRG3
SPRG4 SPRG5 SPRG6 SPRG7
HSRR0 HSRR1
CFAR
LPCR 03d40413
PTCR DAR 772f9539 DSISR 4200
Fix this by setting vcpu->arch.hcall_needed = 0 to indicate completion
of H_ENTER_NESTED before we exit to L0 userspace.
Cc: linuxppc-...@ozlabs.org
Cc: David Gibson
Cc: Paul Mackerras
Cc:
