On Thu, Jun 25, 2020 at 12:28:26PM +0200, Paolo Bonzini wrote:
> On 25/06/20 00:58, Roman Bolshakov wrote:
> > HVF doesn't have a CPU kick and without it it's not possible to perform
> > an action on CPU thread until a VMEXIT happens. The kick is also needed
> > for timely interrupt delivery.
> >
> > Existing implementation of CPU kick sends SIG_IPI (aka SIGUSR1) to vCPU
> > thread, but it's different from what hv_vcpu_interrupt does. The latter
> > one results in invocation of mp_cpus_kick() in XNU kernel [1].
> >
> > While at it, correct type of hvf_fd to the type of hv_vcpuid_t to avoid
> > compilation warnings.
> >
> > 1. https://opensource.apple.com/source/xnu/xnu-6153.81.5/osfmk/i386/mp.c
> >
> > Cc: Cameron Esfahani <di...@apple.com>
> > Signed-off-by: Roman Bolshakov <r.bolsha...@yadro.com>
> > ---
> > cpus.c | 13 +++++++++----
> > include/hw/core/cpu.h | 2 +-
> > include/sysemu/hvf.h | 1 +
> > target/i386/hvf/hvf.c | 11 +++++++++++
> > 4 files changed, 22 insertions(+), 5 deletions(-)
> >
> > diff --git a/cpus.c b/cpus.c
> > index 26709677d3..36f38ce5c8 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -1783,10 +1783,15 @@ static void qemu_cpu_kick_thread(CPUState *cpu)
> > return;
> > }
> > cpu->thread_kicked = true;
> > - err = pthread_kill(cpu->thread->thread, SIG_IPI);
> > - if (err && err != ESRCH) {
> > - fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));
> > - exit(1);
> > +
> > + if (hvf_enabled()) {
> > + hvf_vcpu_kick(cpu);
> > + } else {
> > + err = pthread_kill(cpu->thread->thread, SIG_IPI);
> > + if (err && err != ESRCH) {
> > + fprintf(stderr, "qemu:%s: %s", __func__, strerror(err));
> > + exit(1);
> > + }
> > }
> > #else /* _WIN32 */
> > if (!qemu_cpu_is_self(cpu)) {
> > diff --git a/include/hw/core/cpu.h b/include/hw/core/cpu.h
> > index b3f4b79318..288a2bd57e 100644
> > --- a/include/hw/core/cpu.h
> > +++ b/include/hw/core/cpu.h
> > @@ -438,7 +438,7 @@ struct CPUState {
> >
> > struct hax_vcpu_state *hax_vcpu;
> >
> > - int hvf_fd;
> > + unsigned hvf_fd;
> >
> > /* track IOMMUs whose translations we've cached in the TCG TLB */
> > GArray *iommu_notifiers;
> > diff --git a/include/sysemu/hvf.h b/include/sysemu/hvf.h
> > index 1d40a8ec01..aaa00cbf05 100644
> > --- a/include/sysemu/hvf.h
> > +++ b/include/sysemu/hvf.h
> > @@ -25,6 +25,7 @@ extern bool hvf_allowed;
> >
> > int hvf_init_vcpu(CPUState *);
> > int hvf_vcpu_exec(CPUState *);
> > +void hvf_vcpu_kick(CPUState *);
> > void hvf_cpu_synchronize_state(CPUState *);
> > void hvf_cpu_synchronize_post_reset(CPUState *);
> > void hvf_cpu_synchronize_post_init(CPUState *);
> > diff --git a/target/i386/hvf/hvf.c b/target/i386/hvf/hvf.c
> > index efe9802962..4d254a477a 100644
> > --- a/target/i386/hvf/hvf.c
> > +++ b/target/i386/hvf/hvf.c
> > @@ -966,6 +966,17 @@ int hvf_vcpu_exec(CPUState *cpu)
> > return ret;
> > }
> >
> > +void hvf_vcpu_kick(CPUState *cpu)
> > +{
> > + hv_return_t err;
> > +
> > + err = hv_vcpu_interrupt(&cpu->hvf_fd, 1);
> > + if (err) {
> > + fprintf(stderr, "qemu:%s error %#x\n", __func__, err);
> > + exit(1);
> > + }
> > +}
> > +
> > bool hvf_allowed;
> >
> > static int hvf_accel_init(MachineState *ms)
> >
>
> The documentation isn't clear on whether hv_vcpu_interrupt is able to
> interrupt a *subsequent* hv_vcpu_run, similar to WHPX
> WHvCancelRunVirtualProcessor (is it possible to decompile
> hv_vcpu_interrupt and see what it does?).
hv_vcpu_interrupt sends a KICK IPI using mp_cpus_kick() only if the
destination vCPU thread is running as far as I undrestand the
mp_cpus_kick():
void
mp_cpus_kick(cpumask_t cpus)
{
cpu_t cpu;
boolean_t intrs_enabled = FALSE;
intrs_enabled = ml_set_interrupts_enabled(FALSE);
mp_safe_spin_lock(&x86_topo_lock);
for (cpu = 0; cpu < (cpu_t) real_ncpus; cpu++) {
if ((cpu == (cpu_t) cpu_number())
|| ((cpu_to_cpumask(cpu) & cpus) == 0)
|| !cpu_is_running(cpu)) {
continue;
}
lapic_send_ipi(cpu, LAPIC_VECTOR(KICK));
}
simple_unlock(&x86_topo_lock);
ml_set_interrupts_enabled(intrs_enabled);
}
So, the kick is not delivered to self and in case if destination cpu is
not running. I think it can't interrupt subsequent hv_vcpu_run.
> If not, you can reduce a bit the race window by setting a variable in
> cpu, like
>
> atomic_set(&cpu->deadline, 0);
> hv_vcpu_interrupt(...)
>
> and in the vCPU thread
>
> hv_vcpu_run_until(..., atomic_read(&cpu->deadline));
> atomic_set(&cpu->deadline, HV_DEADLINE_FOREVER);
>
Sure, could you please explain who'll be racing? There's a race if a
kick was sent after VMEXIT, right? So essentially we need a way to
"requeue" a kick that was received outside of hv_vcpu_run to avoid loss
of it?
hv_vcpu_run_until is only available on macOS 10.15+ and we can't use yet
because of three release support rule.
(https://developer.apple.com/documentation/hypervisor/3181548-hv_vcpu_run_until?language=objc)
BTW, I'm totally okay to send v2 if kicks are lost and/or the patch
needs improvements. (and I can address EFER to VMCS Entry Controls
synchronization as well)
Paolo, do you know any particular test in kvm-unit-tests that can
exhibit the issue?
Thanks,
Roman
