Re: [kvm-unit-tests PATCH] x86: hyperv_synic: Hyper-V SynIC test

[Paolo Bonzini]

On 02/11/2015 13:18, Denis V. Lunev wrote:
>> I'm keeping the kernel patches queued for my own testing, but this of
>> course has to be fixed before including them---which will delay this
>> feature to 4.5, unfortunately.
> 
> well, the problem is that it actually uses auto EOI

Ok, no big deal.  We can just disable APICv when SynIC is enabled.

Paolo
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [kvm-unit-tests PATCH] x86: hyperv_synic: Hyper-V SynIC test

[Denis V. Lunev]

On 11/02/2015 03:16 PM, Paolo Bonzini wrote:

On 26/10/2015 10:56, Andrey Smetanin wrote:

Hyper-V SynIC is a Hyper-V synthetic interrupt controller.

The test runs on every vCPU and performs the following steps:
* read from all Hyper-V SynIC MSR's
* setup Hyper-V SynIC evt/msg pages
* setup SINT's routing
* inject SINT's into destination vCPU by 'hyperv-synic-test-device'
* wait for SINT's isr's completion
* clear Hyper-V SynIC evt/msg pages and destroy SINT's routing

Signed-off-by: Andrey Smetanin 
Reviewed-by: Roman Kagan 
Signed-off-by: Denis V. Lunev 
CC: Vitaly Kuznetsov 
CC: "K. Y. Srinivasan" 
CC: Gleb Natapov 
CC: Paolo Bonzini 
CC: Roman Kagan 
CC: Denis V. Lunev 
CC: qemu-de...@nongnu.org
CC: virtualization@lists.linux-foundation.org

Bad news.

The test breaks with APICv, because of the following sequence of events:

1) non-auto-EOI interrupt 176 is injected into IRR and ISR

2) The PPR register is now 176

3) auto-EOI interrupt 179 is injected into IRR only, because (179 &
0xf0) <= (PPR & 0xf0)

4) interrupt 176 ISR performs an EOI

5) at this point, because virtual interrupt delivery is enabled, the
processor does not perform TPR virtualization (SDM 29.1.2).

In addition (and even worse) because virtual interrupt delivery is
enabled, an auto-EOI interrupt that was stashed in IRR can be injected
by the processor, and the auto-EOI behavior will be skipped.

The solution is to have userspace enable KVM_CAP_HYPERV_SYNIC through
KVM_ENABLE_CAP, and modify vmx.c to not use apicv on VMs that have it
enabled.  This requires some changes to the callbacks that only work if
enable_apicv or !enable_apicv:

if (enable_apicv)
kvm_x86_ops->update_cr8_intercept = NULL;
else {
kvm_x86_ops->hwapic_irr_update = NULL;
kvm_x86_ops->hwapic_isr_update = NULL;
kvm_x86_ops->deliver_posted_interrupt = NULL;
kvm_x86_ops->sync_pir_to_irr = vmx_sync_pir_to_irr_dummy;
}

The question then is... does Hyper-V actually use auto-EOI interrupts?
If it doesn't, we might as well not implement them... :/

I'm keeping the kernel patches queued for my own testing, but this of
course has to be fixed before including them---which will delay this
feature to 4.5, unfortunately.

Paolo


well, the problem is that it actually uses auto EOI

Den
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [kvm-unit-tests PATCH] x86: hyperv_synic: Hyper-V SynIC test

[Roman Kagan]

On Mon, Nov 02, 2015 at 01:16:02PM +0100, Paolo Bonzini wrote:
> On 26/10/2015 10:56, Andrey Smetanin wrote:
> > Hyper-V SynIC is a Hyper-V synthetic interrupt controller.
> > 
> > The test runs on every vCPU and performs the following steps:
> > * read from all Hyper-V SynIC MSR's
> > * setup Hyper-V SynIC evt/msg pages
> > * setup SINT's routing
> > * inject SINT's into destination vCPU by 'hyperv-synic-test-device'
> > * wait for SINT's isr's completion
> > * clear Hyper-V SynIC evt/msg pages and destroy SINT's routing
> > 
> > Signed-off-by: Andrey Smetanin 
> > Reviewed-by: Roman Kagan 
> > Signed-off-by: Denis V. Lunev 
> > CC: Vitaly Kuznetsov 
> > CC: "K. Y. Srinivasan" 
> > CC: Gleb Natapov 
> > CC: Paolo Bonzini 
> > CC: Roman Kagan 
> > CC: Denis V. Lunev 
> > CC: qemu-de...@nongnu.org
> > CC: virtualization@lists.linux-foundation.org
> 
> Bad news.
> 
> The test breaks with APICv, because of the following sequence of events:

Thanks for testing and analyzing this!

(... running around looking for an APICv-capable machine to be able to
catch this ourselves before we resubmit ...)

> The question then is... does Hyper-V actually use auto-EOI interrupts?
> If it doesn't, we might as well not implement them... :/

As Den wrote, we've yet to see a hyperv device which doesn't :(

Roman.
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[kvm-unit-tests PATCH] x86: hyperv_synic: Hyper-V SynIC test

[Andrey Smetanin]

Hyper-V SynIC is a Hyper-V synthetic interrupt controller.

The test runs on every vCPU and performs the following steps:
* read from all Hyper-V SynIC MSR's
* setup Hyper-V SynIC evt/msg pages
* setup SINT's routing
* inject SINT's into destination vCPU by 'hyperv-synic-test-device'
* wait for SINT's isr's completion
* clear Hyper-V SynIC evt/msg pages and destroy SINT's routing

Signed-off-by: Andrey Smetanin 
Reviewed-by: Roman Kagan 
Signed-off-by: Denis V. Lunev 
CC: Vitaly Kuznetsov 
CC: "K. Y. Srinivasan" 
CC: Gleb Natapov 
CC: Paolo Bonzini 
CC: Roman Kagan 
CC: Denis V. Lunev 
CC: qemu-de...@nongnu.org
CC: virtualization@lists.linux-foundation.org
---
 config/config-x86-common.mak |   5 +-
 lib/x86/msr.h|  23 +
 x86/hyperv_synic.c   | 229 +++
 x86/run  |  10 +-
 x86/unittests.cfg|   5 +
 5 files changed, 270 insertions(+), 2 deletions(-)
 create mode 100644 x86/hyperv_synic.c

diff --git a/config/config-x86-common.mak b/config/config-x86-common.mak
index c2f9908..dc80eaa 100644
--- a/config/config-x86-common.mak
+++ b/config/config-x86-common.mak
@@ -36,7 +36,8 @@ tests-common = $(TEST_DIR)/vmexit.flat $(TEST_DIR)/tsc.flat \
$(TEST_DIR)/kvmclock_test.flat  $(TEST_DIR)/eventinj.flat \
$(TEST_DIR)/s3.flat $(TEST_DIR)/pmu.flat \
$(TEST_DIR)/tsc_adjust.flat $(TEST_DIR)/asyncpf.flat \
-   $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat
+   $(TEST_DIR)/init.flat $(TEST_DIR)/smap.flat \
+   $(TEST_DIR)/hyperv_synic.flat
 
 ifdef API
 tests-common += api/api-sample
@@ -108,6 +109,8 @@ $(TEST_DIR)/vmx.elf: $(cstart.o) $(TEST_DIR)/vmx.o 
$(TEST_DIR)/vmx_tests.o
 
 $(TEST_DIR)/debug.elf: $(cstart.o) $(TEST_DIR)/debug.o
 
+$(TEST_DIR)/hyperv_synic.elf: $(cstart.o) $(TEST_DIR)/hyperv_synic.o
+
 arch_clean:
$(RM) $(TEST_DIR)/*.o $(TEST_DIR)/*.flat $(TEST_DIR)/*.elf \
$(TEST_DIR)/.*.d lib/x86/.*.d
diff --git a/lib/x86/msr.h b/lib/x86/msr.h
index 281255a..54da420 100644
--- a/lib/x86/msr.h
+++ b/lib/x86/msr.h
@@ -408,4 +408,27 @@
 #define MSR_VM_IGNNE0xc0010115
 #define MSR_VM_HSAVE_PA 0xc0010117
 
+/* Define synthetic interrupt controller model specific registers. */
+#define HV_X64_MSR_SCONTROL 0x4080
+#define HV_X64_MSR_SVERSION 0x4081
+#define HV_X64_MSR_SIEFP0x4082
+#define HV_X64_MSR_SIMP 0x4083
+#define HV_X64_MSR_EOM  0x4084
+#define HV_X64_MSR_SINT00x4090
+#define HV_X64_MSR_SINT10x4091
+#define HV_X64_MSR_SINT20x4092
+#define HV_X64_MSR_SINT30x4093
+#define HV_X64_MSR_SINT40x4094
+#define HV_X64_MSR_SINT50x4095
+#define HV_X64_MSR_SINT60x4096
+#define HV_X64_MSR_SINT70x4097
+#define HV_X64_MSR_SINT80x4098
+#define HV_X64_MSR_SINT90x4099
+#define HV_X64_MSR_SINT10   0x409A
+#define HV_X64_MSR_SINT11   0x409B
+#define HV_X64_MSR_SINT12   0x409C
+#define HV_X64_MSR_SINT13   0x409D
+#define HV_X64_MSR_SINT14   0x409E
+#define HV_X64_MSR_SINT15   0x409F
+
 #endif /* _ASM_X86_MSR_INDEX_H */
diff --git a/x86/hyperv_synic.c b/x86/hyperv_synic.c
new file mode 100644
index 000..5c5a43a
--- /dev/null
+++ b/x86/hyperv_synic.c
@@ -0,0 +1,229 @@
+#include "libcflat.h"
+#include "processor.h"
+#include "msr.h"
+#include "isr.h"
+#include "vm.h"
+#include "apic.h"
+#include "desc.h"
+#include "io.h"
+#include "smp.h"
+#include "atomic.h"
+
+#define MAX_CPUS 4
+#define HYPERV_CPUID_FEATURES   0x4003
+#define HV_X64_MSR_SYNIC_AVAILABLE  (1 << 2)
+#define HV_SYNIC_CONTROL_ENABLE (1ULL << 0)
+#define HV_SYNIC_SIMP_ENABLE(1ULL << 0)
+#define HV_SYNIC_SIEFP_ENABLE   (1ULL << 0)
+#define HV_SYNIC_SINT_MASKED(1ULL << 16)
+#define HV_SYNIC_SINT_AUTO_EOI  (1ULL << 17)
+#define HV_SYNIC_SINT_VECTOR_MASK   (0xFF)
+#define HV_SYNIC_SINT_COUNT 16
+
+enum {
+HV_TEST_DEV_SINT_ROUTE_CREATE = 1,
+HV_TEST_DEV_SINT_ROUTE_DESTROY,
+HV_TEST_DEV_SINT_ROUTE_SET_SINT
+};
+
+static atomic_t isr_enter_count[MAX_CPUS];
+static atomic_t cpus_comp_count;
+
+static bool synic_supported(void)
+{
+   return