[PATCH v2] x86/entry: move .init.text section higher up in the code for readability

[Jane Malalane]

.init.text is a small section currently located amongst .text.entry
code. Move it above .text.entry.

This has no functional change but makes the code a bit more readable.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Reviewed-by: Jan Beulich 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 

v2:
 * code style
 * replace incorrect .popsection with .section
 * add commit message
---
 xen/arch/x86/x86_64/entry.S | 19 +--
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S
index 26bf2f1941..9b34150bc7 100644
--- a/xen/arch/x86/x86_64/entry.S
+++ b/xen/arch/x86/x86_64/entry.S
@@ -140,6 +140,15 @@ process_trap:
 call create_bounce_frame
 jmp  test_all_events
 
+.section .init.text, "ax", @progbits
+ENTRY(early_page_fault)
+ENDBR64
+movl  $TRAP_page_fault, 4(%rsp)
+SAVE_ALL
+movq  %rsp, %rdi
+call  do_early_page_fault
+jmp   restore_all_xen
+
 .section .text.entry, "ax", @progbits
 
 /* %rbx: struct vcpu, interrupts disabled */
@@ -982,16 +991,6 @@ ENTRY(double_fault)
 call  do_double_fault
 BUG   /* do_double_fault() shouldn't return. */
 
-.pushsection .init.text, "ax", @progbits
-ENTRY(early_page_fault)
-ENDBR64
-movl  $TRAP_page_fault,4(%rsp)
-SAVE_ALL
-movq  %rsp,%rdi
-call  do_early_page_fault
-jmp   restore_all_xen
-.popsection
-
 ENTRY(nmi)
 ENDBR64
 pushq $0
-- 
2.11.0

Re: [PATCH 2/4] xen: Port linkage.h from kernel code

[Jane Malalane]

On 16/08/2022 14:06, Jan Beulich wrote:
> On 16.08.2022 12:16, Jane Malalane wrote:
>> On 05/08/2022 10:24, Jan Beulich wrote:
>>> On 04.08.2022 17:04, Jane Malalane wrote:
>>>> Suggested-by: Andrew Cooper 
>>>> Signed-off-by: Jane Malalane 
>>>
>>> In the title you say "port", but then you don't say what customization
>>> you've done beyond the obvious adjustment of inclusion guard and
>>> adjustment (actually removal) of #include-s. How much customization we
>>> want to do is important here, after all. I notice you did, for example,
>>> add new flavors of SYM_INNER_LABEL, but then you didn't add anything to
>>> use .hidden (which I have queued as a new patch on top of a supposed v2
>>> of "x86: annotate entry points with type and size"). At the same time
>>> you've left in objtool related commentary, when we don't use that tool
>>> (and no-one knows if we ever will).
>>>
>>> I'm further not sure I agree with the naming of some of your additions,
>>> as they appear to not really fit with the naming model used elsewhere.
>>> Your additions also look to not always match style used elsewhere in
>>> this file.
>>>
>>> You further want to mention what Linux version this was derived from,
>>> to make it easier to determine what incremental changes to port over
>>> subsequently.
>>>
>>> Overall I'm not convinced this is a model we want to go with, first
>>> and foremost because the commonly used macro names are too long for
>>> my taste. What's wrong with ENTRY() and ENDPROC() / ENDDATA()?
>> Hi Jan,
>>
>> Since I have no particular opinion on this I went through the discussion
>> that took place before those macros were introduced in Linux. One of the
>> points made was that PROC was an ambiguous term to use, since C
>> functions are not actually procedures, at least not all.
> 
> Just one remark here: We're talking about assembly code here, so what's
> a procedure or function isn't well defined anyway. I wouldn't, btw, mind
> ENDFUNC() if that's deemed better than ENDPROC().
Do you then propose that we use ENTRY() and ENDFUNC() and that for inner 
labels we keep them as is (so just "name:"), since using ENTRY() without 
a closing END() for some "functions" and not for others could get a bit 
confusing?

Jane.

Re: [PATCH 2/4] xen: Port linkage.h from kernel code

[Jane Malalane]

On 05/08/2022 10:24, Jan Beulich wrote:
> On 04.08.2022 17:04, Jane Malalane wrote:
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
> 
> In the title you say "port", but then you don't say what customization
> you've done beyond the obvious adjustment of inclusion guard and
> adjustment (actually removal) of #include-s. How much customization we
> want to do is important here, after all. I notice you did, for example,
> add new flavors of SYM_INNER_LABEL, but then you didn't add anything to
> use .hidden (which I have queued as a new patch on top of a supposed v2
> of "x86: annotate entry points with type and size"). At the same time
> you've left in objtool related commentary, when we don't use that tool
> (and no-one knows if we ever will).
> 
> I'm further not sure I agree with the naming of some of your additions,
> as they appear to not really fit with the naming model used elsewhere.
> Your additions also look to not always match style used elsewhere in
> this file.
> 
> You further want to mention what Linux version this was derived from,
> to make it easier to determine what incremental changes to port over
> subsequently.
> 
> Overall I'm not convinced this is a model we want to go with, first
> and foremost because the commonly used macro names are too long for
> my taste. What's wrong with ENTRY() and ENDPROC() / ENDDATA()?
Hi Jan,

Since I have no particular opinion on this I went through the discussion 
that took place before those macros were introduced in Linux. One of the 
points made was that PROC was an ambiguous term to use, since C 
functions are not actually procedures, at least not all. The other was 
that using START/END markers make it easier for a developer to remember 
to add the END marker, and eventhough you suggest using ENTRY and not 
just PROC as the start marker, START might still be clearer.

I welcome other input.

Thank you for your review,

Jane.

[PATCH 2/4] xen: Port linkage.h from kernel code

[Jane Malalane]

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Wei Liu 
---
 xen/include/xen/linkage.h | 260 ++
 1 file changed, 260 insertions(+)
 create mode 100644 xen/include/xen/linkage.h

diff --git a/xen/include/xen/linkage.h b/xen/include/xen/linkage.h
new file mode 100644
index 00..adc00c356b
--- /dev/null
+++ b/xen/include/xen/linkage.h
@@ -0,0 +1,260 @@
+#ifndef __XEN_LINKAGE_H
+#define __XEN_LINKAGE_H
+
+/*
+ * Imported from linux-5.19:include/linux/linkage.h
+ */
+
+/* Some toolchains use other characters (e.g. '`') to mark new line in macro */
+#ifndef ASM_NL
+#define ASM_NL  ;
+#endif
+
+#ifdef __ASSEMBLY__
+
+/* SYM_T_FUNC -- type used by assembler to mark functions */
+#ifndef SYM_T_FUNC
+#define SYM_T_FUNC STT_FUNC
+#endif
+
+/* SYM_T_OBJECT -- type used by assembler to mark data */
+#ifndef SYM_T_OBJECT
+#define SYM_T_OBJECT   STT_OBJECT
+#endif
+
+/* SYM_T_NONE -- type used by assembler to mark entries of unknown type */
+#ifndef SYM_T_NONE
+#define SYM_T_NONE STT_NOTYPE
+#endif
+
+/* SYM_A_* -- align the symbol? */
+#define SYM_A_ALIGNALIGN
+#define SYM_A_NONE /* nothing */
+
+/* SYM_L_* -- linkage of symbols */
+#define SYM_L_GLOBAL(name) .globl name
+#define SYM_L_WEAK(name)   .weak name
+#define SYM_L_LOCAL(name)  /* nothing */
+
+/* === generic annotations === */
+
+/* SYM_ENTRY -- use only if you have to for non-paired symbols */
+#ifndef SYM_ENTRY
+#define SYM_ENTRY(name, linkage, align...) \
+   linkage(name) ASM_NL\
+   align ASM_NL\
+   name:
+#endif
+
+/* SYM_START -- use only if you have to */
+#ifndef SYM_START
+#define SYM_START(name, linkage, align...) \
+   SYM_ENTRY(name, linkage, align)
+#endif
+
+/* SYM_END -- use only if you have to */
+#ifndef SYM_END
+#define SYM_END(name, sym_type)\
+   .type name sym_type ASM_NL  \
+   .set .L__sym_size_##name, .-name ASM_NL \
+   .size name, .L__sym_size_##name
+#endif
+
+/* SYM_ALIAS -- use only if you have to */
+#ifndef SYM_ALIAS
+#define SYM_ALIAS(alias, name, linkage)\
+   linkage(alias) ASM_NL   \
+   .set alias, name ASM_NL
+#endif
+
+/* === code annotations === */
+
+/*
+ * FUNC -- C-like functions (proper stack frame etc.)
+ * CODE -- non-C code (e.g. irq handlers with different, special stack etc.)
+ *
+ * Objtool validates stack for FUNC, but not for CODE.
+ * Objtool generates debug info for both FUNC & CODE, but needs special
+ * annotations for each CODE's start (to describe the actual stack frame).
+ *
+ * Objtool requires that all code must be contained in an ELF symbol. Symbol
+ * names that have a  .L prefix do not emit symbol table entries. .L
+ * prefixed symbols can be used within a code region, but should be avoided for
+ * denoting a range of code via ``SYM_*_START/END`` annotations.
+ *
+ * ALIAS -- does not generate debug info -- the aliased function will
+ */
+
+/* SYM_INNER_LABEL_ALIGN -- only for labels in the middle of code,
+ * w/ alignment
+ */
+#ifndef SYM_INNER_LABEL_ALIGN
+#define SYM_INNER_LABEL_ALIGN(name, linkage)   \
+   .type name SYM_T_NONE ASM_NL\
+   SYM_ENTRY(name, linkage, SYM_A_ALIGN)
+#endif
+
+/* SYM_INNER_LABEL_LOCAL -- only for local labels in the middle of code */
+#ifndef SYM_INNER_LABEL_LOCAL
+#define SYM_INNER_LABEL_LOCAL(name)\
+   .type name SYM_T_NONE ASM_NL\
+   SYM_ENTRY(name, SYM_L_LOCAL, SYM_A_NONE)
+#endif
+
+/* SYM_INNER_LABEL_GLOBAL -- only for global labels in the middle of code */
+#ifndef SYM_INNER_LABEL_GLOBAL
+#define SYM_INNER_LABEL_GLOBAL(name)   \
+.type name SYM_T_NONE ASM_NL\
+   SYM_ENTRY(name, SYM_L_GLOBAL, SYM_A_NONE)
+#endif
+
+/* SYM_FUNC_START -- use for global functions */
+#ifndef SYM_FUNC_START
+#define SYM_FUNC_START(name)   \
+   SYM_START(name, SYM_L_GLOBAL, SYM_A_ALIGN)
+#endif
+
+/* SYM_FUNC_START_NOALIGN -- use for global functions, w/o alignment */
+#ifndef SYM_FUNC_START_NOALIGN
+#define SYM_FUNC_START_NOALIGN(name)   \
+   SYM_START(name, SYM_L_GLOBAL, SYM_A_NONE)
+#endif
+
+/* SYM_FUNC_START_LOCAL -- use for local functions */
+#ifndef SYM_FUNC_START_LOCAL
+#define SYM_FUNC_START_LOCAL(name) \
+   SYM_START(name, SYM_L_LOCAL, SYM_A_ALIGN)
+#endif
+
+/* SYM_FUNC_START_LOCAL_NOALIGN -- use for local functions, w/o alignment */
+#if

[PATCH 1/4] x86/kexec: Add the '.L_' prefix to is_* and call_* labels

[Jane Malalane]

These are local symbols and shouldn't be externally visible.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Andrew Cooper 
CC: Jan Beulich 
CC: "Roger Pau Monné" 
CC: Wei Liu 
---
 xen/arch/x86/x86_64/kexec_reloc.S | 42 +++
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/xen/arch/x86/x86_64/kexec_reloc.S 
b/xen/arch/x86/x86_64/kexec_reloc.S
index 89316bc3a7..f4842025eb 100644
--- a/xen/arch/x86/x86_64/kexec_reloc.S
+++ b/xen/arch/x86/x86_64/kexec_reloc.S
@@ -40,10 +40,10 @@ ENTRY(kexec_reloc)
 movq%rsi, %cr3
 
 /* Jump to identity mapped code. */
-leaq(identity_mapped - kexec_reloc)(%rdi), %rax
+leaq(.L_identity_mapped - kexec_reloc)(%rdi), %rax
 jmpq*%rax
 
-identity_mapped:
+.L_identity_mapped:
 /*
  * Set cr0 to a known state:
  *  - Paging enabled
@@ -70,14 +70,14 @@ identity_mapped:
 
 /* Need to switch to 32-bit mode? */
 testq   $KEXEC_RELOC_FLAG_COMPAT, %r8
-jnz call_32_bit
+jnz .L_call_32_bit
 
-call_64_bit:
+.L_call_64_bit:
 /* Call the image entry point.  This should never return. */
 callq   *%rbp
 ud2
 
-call_32_bit:
+.L_call_32_bit:
 /* Setup IDT. */
 lidtcompat_mode_idt(%rip)
 
@@ -102,41 +102,41 @@ relocate_pages:
 xorl%edi, %edi
 xorl%esi, %esi
 
-next_entry: /* top, read another word for the indirection page */
+.L_next_entry: /* top, read another word for the indirection page */
 
 movq(%rbx), %rcx
 addq$8, %rbx
-is_dest:
+.L_is_dest:
 testb   $IND_DESTINATION, %cl
-jz  is_ind
+jz  .L_is_ind
 movq%rcx, %rdi
 andq$PAGE_MASK, %rdi
-jmp next_entry
-is_ind:
+jmp .L_next_entry
+.L_is_ind:
 testb   $IND_INDIRECTION, %cl
-jz  is_done
+jz  .L_is_done
 movq%rcx, %rbx
 andq$PAGE_MASK, %rbx
-jmp next_entry
-is_done:
+jmp .L_next_entry
+.L_is_done:
 testb   $IND_DONE, %cl
-jnz done
-is_source:
+jnz .L_done
+.L_is_source:
 testb   $IND_SOURCE, %cl
-jz  is_zero
+jz  .L_is_zero
 movq%rcx, %rsi  /* For every source page do a copy */
 andq$PAGE_MASK, %rsi
 movl$(PAGE_SIZE / 8), %ecx
 rep movsq
-jmp next_entry
-is_zero:
+jmp .L_next_entry
+.L_is_zero:
 testb   $IND_ZERO, %cl
-jz  next_entry
+jz  .L_next_entry
 movl$(PAGE_SIZE / 8), %ecx  /* Zero the destination page. */
 xorl%eax, %eax
 rep stosq
-jmp next_entry
-done:
+jmp .L_next_entry
+.L_done:
 popq%rbx
 ret
 
-- 
2.11.0

[PATCH 0/4] xen/x86: import linkage.h and clean up x86/kexec.S and x86/entry.S

[Jane Malalane]

Jane Malalane (4):
  x86/kexec: Add the '.L_' prefix to is_* and call_* labels
  xen: Port linkage.h from kernel code
  x86/entry: move .init.text section higher up in the code for
readability
  x86: Use linkage.h helpers to add tags to symbols

 xen/arch/x86/x86_64/entry.S   | 124 +++---
 xen/arch/x86/x86_64/kexec_reloc.S |  85 ++---
 xen/include/xen/linkage.h | 260 ++
 3 files changed, 376 insertions(+), 93 deletions(-)
 create mode 100644 xen/include/xen/linkage.h

-- 
2.11.0

[PATCH 3/4] x86/entry: move .init.text section higher up in the code for readability

[Jane Malalane]

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 
---
 xen/arch/x86/x86_64/entry.S | 19 +--
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S
index 26bf2f1941..4ad25d9c90 100644
--- a/xen/arch/x86/x86_64/entry.S
+++ b/xen/arch/x86/x86_64/entry.S
@@ -140,6 +140,15 @@ process_trap:
 call create_bounce_frame
 jmp  test_all_events
 
+.pushsection .init.text, "ax", @progbits
+ENTRY(early_page_fault)
+ENDBR64
+movl  $TRAP_page_fault,4(%rsp)
+SAVE_ALL
+movq  %rsp,%rdi
+call  do_early_page_fault
+jmp   restore_all_xen
+
 .section .text.entry, "ax", @progbits
 
 /* %rbx: struct vcpu, interrupts disabled */
@@ -982,16 +991,6 @@ ENTRY(double_fault)
 call  do_double_fault
 BUG   /* do_double_fault() shouldn't return. */
 
-.pushsection .init.text, "ax", @progbits
-ENTRY(early_page_fault)
-ENDBR64
-movl  $TRAP_page_fault,4(%rsp)
-SAVE_ALL
-movq  %rsp,%rdi
-call  do_early_page_fault
-jmp   restore_all_xen
-.popsection
-
 ENTRY(nmi)
 ENDBR64
 pushq $0
-- 
2.11.0

[RFC PATCH 4/4] x86: Use linkage.h helpers to add tags to symbols

[Jane Malalane]

Clean up x86_64/kexec_reloc.S and x86_64/entry.S.

This fixes the livepatching contents of entry.S.

RFC: I'm unsure on where the page_fault symbol should end, i.e. if
unlike current code handle_exception_saved should be within page_fault
like handle_exception is or not.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 
---
 xen/arch/x86/x86_64/entry.S   | 105 +-
 xen/arch/x86/x86_64/kexec_reloc.S |  43 ++--
 2 files changed, 86 insertions(+), 62 deletions(-)

diff --git a/xen/arch/x86/x86_64/entry.S b/xen/arch/x86/x86_64/entry.S
index 4ad25d9c90..7dc280aafa 100644
--- a/xen/arch/x86/x86_64/entry.S
+++ b/xen/arch/x86/x86_64/entry.S
@@ -11,6 +11,7 @@
 #include 
 #include 
 #include 
+#include 
 
 /* %rsp: struct cpu_user_regs */
 .macro ASSERT_CONTEXT_IS_XEN
@@ -152,7 +153,7 @@ process_trap:
 .section .text.entry, "ax", @progbits
 
 /* %rbx: struct vcpu, interrupts disabled */
-restore_all_guest:
+SYM_CODE_START_LOCAL(restore_all_guest)
 ASSERT_INTERRUPTS_DISABLED
 
 /* Stash guest SPEC_CTRL value while we can read struct vcpu. */
@@ -239,6 +240,7 @@ iret_exit_to_guest:
 addq  $8,%rsp
 .Lft0:  iretq
 _ASM_PRE_EXTABLE(.Lft0, handle_exception)
+SYM_CODE_END(restore_all_guest)
 
 /*
  * When entering SYSCALL from kernel mode:
@@ -255,7 +257,7 @@ iret_exit_to_guest:
  *  - Guest %rsp stored in %rax
  *  - Xen stack loaded, pointing at the %ss slot
  */
-ENTRY(lstar_enter)
+SYM_CODE_START(lstar_enter)
 #ifdef CONFIG_XEN_SHSTK
 ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK
 #endif
@@ -290,9 +292,10 @@ ENTRY(lstar_enter)
 mov   %rsp, %rdi
 call  pv_hypercall
 jmp   test_all_events
+SYM_CODE_END(lstar_enter)
 
 /* See lstar_enter for entry register state. */
-ENTRY(cstar_enter)
+SYM_CODE_START(cstar_enter)
 #ifdef CONFIG_XEN_SHSTK
 ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK
 #endif
@@ -329,8 +332,9 @@ ENTRY(cstar_enter)
 jne   compat_syscall
 #endif
 jmp   switch_to_kernel
+SYM_CODE_END(cstar_enter)
 
-ENTRY(sysenter_entry)
+SYM_CODE_START(sysenter_entry)
 ENDBR64
 #ifdef CONFIG_XEN_SHSTK
 ALTERNATIVE "", "setssbsy", X86_FEATURE_XEN_SHSTK
@@ -339,7 +343,7 @@ ENTRY(sysenter_entry)
 pushq $FLAT_USER_SS
 pushq $0
 pushfq
-GLOBAL(sysenter_eflags_saved)
+SYM_INNER_LABEL_GLOBAL(sysenter_eflags_saved)
 ALTERNATIVE "", clac, X86_FEATURE_XEN_SMAP
 pushq $3 /* ring 3 null cs */
 pushq $0 /* null rip */
@@ -393,8 +397,9 @@ UNLIKELY_END(sysenter_gpf)
 jne   compat_sysenter
 #endif
 jmp   .Lbounce_exception
+SYM_CODE_END(sysenter_entry)
 
-ENTRY(int80_direct_trap)
+SYM_CODE_START(int80_direct_trap)
 ENDBR64
 ALTERNATIVE "", clac, X86_FEATURE_XEN_SMAP
 pushq $0
@@ -465,8 +470,9 @@ UNLIKELY_END(msi_check)
 
 call  create_bounce_frame
 jmp   test_all_events
+SYM_CODE_END(int80_direct_trap)
 
-int80_slow_path:
+SYM_CODE_START_LOCAL(int80_slow_path)
 /* 
  * Setup entry vector and error code as if this was a GPF caused by an
  * IDT entry with DPL==0.
@@ -482,6 +488,7 @@ int80_slow_path:
  */
 GET_STACK_END(14)
 jmp   handle_exception_saved
+SYM_CODE_END(int80_slow_path)
 
 /* create_bounce_frame & helpers don't need to be in .text.entry */
 .text
@@ -657,9 +664,8 @@ ret_from_intr:
 
 .section .text.entry, "ax", @progbits
 
-ALIGN
 /* No special register assumptions. */
-restore_all_xen:
+SYM_CODE_START_LOCAL(restore_all_xen)
 /*
  * Check whether we need to switch to the per-CPU page tables, in
  * case we return to late PV exit code (from an NMI or #MC).
@@ -676,8 +682,9 @@ UNLIKELY_END(exit_cr3)
 
 RESTORE_ALL adj=8
 iretq
+SYM_CODE_END(restore_all_xen)
 
-ENTRY(common_interrupt)
+SYM_CODE_START(common_interrupt)
 ALTERNATIVE "", clac, X86_FEATURE_XEN_SMAP
 SAVE_ALL
 
@@ -706,12 +713,14 @@ ENTRY(common_interrupt)
 mov   %r15, STACK_CPUINFO_FIELD(xen_cr3)(%r14)
 mov   %bl, STACK_CPUINFO_FIELD(use_pv_cr3)(%r14)
 jmp ret_from_intr
+SYM_CODE_END(common_interrupt)
 
-ENTRY(page_fault)
+SYM_CODE_START(page_fault)
 ENDBR64
 movl  $TRAP_page_fault,4(%rsp)
+
 /* No special register assumptions. */
-GLOBAL(handle_exception)
+SYM_INNER_LABEL_GLOBAL(handle_exception)
 ALTERNATIVE "", clac, X86_FEATURE_XEN_SMAP
 SAVE_ALL
 
@@ -734,7 +743,7 @@ GLOBAL(handle_exception)
 cmovnz %r12d, %r13d
 .Lxcpt_cr3_okay:
 
-handle_exception_saved:
+SYM_INNER_LABEL_LOCAL(handle_exception_saved)
 GET_CURRENT(bx)
 testb $X86_EFLAGS_IF>>8,UREGS_eflags+

[PATCH v4] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

Implement support for the HVMOP_set_evtchn_upcall_vector hypercall in
order to set the per-vCPU event channel vector callback on Linux and
use it in preference of HVM_PARAM_CALLBACK_IRQ.

If the per-VCPU vector setup is successful on BSP, use this method
for the APs. If not, fallback to the global vector-type callback.

Also register callback_irq at per-vCPU event channel setup to trick
toolstack to think the domain is enlightened.

Suggested-by: "Roger Pau Monné" 
Signed-off-by: Jane Malalane 
Reviewed-by: Boris Ostrovsky 
---
CC: Juergen Gross 
CC: Boris Ostrovsky 
CC: Thomas Gleixner 
CC: Ingo Molnar 
CC: Borislav Petkov 
CC: Dave Hansen 
CC: x...@kernel.org
CC: "H. Peter Anvin" 
CC: Stefano Stabellini 
CC: Oleksandr Tyshchenko 
CC: Jan Beulich 
CC: Maximilian Heyne 
CC: xen-devel@lists.xenproject.org

v4:
 * amend code comment

v3:
 * comment style
 * add comment on toolstack trick
 * remove unnecessary variable and function call
 * surround x86-specific code with #ifdef

v2:
 * remove no_vector_callback
 * make xen_have_vector_callback a bool
 * rename xen_ack_upcall to xen_percpu_upcall
 * fail to bring CPU up on init instead of crashing kernel
 * add and use xen_set_upcall_vector where suitable
 * xen_setup_upcall_vector -> xen_init_setup_upcall_vector for clarity
---
 arch/x86/include/asm/xen/cpuid.h   |  2 ++
 arch/x86/include/asm/xen/events.h  |  3 ++-
 arch/x86/xen/enlighten.c   |  2 +-
 arch/x86/xen/enlighten_hvm.c   | 24 -
 arch/x86/xen/suspend_hvm.c | 10 ++-
 drivers/xen/events/events_base.c   | 53 +-
 include/xen/hvm.h  |  2 ++
 include/xen/interface/hvm/hvm_op.h | 19 ++
 8 files changed, 100 insertions(+), 15 deletions(-)

diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
index 78e667a31d6c..6daa9b0c8d11 100644
--- a/arch/x86/include/asm/xen/cpuid.h
+++ b/arch/x86/include/asm/xen/cpuid.h
@@ -107,6 +107,8 @@
  * ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/* Per-vCPU event channel upcalls */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
diff --git a/arch/x86/include/asm/xen/events.h 
b/arch/x86/include/asm/xen/events.h
index 068d9b067c83..62bdceb594f1 100644
--- a/arch/x86/include/asm/xen/events.h
+++ b/arch/x86/include/asm/xen/events.h
@@ -23,7 +23,7 @@ static inline int xen_irqs_disabled(struct pt_regs *regs)
 /* No need for a barrier -- XCHG is a barrier on x86. */
 #define xchg_xen_ulong(ptr, val) xchg((ptr), (val))
 
-extern int xen_have_vector_callback;
+extern bool xen_have_vector_callback;
 
 /*
  * Events delivered via platform PCI interrupts are always
@@ -34,4 +34,5 @@ static inline bool xen_support_evtchn_rebind(void)
return (!xen_hvm_domain() || xen_have_vector_callback);
 }
 
+extern bool xen_percpu_upcall;
 #endif /* _ASM_X86_XEN_EVENTS_H */
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 30c6e986a6cd..b8db2148c07d 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -51,7 +51,7 @@ EXPORT_SYMBOL_GPL(xen_start_info);
 
 struct shared_info xen_dummy_shared_info;
 
-__read_mostly int xen_have_vector_callback;
+__read_mostly bool xen_have_vector_callback = true;
 EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 
 /*
diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index 8b71b1dd7639..198d3cd3e9a5 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -7,6 +7,8 @@
 
 #include 
 #include 
+#include 
+#include 
 #include 
 
 #include 
@@ -30,6 +32,9 @@
 
 static unsigned long shared_info_pfn;
 
+__ro_after_init bool xen_percpu_upcall;
+EXPORT_SYMBOL_GPL(xen_percpu_upcall);
+
 void xen_hvm_init_shared_info(void)
 {
struct xen_add_to_physmap xatp;
@@ -125,6 +130,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
 {
struct pt_regs *old_regs = set_irq_regs(regs);
 
+   if (xen_percpu_upcall)
+   ack_APIC_irq();
+
inc_irq_stat(irq_hv_callback_count);
 
xen_hvm_evtchn_do_upcall();
@@ -168,6 +176,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
if (!xen_have_vector_callback)
return 0;
 
+   if (xen_percpu_upcall) {
+   rc = xen_set_upcall_vector(cpu);
+   if (rc) {
+   WARN(1, "HVMOP_set_evtchn_upcall_vector"
+" for CPU %d failed: %d\n", cpu, rc);
+   return rc;
+   }
+   }
+
if (xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_timer(cpu);
 
@@ -188,8 +205,6 @@ static int xen_cpu_dead_hvm(unsigned int cpu)
return 0;
 }
 
-static bool no_vector_callback __initdata;
-
 static void __init xen_hvm_guest_init(void)
 {
if (xen_pv_domain())
@@ -211

[PATCH v3] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

Implement support for the HVMOP_set_evtchn_upcall_vector hypercall in
order to set the per-vCPU event channel vector callback on Linux and
use it in preference of HVM_PARAM_CALLBACK_IRQ.

If the per-VCPU vector setup is successful on BSP, use this method
for the APs. If not, fallback to the global vector-type callback.

Also register callback_irq at per-vCPU event channel setup to trick
toolstack to think the domain is enlightened.

Suggested-by: "Roger Pau Monné" 
Signed-off-by: Jane Malalane 
---
CC: Juergen Gross 
CC: Boris Ostrovsky 
CC: Thomas Gleixner 
CC: Ingo Molnar 
CC: Borislav Petkov 
CC: Dave Hansen 
CC: x...@kernel.org
CC: "H. Peter Anvin" 
CC: Stefano Stabellini 
CC: Oleksandr Tyshchenko 
CC: Jan Beulich 
CC: Maximilian Heyne 
CC: xen-devel@lists.xenproject.org

v3:
 * comment style
 * add comment on toolstack trick
 * remove unnecessary variable and function call
 * surround x86-specific code with #ifdef

v2:
 * remove no_vector_callback
 * make xen_have_vector_callback a bool
 * rename xen_ack_upcall to xen_percpu_upcall
 * fail to bring CPU up on init instead of crashing kernel
 * add and use xen_set_upcall_vector where suitable
 * xen_setup_upcall_vector -> xen_init_setup_upcall_vector for clarity
---
 arch/x86/include/asm/xen/cpuid.h   |  2 ++
 arch/x86/include/asm/xen/events.h  |  3 ++-
 arch/x86/xen/enlighten.c   |  2 +-
 arch/x86/xen/enlighten_hvm.c   | 24 -
 arch/x86/xen/suspend_hvm.c | 10 ++-
 drivers/xen/events/events_base.c   | 54 +-
 include/xen/hvm.h  |  2 ++
 include/xen/interface/hvm/hvm_op.h | 19 ++
 8 files changed, 101 insertions(+), 15 deletions(-)

diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
index 78e667a31d6c..6daa9b0c8d11 100644
--- a/arch/x86/include/asm/xen/cpuid.h
+++ b/arch/x86/include/asm/xen/cpuid.h
@@ -107,6 +107,8 @@
  * ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/* Per-vCPU event channel upcalls */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
diff --git a/arch/x86/include/asm/xen/events.h 
b/arch/x86/include/asm/xen/events.h
index 068d9b067c83..62bdceb594f1 100644
--- a/arch/x86/include/asm/xen/events.h
+++ b/arch/x86/include/asm/xen/events.h
@@ -23,7 +23,7 @@ static inline int xen_irqs_disabled(struct pt_regs *regs)
 /* No need for a barrier -- XCHG is a barrier on x86. */
 #define xchg_xen_ulong(ptr, val) xchg((ptr), (val))
 
-extern int xen_have_vector_callback;
+extern bool xen_have_vector_callback;
 
 /*
  * Events delivered via platform PCI interrupts are always
@@ -34,4 +34,5 @@ static inline bool xen_support_evtchn_rebind(void)
return (!xen_hvm_domain() || xen_have_vector_callback);
 }
 
+extern bool xen_percpu_upcall;
 #endif /* _ASM_X86_XEN_EVENTS_H */
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 30c6e986a6cd..b8db2148c07d 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -51,7 +51,7 @@ EXPORT_SYMBOL_GPL(xen_start_info);
 
 struct shared_info xen_dummy_shared_info;
 
-__read_mostly int xen_have_vector_callback;
+__read_mostly bool xen_have_vector_callback = true;
 EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 
 /*
diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index 8b71b1dd7639..198d3cd3e9a5 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -7,6 +7,8 @@
 
 #include 
 #include 
+#include 
+#include 
 #include 
 
 #include 
@@ -30,6 +32,9 @@
 
 static unsigned long shared_info_pfn;
 
+__ro_after_init bool xen_percpu_upcall;
+EXPORT_SYMBOL_GPL(xen_percpu_upcall);
+
 void xen_hvm_init_shared_info(void)
 {
struct xen_add_to_physmap xatp;
@@ -125,6 +130,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
 {
struct pt_regs *old_regs = set_irq_regs(regs);
 
+   if (xen_percpu_upcall)
+   ack_APIC_irq();
+
inc_irq_stat(irq_hv_callback_count);
 
xen_hvm_evtchn_do_upcall();
@@ -168,6 +176,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
if (!xen_have_vector_callback)
return 0;
 
+   if (xen_percpu_upcall) {
+   rc = xen_set_upcall_vector(cpu);
+   if (rc) {
+   WARN(1, "HVMOP_set_evtchn_upcall_vector"
+" for CPU %d failed: %d\n", cpu, rc);
+   return rc;
+   }
+   }
+
if (xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_timer(cpu);
 
@@ -188,8 +205,6 @@ static int xen_cpu_dead_hvm(unsigned int cpu)
return 0;
 }
 
-static bool no_vector_callback __initdata;
-
 static void __init xen_hvm_guest_init(void)
 {
if (xen_pv_domain())
@@ -211,9 +226,6 @@ static void __init xen_hvm_guest_init(void)
 
xen_

Re: [PATCH v2] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 27/07/2022 13:32, Julien Grall wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
> attachments unless you have verified the sender and know the content is 
> safe.
> 
> Hi Jane,
> 
> On 26/07/2022 13:56, Jane Malalane wrote:
>> diff --git a/arch/x86/xen/suspend_hvm.c b/arch/x86/xen/suspend_hvm.c
>> index 9d548b0c772f..0c4f7554b7cc 100644
>> --- a/arch/x86/xen/suspend_hvm.c
>> +++ b/arch/x86/xen/suspend_hvm.c
>> @@ -5,6 +5,7 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>>   #include "xen-ops.h"
>> @@ -14,6 +15,13 @@ void xen_hvm_post_suspend(int suspend_cancelled)
>>   xen_hvm_init_shared_info();
>>   xen_vcpu_restore();
>>   }
>> -    xen_setup_callback_vector();
>> +    if (xen_percpu_upcall) {
>> +    unsigned int cpu;
>> +
>> +    for_each_online_cpu(cpu)
>> +    BUG_ON(xen_set_upcall_vector(cpu));
>> +    } else {
>> +    xen_setup_callback_vector();
>> +    }
>>   xen_unplug_emulated_devices();
>>   }
>> diff --git a/drivers/xen/events/events_base.c 
>> b/drivers/xen/events/events_base.c
>> index 46d9295d9a6e..2ad93595d03a 100644
>> --- a/drivers/xen/events/events_base.c
>> +++ b/drivers/xen/events/events_base.c
>> @@ -48,6 +48,7 @@
>>   #include 
>>   #endif
>>   #include 
>> +#include 
> 
> This include doesn't exist on Arm and will result to a build failure:
> 
> linux/drivers/xen/events/events_base.c:51:10: fatal error: 
> asm/xen/cpuid.h: No such file or directory
>     51 | #include 
>    |  ^
Thanks, will place it inside the #ifdef CONFIG_X86.
> 
>>   #include 
>>   #include 
>>   #include 
>> @@ -2195,11 +2196,48 @@ void xen_setup_callback_vector(void)
>>   callback_via = HVM_CALLBACK_VECTOR(HYPERVISOR_CALLBACK_VECTOR);
>>   if (xen_set_callback_via(callback_via)) {
>>   pr_err("Request for Xen HVM callback vector failed\n");
>> -    xen_have_vector_callback = 0;
>> +    xen_have_vector_callback = false;
>>   }
>>   }
>>   }
>> +/* Setup per-vCPU vector-type callbacks and trick toolstack to think
>> + * we are enlightened. If this setup is unavailable, fallback to the
>> + * global vector-type callback. */
>> +static __init void xen_init_setup_upcall_vector(void)
>> +{
>> +    unsigned int cpu = 0;
>> +
>> +    if (!xen_have_vector_callback)
>> +    return;
>> +
>> +    if ((cpuid_eax(xen_cpuid_base() + 4) & 
>> XEN_HVM_CPUID_UPCALL_VECTOR) &&
>> +    !xen_set_upcall_vector(cpu) && !xen_set_callback_via(1))
> 
> xen_cpuid_base() is an x86-ism. This is going to build because 
> CONFIG_XEN_PVHVM is only set for x86. However, I think this is quite 
> fragile.
> 
> You are also using more variable defined only on x86. So it feels to me 
> that these functions should be implemented in x86 code.
I can surround those 4 callback/upcall functions with an ##ifdef.>> 
+    xen_percpu_upcall = true;
>> +    else if (xen_feature(XENFEAT_hvm_callback_vector))
>> +    xen_setup_callback_vector();
>> +    else
>> +    xen_have_vector_callback = false;
>> +}
>> +
>> +int xen_set_upcall_vector(unsigned int cpu)
>> +{
>> +    int rc;
>> +    xen_hvm_evtchn_upcall_vector_t op = {
>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>> +    };
>> +
>> +    rc = HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector, );
>> +    if (rc)
>> +    return rc;
>> +
>> +    if (!cpu)
>> +    rc = xen_set_callback_via(1);
>> +
>> +    return rc;
>> +}
>> +
>>   static __init void xen_alloc_callback_vector(void)
>>   {
>>   if (!xen_have_vector_callback)
>> @@ -2210,6 +2248,8 @@ static __init void xen_alloc_callback_vector(void)
>>   }
>>   #else
>>   void xen_setup_callback_vector(void) {}
>> +static inline void xen_init_setup_upcall_vector(void) {}
>> +int xen_set_upcall_vector(unsigned int cpu) {}
>>   static inline void xen_alloc_callback_vector(void) {}
>>   #endif
>> @@ -2271,10 +2311,9 @@ void __init xen_init_IRQ(void)
>>   if (xen_initial_domain())
>>   pci_xen_initial_domain();
>>   }
>> -    if (xen_feature(XENFEAT_hvm_callback_vector)) {
>> -    xen_setup_callback_vector();
>> -    xen_alloc_callback_vector();
>> -    }
>

Re: [PATCH v2] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 27/07/2022 00:31, Boris Ostrovsky wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
> attachments unless you have verified the sender and know the content is 
> safe.
> 
> On 7/26/22 8:56 AM, Jane Malalane wrote:
>> +/* Setup per-vCPU vector-type callbacks and trick toolstack to think
>> + * we are enlightened. If this setup is unavailable, fallback to the
>> + * global vector-type callback. */
> 
> 
> Comment style.
> 
> 
>> +static __init void xen_init_setup_upcall_vector(void)
>> +{
>> +    unsigned int cpu = 0;
> 
> 
> Unnecessary variable.
> 
> 
>> +
>> +    if (!xen_have_vector_callback)
>> +    return;
>> +
>> +    if ((cpuid_eax(xen_cpuid_base() + 4) & 
>> XEN_HVM_CPUID_UPCALL_VECTOR) &&
>> +    !xen_set_upcall_vector(cpu) && !xen_set_callback_via(1))
>> +    xen_percpu_upcall = true;
>> +    else if (xen_feature(XENFEAT_hvm_callback_vector))
>> +    xen_setup_callback_vector();
>> +    else
>> +    xen_have_vector_callback = false;
>> +}
>> +
>> +int xen_set_upcall_vector(unsigned int cpu)
>> +{
>> +    int rc;
>> +    xen_hvm_evtchn_upcall_vector_t op = {
>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>> +    };
>> +
>> +    rc = HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector, );
>> +    if (rc)
>> +    return rc;
>> +
>> +    if (!cpu)
> 
> 
> A comment (e.g. "Let toolstack know that we are enlightened." or 
> something along these lines) would be useful here.
>Thanks, will include all these changes in a v3.

Jane.

Re: [PATCH v2] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 26/07/2022 13:56, Jane Malalane wrote:
> Implement support for the HVMOP_set_evtchn_upcall_vector hypercall in
> order to set the per-vCPU event channel vector callback on Linux and
> use it in preference of HVM_PARAM_CALLBACK_IRQ.
> 
> If the per-VCPU vector setup is successful on BSP, use this method
> for the APs. If not, fallback to the global vector-type callback.
> 
> Also register callback_irq at per-vCPU event channel setup to trick
> toolstack to think the domain is enlightened.
> 
> Suggested-by: "Roger Pau Monné" 
> Signed-off-by: Jane Malalane 
> ---
> CC: Juergen Gross 
> CC: Boris Ostrovsky 
> CC: Thomas Gleixner 
> CC: Ingo Molnar 
> CC: Borislav Petkov 
> CC: Dave Hansen 
> CC: x...@kernel.org
> CC: "H. Peter Anvin" 
> CC: Stefano Stabellini 
> CC: Oleksandr Tyshchenko 
> CC: Jane Malalane 
> CC: Maximilian Heyne 
> CC: Jan Beulich 
> CC: Colin Ian King 
> CC: xen-devel@lists.xenproject.org
> 
> v2:
>   * remove no_vector_callback
>   * make xen_have_vector_callback a bool
>   * rename xen_ack_upcall to xen_percpu_upcall
>   * fail to bring CPU up on init instead of crashing kernel
>   * add and use xen_set_upcall_vector where suitable
>   * xen_setup_upcall_vector -> xen_init_setup_upcall_vector for clarity
> ---
>   arch/x86/include/asm/xen/cpuid.h   |  2 ++
>   arch/x86/include/asm/xen/events.h  |  3 ++-
>   arch/x86/xen/enlighten.c   |  2 +-
>   arch/x86/xen/enlighten_hvm.c   | 24 ++-
>   arch/x86/xen/suspend_hvm.c | 10 +++-
>   drivers/xen/events/events_base.c   | 49 
> ++
>   include/xen/hvm.h  |  2 ++
>   include/xen/interface/hvm/hvm_op.h | 15 
>   8 files changed, 93 insertions(+), 14 deletions(-)
> 
> diff --git a/arch/x86/include/asm/xen/cpuid.h 
> b/arch/x86/include/asm/xen/cpuid.h
> index 78e667a31d6c..6daa9b0c8d11 100644
> --- a/arch/x86/include/asm/xen/cpuid.h
> +++ b/arch/x86/include/asm/xen/cpuid.h
> @@ -107,6 +107,8 @@
>* ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
>*/
>   #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
> +/* Per-vCPU event channel upcalls */
> +#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
>   
>   /*
>* Leaf 6 (0x4x05)
> diff --git a/arch/x86/include/asm/xen/events.h 
> b/arch/x86/include/asm/xen/events.h
> index 068d9b067c83..62bdceb594f1 100644
> --- a/arch/x86/include/asm/xen/events.h
> +++ b/arch/x86/include/asm/xen/events.h
> @@ -23,7 +23,7 @@ static inline int xen_irqs_disabled(struct pt_regs *regs)
>   /* No need for a barrier -- XCHG is a barrier on x86. */
>   #define xchg_xen_ulong(ptr, val) xchg((ptr), (val))
>   
> -extern int xen_have_vector_callback;
> +extern bool xen_have_vector_callback;
>   
>   /*
>* Events delivered via platform PCI interrupts are always
> @@ -34,4 +34,5 @@ static inline bool xen_support_evtchn_rebind(void)
>   return (!xen_hvm_domain() || xen_have_vector_callback);
>   }
>   
> +extern bool xen_percpu_upcall;
>   #endif /* _ASM_X86_XEN_EVENTS_H */
> diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
> index 30c6e986a6cd..b8db2148c07d 100644
> --- a/arch/x86/xen/enlighten.c
> +++ b/arch/x86/xen/enlighten.c
> @@ -51,7 +51,7 @@ EXPORT_SYMBOL_GPL(xen_start_info);
>   
>   struct shared_info xen_dummy_shared_info;
>   
> -__read_mostly int xen_have_vector_callback;
> +__read_mostly bool xen_have_vector_callback = true;
>   EXPORT_SYMBOL_GPL(xen_have_vector_callback);
>   
>   /*
> diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
> index 8b71b1dd7639..198d3cd3e9a5 100644
> --- a/arch/x86/xen/enlighten_hvm.c
> +++ b/arch/x86/xen/enlighten_hvm.c
> @@ -7,6 +7,8 @@
>   
>   #include 
>   #include 
> +#include 
> +#include 
>   #include 
>   
>   #include 
> @@ -30,6 +32,9 @@
>   
>   static unsigned long shared_info_pfn;
>   
> +__ro_after_init bool xen_percpu_upcall;
> +EXPORT_SYMBOL_GPL(xen_percpu_upcall);
> +
>   void xen_hvm_init_shared_info(void)
>   {
>   struct xen_add_to_physmap xatp;
> @@ -125,6 +130,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
>   {
>   struct pt_regs *old_regs = set_irq_regs(regs);
>   
> + if (xen_percpu_upcall)
> + ack_APIC_irq();
> +
>   inc_irq_stat(irq_hv_callback_count);
>   
>   xen_hvm_evtchn_do_upcall();
> @@ -168,6 +176,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
>   if (!xen_have_vector_callback)
>   return 0;
>   
> + if (xen_percpu_upcall) {
> + rc = xen_set_upcall_vector

[PATCH v2] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

Implement support for the HVMOP_set_evtchn_upcall_vector hypercall in
order to set the per-vCPU event channel vector callback on Linux and
use it in preference of HVM_PARAM_CALLBACK_IRQ.

If the per-VCPU vector setup is successful on BSP, use this method
for the APs. If not, fallback to the global vector-type callback.

Also register callback_irq at per-vCPU event channel setup to trick
toolstack to think the domain is enlightened.

Suggested-by: "Roger Pau Monné" 
Signed-off-by: Jane Malalane 
---
CC: Juergen Gross 
CC: Boris Ostrovsky 
CC: Thomas Gleixner 
CC: Ingo Molnar 
CC: Borislav Petkov 
CC: Dave Hansen 
CC: x...@kernel.org
CC: "H. Peter Anvin" 
CC: Stefano Stabellini 
CC: Oleksandr Tyshchenko 
CC: Jane Malalane 
CC: Maximilian Heyne 
CC: Jan Beulich 
CC: Colin Ian King 
CC: xen-devel@lists.xenproject.org

v2:
 * remove no_vector_callback
 * make xen_have_vector_callback a bool
 * rename xen_ack_upcall to xen_percpu_upcall
 * fail to bring CPU up on init instead of crashing kernel
 * add and use xen_set_upcall_vector where suitable
 * xen_setup_upcall_vector -> xen_init_setup_upcall_vector for clarity
---
 arch/x86/include/asm/xen/cpuid.h   |  2 ++
 arch/x86/include/asm/xen/events.h  |  3 ++-
 arch/x86/xen/enlighten.c   |  2 +-
 arch/x86/xen/enlighten_hvm.c   | 24 ++-
 arch/x86/xen/suspend_hvm.c | 10 +++-
 drivers/xen/events/events_base.c   | 49 ++
 include/xen/hvm.h  |  2 ++
 include/xen/interface/hvm/hvm_op.h | 15 
 8 files changed, 93 insertions(+), 14 deletions(-)

diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
index 78e667a31d6c..6daa9b0c8d11 100644
--- a/arch/x86/include/asm/xen/cpuid.h
+++ b/arch/x86/include/asm/xen/cpuid.h
@@ -107,6 +107,8 @@
  * ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/* Per-vCPU event channel upcalls */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
diff --git a/arch/x86/include/asm/xen/events.h 
b/arch/x86/include/asm/xen/events.h
index 068d9b067c83..62bdceb594f1 100644
--- a/arch/x86/include/asm/xen/events.h
+++ b/arch/x86/include/asm/xen/events.h
@@ -23,7 +23,7 @@ static inline int xen_irqs_disabled(struct pt_regs *regs)
 /* No need for a barrier -- XCHG is a barrier on x86. */
 #define xchg_xen_ulong(ptr, val) xchg((ptr), (val))
 
-extern int xen_have_vector_callback;
+extern bool xen_have_vector_callback;
 
 /*
  * Events delivered via platform PCI interrupts are always
@@ -34,4 +34,5 @@ static inline bool xen_support_evtchn_rebind(void)
return (!xen_hvm_domain() || xen_have_vector_callback);
 }
 
+extern bool xen_percpu_upcall;
 #endif /* _ASM_X86_XEN_EVENTS_H */
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 30c6e986a6cd..b8db2148c07d 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -51,7 +51,7 @@ EXPORT_SYMBOL_GPL(xen_start_info);
 
 struct shared_info xen_dummy_shared_info;
 
-__read_mostly int xen_have_vector_callback;
+__read_mostly bool xen_have_vector_callback = true;
 EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 
 /*
diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index 8b71b1dd7639..198d3cd3e9a5 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -7,6 +7,8 @@
 
 #include 
 #include 
+#include 
+#include 
 #include 
 
 #include 
@@ -30,6 +32,9 @@
 
 static unsigned long shared_info_pfn;
 
+__ro_after_init bool xen_percpu_upcall;
+EXPORT_SYMBOL_GPL(xen_percpu_upcall);
+
 void xen_hvm_init_shared_info(void)
 {
struct xen_add_to_physmap xatp;
@@ -125,6 +130,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
 {
struct pt_regs *old_regs = set_irq_regs(regs);
 
+   if (xen_percpu_upcall)
+   ack_APIC_irq();
+
inc_irq_stat(irq_hv_callback_count);
 
xen_hvm_evtchn_do_upcall();
@@ -168,6 +176,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
if (!xen_have_vector_callback)
return 0;
 
+   if (xen_percpu_upcall) {
+   rc = xen_set_upcall_vector(cpu);
+   if (rc) {
+   WARN(1, "HVMOP_set_evtchn_upcall_vector"
+" for CPU %d failed: %d\n", cpu, rc);
+   return rc;
+   }
+   }
+
if (xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_timer(cpu);
 
@@ -188,8 +205,6 @@ static int xen_cpu_dead_hvm(unsigned int cpu)
return 0;
 }
 
-static bool no_vector_callback __initdata;
-
 static void __init xen_hvm_guest_init(void)
 {
if (xen_pv_domain())
@@ -211,9 +226,6 @@ static void __init xen_hvm_guest_init(void)
 
xen_panic_handler_init();
 
-   if (!no_vector_callback && xen_feature(XENFEAT_hvm_callback_vector))
-

Re: [PATCH] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 25/07/2022 21:46, Boris Ostrovsky wrote:
> 
> On 7/25/22 6:03 AM, Jane Malalane wrote:
>> On 18/07/2022 14:59, Boris Ostrovsky wrote:
>>> On 7/18/22 4:56 AM, Andrew Cooper wrote:
>>>> On 15/07/2022 14:10, Boris Ostrovsky wrote:
>>>>> On 7/15/22 5:50 AM, Andrew Cooper wrote:
>>>>>> On 15/07/2022 09:18, Jane Malalane wrote:
>>>>>>> On 14/07/2022 00:27, Boris Ostrovsky wrote:
>>>>>>>>>  xen_hvm_smp_init();
>>>>>>>>>  WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm,
>>>>>>>>> xen_cpu_dead_hvm));
>>>>>>>>> diff --git a/arch/x86/xen/suspend_hvm.c 
>>>>>>>>> b/arch/x86/xen/suspend_hvm.c
>>>>>>>>> index 9d548b0c772f..be66e027ef28 100644
>>>>>>>>> --- a/arch/x86/xen/suspend_hvm.c
>>>>>>>>> +++ b/arch/x86/xen/suspend_hvm.c
>>>>>>>>> @@ -5,6 +5,7 @@
>>>>>>>>>  #include 
>>>>>>>>>  #include 
>>>>>>>>>  #include 
>>>>>>>>> +#include 
>>>>>>>>>  #include "xen-ops.h"
>>>>>>>>> @@ -14,6 +15,23 @@ void xen_hvm_post_suspend(int 
>>>>>>>>> suspend_cancelled)
>>>>>>>>>  xen_hvm_init_shared_info();
>>>>>>>>>  xen_vcpu_restore();
>>>>>>>>>  }
>>>>>>>>> -    xen_setup_callback_vector();
>>>>>>>>> +    if (xen_ack_upcall) {
>>>>>>>>> +    unsigned int cpu;
>>>>>>>>> +
>>>>>>>>> +    for_each_online_cpu(cpu) {
>>>>>>>>> +    xen_hvm_evtchn_upcall_vector_t op = {
>>>>>>>>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>>>>>>>>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>>>>>>>>> +    };
>>>>>>>>> +
>>>>>>>>> +
>>>>>>>>> BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector,
>>>>>>>>> + ));
>>>>>>>>> +    /* Trick toolstack to think we are enlightened. */
>>>>>>>>> +    if (!cpu)
>>>>>>>>> +    BUG_ON(xen_set_callback_via(1));
>>>>>>>> What are you trying to make the toolstack aware of? That we have 
>>>>>>>> *a*
>>>>>>>> callback (either global or percpu)?
>>>>>>> Yes, specifically for the check in 
>>>>>>> libxl__domain_pvcontrol_available.
>>>>>> And others.
>>>>>>
>>>>>> This is all a giant bodge, but basically a lot of tooling uses the
>>>>>> non-zero-ness of the CALLBACK_VIA param to determine whether the 
>>>>>> VM has
>>>>>> Xen-aware drivers loaded or not.
>>>>>>
>>>>>> The value 1 is a CALLBACK_VIA value which encodes GSI 1, and the only
>>>>>> reason this doesn't explode everywhere is because the
>>>>>> evtchn_upcall_vector registration takes priority over GSI delivery.
>>>>>>
>>>>>> This is decades of tech debt piled on top of tech debt.
>>>>> Feels like it (setting the callback parameter) is something that the
>>>>> hypervisor should do --- no need to expose guests to this.
>>>> Sensible or not, it is the ABI.
>>>>
>>>> Linux still needs to work (nicely) with older Xen's in the world, 
>>>> and we
>>>> can't just retrofit a change in the hypervisor which says "btw, this 
>>>> ABI
>>>> we've just changed now has a side effect of modifying a field that you
>>>> also logically own".
>>>
>>> The hypercall has been around for a while so I understand ABI concerns
>>> there but XEN_HVM_CPUID_UPCALL_VECTOR was introduced only a month ago.
>>> Why not tie presence of this bit to no longer having to explicitly set
>>> the callback field?
>>>
>> Any other opinions on this?
>>
>> (i.e., calling xen_set_callback_via(1) after
>> HVMOP_set_evtchn_upcall_vector OR not exposing this to guests and
>> instead having Xen call this function (in hvmop_set_evtchn_upcall_vector
>> maybe) and tieing its presense to XEN_HVM_CPUID_UPCALL_VECTOR which was
>> recently added)
> 
> 
> CPUID won't help here, I wasn't thinking clearly.
> 
> 
> Can we wrap the HVMOP_set_evtchn_upcall_vector hypercall in a function 
> that will decide whether or not to also do xen_set_callback_via(1)?
> 
Okay. Will do this in a v2.

Thanks,

Jane.

Re: [PATCH] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 18/07/2022 14:59, Boris Ostrovsky wrote:
> 
> On 7/18/22 4:56 AM, Andrew Cooper wrote:
>> On 15/07/2022 14:10, Boris Ostrovsky wrote:
>>> On 7/15/22 5:50 AM, Andrew Cooper wrote:
>>>> On 15/07/2022 09:18, Jane Malalane wrote:
>>>>> On 14/07/2022 00:27, Boris Ostrovsky wrote:
>>>>>>>     xen_hvm_smp_init();
>>>>>>>     WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm,
>>>>>>> xen_cpu_dead_hvm));
>>>>>>> diff --git a/arch/x86/xen/suspend_hvm.c b/arch/x86/xen/suspend_hvm.c
>>>>>>> index 9d548b0c772f..be66e027ef28 100644
>>>>>>> --- a/arch/x86/xen/suspend_hvm.c
>>>>>>> +++ b/arch/x86/xen/suspend_hvm.c
>>>>>>> @@ -5,6 +5,7 @@
>>>>>>>     #include 
>>>>>>>     #include 
>>>>>>>     #include 
>>>>>>> +#include 
>>>>>>>     #include "xen-ops.h"
>>>>>>> @@ -14,6 +15,23 @@ void xen_hvm_post_suspend(int suspend_cancelled)
>>>>>>>     xen_hvm_init_shared_info();
>>>>>>>     xen_vcpu_restore();
>>>>>>>     }
>>>>>>> -    xen_setup_callback_vector();
>>>>>>> +    if (xen_ack_upcall) {
>>>>>>> +    unsigned int cpu;
>>>>>>> +
>>>>>>> +    for_each_online_cpu(cpu) {
>>>>>>> +    xen_hvm_evtchn_upcall_vector_t op = {
>>>>>>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>>>>>>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>>>>>>> +    };
>>>>>>> +
>>>>>>> +
>>>>>>> BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector,
>>>>>>> + ));
>>>>>>> +    /* Trick toolstack to think we are enlightened. */
>>>>>>> +    if (!cpu)
>>>>>>> +    BUG_ON(xen_set_callback_via(1));
>>>>>> What are you trying to make the toolstack aware of? That we have *a*
>>>>>> callback (either global or percpu)?
>>>>> Yes, specifically for the check in libxl__domain_pvcontrol_available.
>>>> And others.
>>>>
>>>> This is all a giant bodge, but basically a lot of tooling uses the
>>>> non-zero-ness of the CALLBACK_VIA param to determine whether the VM has
>>>> Xen-aware drivers loaded or not.
>>>>
>>>> The value 1 is a CALLBACK_VIA value which encodes GSI 1, and the only
>>>> reason this doesn't explode everywhere is because the
>>>> evtchn_upcall_vector registration takes priority over GSI delivery.
>>>>
>>>> This is decades of tech debt piled on top of tech debt.
>>>
>>> Feels like it (setting the callback parameter) is something that the
>>> hypervisor should do --- no need to expose guests to this.
>> Sensible or not, it is the ABI.
>>
>> Linux still needs to work (nicely) with older Xen's in the world, and we
>> can't just retrofit a change in the hypervisor which says "btw, this ABI
>> we've just changed now has a side effect of modifying a field that you
>> also logically own".
> 
> 
> The hypercall has been around for a while so I understand ABI concerns 
> there but XEN_HVM_CPUID_UPCALL_VECTOR was introduced only a month ago. 
> Why not tie presence of this bit to no longer having to explicitly set 
> the callback field?
> 
Any other opinions on this?

(i.e., calling xen_set_callback_via(1) after 
HVMOP_set_evtchn_upcall_vector OR not exposing this to guests and 
instead having Xen call this function (in hvmop_set_evtchn_upcall_vector 
maybe) and tieing its presense to XEN_HVM_CPUID_UPCALL_VECTOR which was 
recently added)

Thank you,

Jane.

Re: [PATCH] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

On 14/07/2022 00:27, Boris Ostrovsky wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
> attachments unless you have verified the sender and know the content is 
> safe.
> 
> On 7/11/22 11:22 AM, Jane Malalane wrote:
>> --- a/arch/x86/xen/enlighten_hvm.c
>> +++ b/arch/x86/xen/enlighten_hvm.c
>> @@ -7,6 +7,7 @@
>>   #include 
>>   #include 
>> +#include 
>>   #include 
>>   #include 
>> @@ -30,6 +31,9 @@
>>   static unsigned long shared_info_pfn;
>> +__ro_after_init bool xen_ack_upcall;
>> +EXPORT_SYMBOL_GPL(xen_ack_upcall);
> 
> 
> Shouldn't this be called something like xen_percpu_upcall?
Sure.
> 
> 
>> +
>>   void xen_hvm_init_shared_info(void)
>>   {
>>   struct xen_add_to_physmap xatp;
>> @@ -125,6 +129,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
>>   {
>>   struct pt_regs *old_regs = set_irq_regs(regs);
>> +    if (xen_ack_upcall)
>> +    ack_APIC_irq();
>> +
>>   inc_irq_stat(irq_hv_callback_count);
>>   xen_hvm_evtchn_do_upcall();
>> @@ -168,6 +175,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
>>   if (!xen_have_vector_callback)
>>   return 0;
>> +    if (xen_ack_upcall) {
>> +    xen_hvm_evtchn_upcall_vector_t op = {
>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>> +    };
>> +
>> +    BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector, ));
> 
> 
> Does this have to be fatal? Can't we just fail bringing this vcpu up?
Yes, will amend.
> 
> 
>> +    }
>> +
>>   if (xen_feature(XENFEAT_hvm_safe_pvclock))
>>   xen_setup_timer(cpu);
>> @@ -211,8 +227,7 @@ static void __init xen_hvm_guest_init(void)
>>   xen_panic_handler_init();
>> -    if (!no_vector_callback && xen_feature(XENFEAT_hvm_callback_vector))
>> -    xen_have_vector_callback = 1;
>> +    xen_have_vector_callback = !no_vector_callback;
> 
> 
> Can we get rid of one of those two variables then?
I'll remove no_vector_callback for less code changes.
> 
> 
>>   xen_hvm_smp_init();
>>   WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm));
>> diff --git a/arch/x86/xen/suspend_hvm.c b/arch/x86/xen/suspend_hvm.c
>> index 9d548b0c772f..be66e027ef28 100644
>> --- a/arch/x86/xen/suspend_hvm.c
>> +++ b/arch/x86/xen/suspend_hvm.c
>> @@ -5,6 +5,7 @@
>>   #include 
>>   #include 
>>   #include 
>> +#include 
>>   #include "xen-ops.h"
>> @@ -14,6 +15,23 @@ void xen_hvm_post_suspend(int suspend_cancelled)
>>   xen_hvm_init_shared_info();
>>   xen_vcpu_restore();
>>   }
>> -    xen_setup_callback_vector();
>> +    if (xen_ack_upcall) {
>> +    unsigned int cpu;
>> +
>> +    for_each_online_cpu(cpu) {
>> +    xen_hvm_evtchn_upcall_vector_t op = {
>> +    .vector = HYPERVISOR_CALLBACK_VECTOR,
>> +    .vcpu = per_cpu(xen_vcpu_id, cpu),
>> +    };
>> +
>> +    BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector,
>> + ));
>> +    /* Trick toolstack to think we are enlightened. */
>> +    if (!cpu)
>> +    BUG_ON(xen_set_callback_via(1));
> 
> 
> What are you trying to make the toolstack aware of? That we have *a* 
> callback (either global or percpu)?
Yes, specifically for the check in libxl__domain_pvcontrol_available.
> 
> 
> BTW, you can take it out the loop. And maybe @op definition too, except 
> for .vcpu assignment.
> 
> 
>> +    }
>> +    } else {
>> +    xen_setup_callback_vector();
>> +    }
>>   xen_unplug_emulated_devices();
>>   }
> 
> 
> -boris
> 
> 

Thank you,

Jane.

[PATCH] x86/xen: Add support for HVMOP_set_evtchn_upcall_vector

[Jane Malalane]

Implement support for the HVMOP_set_evtchn_upcall_vector hypercall in
order to set the per-vCPU event channel vector callback on Linux and
use it in preference of HVM_PARAM_CALLBACK_IRQ.

If the per-VCPU vector setup is successful on BSP, use this method
for the APs. If not, fallback to the global vector-type callback.

Also register callback_irq at per-vCPU event channel setup to trick
toolstack to think the domain is enlightened.

Suggested-by: "Roger Pau Monné" 
Signed-off-by: Jane Malalane 
---
CC: Juergen Gross 
CC: Boris Ostrovsky 
CC: Thomas Gleixner 
CC: Ingo Molnar 
CC: Borislav Petkov 
CC: Dave Hansen 
CC: x...@kernel.org
CC: "H. Peter Anvin" 
CC: Stefano Stabellini 
CC: Oleksandr Tyshchenko 
CC: Jane Malalane 
CC: Maximilian Heyne 
CC: Jan Beulich 
CC: Colin Ian King 
CC: xen-devel@lists.xenproject.org
---
 arch/x86/include/asm/xen/cpuid.h   |  2 ++
 arch/x86/include/asm/xen/events.h  |  1 +
 arch/x86/xen/enlighten_hvm.c   | 19 +--
 arch/x86/xen/suspend_hvm.c | 20 +++-
 drivers/xen/events/events_base.c   | 32 
 include/xen/interface/hvm/hvm_op.h | 15 +++
 6 files changed, 82 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/xen/cpuid.h b/arch/x86/include/asm/xen/cpuid.h
index 78e667a31d6c..6daa9b0c8d11 100644
--- a/arch/x86/include/asm/xen/cpuid.h
+++ b/arch/x86/include/asm/xen/cpuid.h
@@ -107,6 +107,8 @@
  * ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/* Per-vCPU event channel upcalls */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
diff --git a/arch/x86/include/asm/xen/events.h 
b/arch/x86/include/asm/xen/events.h
index 068d9b067c83..b2d86c761bf8 100644
--- a/arch/x86/include/asm/xen/events.h
+++ b/arch/x86/include/asm/xen/events.h
@@ -34,4 +34,5 @@ static inline bool xen_support_evtchn_rebind(void)
return (!xen_hvm_domain() || xen_have_vector_callback);
 }
 
+extern bool xen_ack_upcall;
 #endif /* _ASM_X86_XEN_EVENTS_H */
diff --git a/arch/x86/xen/enlighten_hvm.c b/arch/x86/xen/enlighten_hvm.c
index 8b71b1dd7639..847d1da46ff7 100644
--- a/arch/x86/xen/enlighten_hvm.c
+++ b/arch/x86/xen/enlighten_hvm.c
@@ -7,6 +7,7 @@
 
 #include 
 #include 
+#include 
 #include 
 
 #include 
@@ -30,6 +31,9 @@
 
 static unsigned long shared_info_pfn;
 
+__ro_after_init bool xen_ack_upcall;
+EXPORT_SYMBOL_GPL(xen_ack_upcall);
+
 void xen_hvm_init_shared_info(void)
 {
struct xen_add_to_physmap xatp;
@@ -125,6 +129,9 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_xen_hvm_callback)
 {
struct pt_regs *old_regs = set_irq_regs(regs);
 
+   if (xen_ack_upcall)
+   ack_APIC_irq();
+
inc_irq_stat(irq_hv_callback_count);
 
xen_hvm_evtchn_do_upcall();
@@ -168,6 +175,15 @@ static int xen_cpu_up_prepare_hvm(unsigned int cpu)
if (!xen_have_vector_callback)
return 0;
 
+   if (xen_ack_upcall) {
+   xen_hvm_evtchn_upcall_vector_t op = {
+   .vector = HYPERVISOR_CALLBACK_VECTOR,
+   .vcpu = per_cpu(xen_vcpu_id, cpu),
+   };
+
+   BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector, ));
+   }
+
if (xen_feature(XENFEAT_hvm_safe_pvclock))
xen_setup_timer(cpu);
 
@@ -211,8 +227,7 @@ static void __init xen_hvm_guest_init(void)
 
xen_panic_handler_init();
 
-   if (!no_vector_callback && xen_feature(XENFEAT_hvm_callback_vector))
-   xen_have_vector_callback = 1;
+   xen_have_vector_callback = !no_vector_callback;
 
xen_hvm_smp_init();
WARN_ON(xen_cpuhp_setup(xen_cpu_up_prepare_hvm, xen_cpu_dead_hvm));
diff --git a/arch/x86/xen/suspend_hvm.c b/arch/x86/xen/suspend_hvm.c
index 9d548b0c772f..be66e027ef28 100644
--- a/arch/x86/xen/suspend_hvm.c
+++ b/arch/x86/xen/suspend_hvm.c
@@ -5,6 +5,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "xen-ops.h"
 
@@ -14,6 +15,23 @@ void xen_hvm_post_suspend(int suspend_cancelled)
xen_hvm_init_shared_info();
xen_vcpu_restore();
}
-   xen_setup_callback_vector();
+   if (xen_ack_upcall) {
+   unsigned int cpu;
+
+   for_each_online_cpu(cpu) {
+   xen_hvm_evtchn_upcall_vector_t op = {
+   .vector = HYPERVISOR_CALLBACK_VECTOR,
+   .vcpu = per_cpu(xen_vcpu_id, cpu),
+   };
+
+   BUG_ON(HYPERVISOR_hvm_op(HVMOP_set_evtchn_upcall_vector,
+));
+   /* Trick toolstack to think we are enlightened. */
+   if (!cpu)
+   BUG_ON(xen_set_callback_via(1));
+   }
+   

Re: [PATCH RESEND v10 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 11/07/2022 09:26, Jan Beulich wrote:
> On 11.07.2022 10:00, Jane Malalane wrote:
>> On 30/06/2022 07:03, Jan Beulich wrote:
>>> On 30.06.2022 05:25, Tian, Kevin wrote:
>>>>> From: Jane Malalane 
>>>>> Sent: Wednesday, June 29, 2022 9:56 PM
>>>>>
>>>>> Introduce a new per-domain creation x86 specific flag to
>>>>> select whether hardware assisted virtualization should be used for
>>>>> x{2}APIC.
>>>>>
>>>>> A per-domain option is added to xl in order to select the usage of
>>>>> x{2}APIC hardware assisted virtualization, as well as a global
>>>>> configuration option.
>>>>>
>>>>> Having all APIC interaction exit to Xen for emulation is slow and can
>>>>> induce much overhead. Hardware can speed up x{2}APIC by decoding the
>>>>> APIC access and providing a VM exit with a more specific exit reason
>>>>> than a regular EPT fault or by altogether avoiding a VM exit.
>>>>
>>>> Above is obvious and could be removed.
>>>>
>>>> I think the key is just the next paragraph for why we
>>>> want this per-domain control.
>>>
>>> Indeed, but the paragraph above sets the context. It might be possible
>>> to shorten it, but ...
>>>
>>>> Apart from that:
>>>>
>>>> Reviewed-by: Kevin Tian 
>>>>
>>>>>
>>>>> On the other hand, being able to disable x{2}APIC hardware assisted
>>>>> virtualization can be useful for testing and debugging purposes.
>>>
>>> ... I think it is desirable for this sentence to start with "Otoh" or
>>> alike.
>>>
>>> JanHello Jan,
>>
>> In the previous email, I was referring to this discussion about the
>> commit message. I haven't sent out a v11 because there was no change
>> other than this one suggested. What I said earlier was that I thought
>> the "Having all APIC interaction exit to Xen for emulation is slow..."
>> paragraph provided context for what I say after but I am happy to remove it.
> 
> I'd be fine for it to be kept as you had it, but you really should have
> sent out both patches. There are rare cases where sending out individual
> updates within a series is reasonable (e.g. to avoid spamming the list
> with a large amount of unchanged patches), but I think here you want to
> make things easy for committers and not have them hunt down the earlier
> version.

Apologies, makes sense.

Thanks,

Jane.

[PATCH v11 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note:

- vmx_install_vlapic_mapping doesn't require modifications regardless
of whether the guest has "Virtualize APIC accesses" enabled or not,
i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long as
virtualize_apic_accesses is supported by the CPU.

- Both per-domain and global assisted_x{2}apic options are not part of
the migration stream, unless explicitly set in the respective
configuration files. Default settings of assisted_x{2}apic done
internally by the toolstack, based on host capabilities at create
time, are not migrated.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Acked-by: Christian Lindig 
Reviewed-by: "Roger Pau Monné" 
Reviewed-by: Anthony PERARD 
Reviewed-by: Kevin Tian 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: George Dunlap 
CC: Nick Rosbrook 
CC: Juergen Gross 
CC: Christian Lindig 
CC: David Scott 
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Jun Nakajima 
CC: Kevin Tian 

v11:
 * No change

v10:
 * Improve commit message note on migration

v9:
 * Fix style issues
 * Fix exit() logic for assisted_x{2}apic parsing
 * Add and use XEN_X86_MISC_FLAGS_MAX for ABI checking instead of
   using XEN_X86_ASSISTED_X2APIC directly
 * Expand commit message to mention migration is safe

v8:
 * Widen assisted_x{2}apic parsing to PVH guests in
   parse_config_data()

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h | 

Re: [PATCH RESEND v10 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 30/06/2022 07:03, Jan Beulich wrote:
> On 30.06.2022 05:25, Tian, Kevin wrote:
>>> From: Jane Malalane 
>>> Sent: Wednesday, June 29, 2022 9:56 PM
>>>
>>> Introduce a new per-domain creation x86 specific flag to
>>> select whether hardware assisted virtualization should be used for
>>> x{2}APIC.
>>>
>>> A per-domain option is added to xl in order to select the usage of
>>> x{2}APIC hardware assisted virtualization, as well as a global
>>> configuration option.
>>>
>>> Having all APIC interaction exit to Xen for emulation is slow and can
>>> induce much overhead. Hardware can speed up x{2}APIC by decoding the
>>> APIC access and providing a VM exit with a more specific exit reason
>>> than a regular EPT fault or by altogether avoiding a VM exit.
>>
>> Above is obvious and could be removed.
>>
>> I think the key is just the next paragraph for why we
>> want this per-domain control.
> 
> Indeed, but the paragraph above sets the context. It might be possible
> to shorten it, but ...
> 
>> Apart from that:
>>
>> Reviewed-by: Kevin Tian 
>>
>>>
>>> On the other hand, being able to disable x{2}APIC hardware assisted
>>> virtualization can be useful for testing and debugging purposes.
> 
> ... I think it is desirable for this sentence to start with "Otoh" or
> alike.
> 
> JanHello Jan,

In the previous email, I was referring to this discussion about the 
commit message. I haven't sent out a v11 because there was no change 
other than this one suggested. What I said earlier was that I thought 
the "Having all APIC interaction exit to Xen for emulation is slow..." 
paragraph provided context for what I say after but I am happy to remove it.

Thanks,

Jane.

Re: [PATCH v11 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 08/07/2022 15:33, Christian Lindig wrote:
> 
> 
> On 8 Jul 2022, at 14:55, Jane Malalane 
> mailto:jane.malal...@citrix.com>> wrote:
> 
> tools/ocaml/libs/xc/xenctrl.ml   |  9 +
> tools/ocaml/libs/xc/xenctrl.mli  |  8 
> tools/ocaml/libs/xc/xenctrl_stubs.c  | 18 —
> 
> Acked-by: Christian Lindig 
> mailto:christian.lin...@citrix.com>>
> 
> 
Hello Jan,

I believe I now have all the acks needed for this patch series. 
Regarding the commit message of patch 2, I thought that explanation made 
it clearer but I can always remove it.

Thank you,

Jane.

[PATCH v11 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Reviewed-by: "Roger Pau Monné" 
Reviewed-by: Jan Beulich 
Reviewed-by: Anthony PERARD 
Reviewed-by: Kevin Tian 
Reviewed-by: George Dunlap 
---
CC: George Dunlap 
CC: Nick Rosbrook 
CC: Wei Liu 
CC: Anthony PERARD 
CC: Andrew Cooper 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Juergen Gross 
CC: Christian Lindig 
CC: David Scott 
CC: "Roger Pau Monné" 
CC: Jun Nakajima 
CC: Kevin Tian 

v11:
 * explicity define arch_cap_list in stub_xc_physinfo for non-x86 arch

v10:
 * Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()

v9:
 * Move assisted_x{2}apic_available to vmx_vmcs_init() so they get
   declared at boot time, after vmx_secondary_exec_control is set

v8:
 * Improve commit message

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/gnttab/st8ojiU6   |  0
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  9 +
 tools/ocaml/libs/xc/xenctrl.mli  |  8 
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 18 --
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  7 +++
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 18 files changed, 104 insertions(+), 5 deletions(-)
 create mode 100644 tools/libs/gnttab/st8ojiU6

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGn

Re: [PATCH RESEND v10 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 29/06/2022 15:49, Christian Lindig wrote:
> 
> 
> On 29 Jun 2022, at 14:55, Jane Malalane 
> mailto:jane.malal...@citrix.com>> wrote:
> 
> + physinfo = caml_alloc_tuple(11);
> Store_field(physinfo, 0, Val_int(c_physinfo.threads_per_core));
> Store_field(physinfo, 1, Val_int(c_physinfo.cores_per_socket));
> Store_field(physinfo, 2, Val_int(c_physinfo.nr_cpus));
> @@ -749,6 +749,17 @@ CAMLprim value stub_xc_physinfo(value xch)
> Store_field(physinfo, 8, cap_list);
> Store_field(physinfo, 9, Val_int(c_physinfo.max_cpu_id + 1));
> 
> +#if defined(__i386__) || defined(__x86_64__)
> + /*
> +  * arch_capabilities: physinfo_arch_cap_flag list;
> +  */
> + arch_cap_list = c_bitmap_to_ocaml_list
> + /* ! physinfo_arch_cap_flag CAP_ none */
> + /* ! XEN_SYSCTL_PHYSCAP_ XEN_SYSCTL_PHYSCAP_X86_MAX max */
> + (c_physinfo.arch_capabilities);
> + Store_field(physinfo, 10, arch_cap_list);
> +#endif
> +
> CAMLreturn(physinfo);
> }
> 
> I this extending the tuple but only defining a value on x86? Does this not 
> lead to undefined fields on other architectures?

You're right, it's missing a definition, I will send a new version - 
will just give some time for more eventual comments from others on the 
series overall.

Thank you,

Jane

Re: [PATCH RESEND v10 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 29/06/2022 15:26, Jan Beulich wrote:
> On 29.06.2022 15:55, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
>> XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
>> x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
>> can subsequently be enabled on a per-domain basis.
>> No such features are currently implemented on AMD hardware.
>>
>> HW assisted xAPIC virtualization will be reported if HW, at the
>> minimum, supports virtualize_apic_accesses as this feature alone means
>> that an access to the APIC page will cause an APIC-access VM exit. An
>> APIC-access VM exit provides a VMM with information about the access
>> causing the VM exit, unlike a regular EPT fault, thus simplifying some
>> internal handling.
>>
>> HW assisted x2APIC virtualization will be reported if HW supports
>> virtualize_x2apic_mode and, at least, either apic_reg_virt or
>> virtual_intr_delivery. This also means that
>> sysctl follows the conditionals in vmx_vlapic_msr_changed().
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Note that this interface is intended to be compatible with AMD so that
>> AVIC support can be introduced in a future patch. Unlike Intel that
>> has multiple controls for APIC Virtualization, AMD has one global
>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>> control cannot be done on a common interface.
>>
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
>> Reviewed-by: "Roger Pau Monné" 
>> Reviewed-by: Jan Beulich 
>> Reviewed-by: Anthony PERARD 
> 
> Could you please clarify whether you did drop Kevin's R-b (which, a
> little unhelpfully, he provided in reply to v9 a week after you had
> posted v10) because of ...
> 
>> v10:
>>   * Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()
> 
> ... this, requiring him to re-offer the tag? Until told otherwise I
> will assume so.

It wasn't intentional but yes, that is right. There was a change, albeit 
minor, in vmx from v9 to v10 so I do require Kevin Tian or Jun Nakajima 
to review it.

Thank you,

Jane.<>

[PATCH RESEND v10 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note:

- vmx_install_vlapic_mapping doesn't require modifications regardless
of whether the guest has "Virtualize APIC accesses" enabled or not,
i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long as
virtualize_apic_accesses is supported by the CPU.

- Both per-domain and global assisted_x{2}apic options are not part of
the migration stream, unless explicitly set in the respective
configuration files. Default settings of assisted_x{2}apic done
internally by the toolstack, based on host capabilities at create
time, are not migrated.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Acked-by: Christian Lindig 
Reviewed-by: "Roger Pau Monné" 
Reviewed-by: Anthony PERARD 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: George Dunlap 
CC: Nick Rosbrook 
CC: Juergen Gross 
CC: Christian Lindig 
CC: David Scott 
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Jun Nakajima 
CC: Kevin Tian 

v10:
 * Improve commit message note on migration

v9:
 * Fix style issues
 * Fix exit() logic for assisted_x{2}apic parsing
 * Add and use XEN_X86_MISC_FLAGS_MAX for ABI checking instead of
   using XEN_X86_ASSISTED_X2APIC directly
 * Expand commit message to mention migration is safe

v8:
 * Widen assisted_x{2}apic parsing to PVH guests in
   parse_config_data()

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light

[PATCH RESEND v10 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Reviewed-by: "Roger Pau Monné" 
Reviewed-by: Jan Beulich 
Reviewed-by: Anthony PERARD 
---
CC: George Dunlap 
CC: Nick Rosbrook 
CC: Wei Liu 
CC: Anthony PERARD 
CC: Andrew Cooper 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Juergen Gross 
CC: Christian Lindig 
CC: David Scott 
CC: "Roger Pau Monné" 
CC: Jun Nakajima 
CC: Kevin Tian 

v10:
 * Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()

v9:
 * Move assisted_x{2}apic_available to vmx_vmcs_init() so they get
   declared at boot time, after vmx_secondary_exec_control is set

v8:
 * Improve commit message

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  7 +++
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 94 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C

[PATCH RESEND v10 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 15 ++
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 14 ++---
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 17 ---
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 19 +
 xen/arch/x86/domain.c | 29 +-
 xen/arch/x86/hvm/hvm.c|  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c   | 11 ++
 xen/arch/x86/hvm/vmx/vmx.c| 13 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h| 10 +
 xen/arch/x86/sysctl.c |  4 
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  5 +
 xen/include/public/sysctl.h   | 11 +-
 28 files changed, 281 insertions(+), 34 deletions(-)

-- 
2.11.0

[PATCH v4] x86/hvm: Widen condition for is_hvm_pv_evtchn_domain() and report fix in CPUID

[Jane Malalane]

Have is_hvm_pv_evtchn_domain() return true for vector callbacks for
evtchn delivery set up on a per-vCPU basis via
HVMOP_set_evtchn_upcall_vector.

Assume that if vCPU0 uses HVMOP_set_evtchn_upcall_vector, all
remaining vCPUs will too and thus remove is_hvm_pv_evtchn_vcpu() and
replace sole caller with is_hvm_pv_evtchn_domain().

is_hvm_pv_evtchn_domain() returning true is a condition for setting up
physical IRQ to event channel mappings. Therefore, also add a CPUID
bit so that guests know whether the check in is_hvm_pv_evtchn_domain()
will fail when using HVMOP_set_evtchn_upcall_vector. This matters for
guests that route PIRQs over event channels since
is_hvm_pv_evtchn_domain() is a condition in physdev_map_pirq().

The naming of the CPUID bit is quite generic about upcall support
being available. That's done so that the define name doesn't become
overly long.

A guest that doesn't care about physical interrupts routed over event
channels can just test for the availability of the hypercall directly
(HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.

Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 

v4:
 * Remove is_hvm_pv_evtchn_vcpu and replace sole caller.

v3:
 * Improve commit message and title.

v2:
 * Since the naming of the CPUID bit is quite generic, better explain
   when it should be checked for, in code comments and commit message.
---
 xen/arch/x86/hvm/irq.c  | 2 +-
 xen/arch/x86/include/asm/domain.h   | 9 +++--
 xen/arch/x86/traps.c| 6 ++
 xen/include/public/arch-x86/cpuid.h | 5 +
 4 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/hvm/irq.c b/xen/arch/x86/hvm/irq.c
index 5a7f39b54f..19252448cb 100644
--- a/xen/arch/x86/hvm/irq.c
+++ b/xen/arch/x86/hvm/irq.c
@@ -325,7 +325,7 @@ void hvm_assert_evtchn_irq(struct vcpu *v)
 
 vlapic_set_irq(vcpu_vlapic(v), vector, 0);
 }
-else if ( is_hvm_pv_evtchn_vcpu(v) )
+else if ( is_hvm_pv_evtchn_domain(v->domain) )
 vcpu_kick(v);
 else if ( v->vcpu_id == 0 )
 hvm_set_callback_irq_level(v);
diff --git a/xen/arch/x86/include/asm/domain.h 
b/xen/arch/x86/include/asm/domain.h
index 35898d725f..dcd221cc6f 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -14,9 +14,14 @@
 
 #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
 
+/*
+ * Set to true if either the global vector-type callback or per-vCPU
+ * LAPIC vectors are used. Assume all vCPUs will use
+ * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
+ */
 #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
-(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
-#define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
+((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
+ (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
 #define is_domain_direct_mapped(d) ((void)(d), 0)
 
 #define VCPU_TRAP_NONE 0
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 25bffe47d7..1a7f9df067 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1152,6 +1152,12 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
uint32_t leaf,
 res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
 res->c = d->domain_id;
 
+/*
+ * Per-vCPU event channel upcalls are implemented and work
+ * correctly with PIRQs routed over event channels.
+ */
+res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
+
 break;
 
 case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h 
b/xen/include/public/arch-x86/cpuid.h
index f2b2b3632c..c49eefeaf8 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -109,6 +109,11 @@
  * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/*
+ * Per-vCPU event channel upcalls work correctly with physical IRQs
+ * bound to event channels.
+ */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
-- 
2.11.0

Re: [PATCH v3] x86/hvm: Widen condition for is_hvm_pv_evtchn_domain() and report fix in CPUID

[Jane Malalane]

On 24/05/2022 16:14, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 18.05.2022 15:27, Jane Malalane wrote:
>> --- a/xen/arch/x86/include/asm/domain.h
>> +++ b/xen/arch/x86/include/asm/domain.h
>> @@ -14,8 +14,14 @@
>>   
>>   #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
>>   
>> +/*
>> + * Set to true if either the global vector-type callback or per-vCPU
>> + * LAPIC vectors are used. Assume all vCPUs will use
>> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
>> + */
>>   #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
>> -(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
>> +((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
>> + (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
>>   #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
> 
> I continue to think that with the vCPU0 dependency added to
> is_hvm_pv_evtchn_domain(), is_hvm_pv_evtchn_vcpu() should either
> be adjusted as well (to check the correct vCPU's field) or be
> deleted (and the sole caller be replaced).
> 
> Jan
I will replace it in a newer version of the patch.

Thank you.

Jane.

[PATCH v3] x86/hvm: Widen condition for is_hvm_pv_evtchn_domain() and report fix in CPUID

[Jane Malalane]

Have is_hvm_pv_evtchn_domain() return true for vector callbacks for
evtchn delivery set up on a per-vCPU basis via
HVMOP_set_evtchn_upcall_vector.

is_hvm_pv_evtchn_domain() returning true is a condition for setting up
physical IRQ to event channel mappings.

Therefore, a CPUID bit is added so that guests know whether the check
in is_hvm_pv_evtchn_domain() will fail when using
HVMOP_set_evtchn_upcall_vector. This matters for guests that route
PIRQs over event channels since is_hvm_pv_evtchn_domain() is a
condition in physdev_map_pirq().

The naming of the CPUID bit is quite generic about upcall support
being available. That's done so that the define name doesn't become
overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
such.

A guest that doesn't care about physical interrupts routed over event
channels can just test for the availability of the hypercall directly
(HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.

Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 

v3:
 * Improve commit message and title.

v2:
 * Since the naming of the CPUID bit is quite generic, better explain
   when it should be checked for, in code comments and commit message.
---
 xen/arch/x86/include/asm/domain.h   | 8 +++-
 xen/arch/x86/traps.c| 6 ++
 xen/include/public/arch-x86/cpuid.h | 5 +
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/include/asm/domain.h 
b/xen/arch/x86/include/asm/domain.h
index 35898d725f..f044e0a492 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -14,8 +14,14 @@
 
 #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
 
+/*
+ * Set to true if either the global vector-type callback or per-vCPU
+ * LAPIC vectors are used. Assume all vCPUs will use
+ * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
+ */
 #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
-(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
+((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
+ (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
 #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
 #define is_domain_direct_mapped(d) ((void)(d), 0)
 
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 25bffe47d7..1a7f9df067 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1152,6 +1152,12 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
uint32_t leaf,
 res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
 res->c = d->domain_id;
 
+/*
+ * Per-vCPU event channel upcalls are implemented and work
+ * correctly with PIRQs routed over event channels.
+ */
+res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
+
 break;
 
 case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h 
b/xen/include/public/arch-x86/cpuid.h
index f2b2b3632c..c49eefeaf8 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -109,6 +109,11 @@
  * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/*
+ * Per-vCPU event channel upcalls work correctly with physical IRQs
+ * bound to event channels.
+ */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
-- 
2.11.0

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 18/05/2022 10:07, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 11.05.2022 17:14, Jane Malalane wrote:
>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>> evtchn delivery set up on a per-vCPU basis via
>> HVMOP_set_evtchn_upcall_vector.
> 
> I'm confused: You say "per-vCPU" here, but ...
> 
>> --- a/xen/arch/x86/include/asm/domain.h
>> +++ b/xen/arch/x86/include/asm/domain.h
>> @@ -14,8 +14,14 @@
>>   
>>   #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
>>   
>> +/*
>> + * Set to true if either the global vector-type callback or per-vCPU
>> + * LAPIC vectors are used. Assume all vCPUs will use
>> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
>> + */
>>   #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
>> -(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
>> +((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
>> + (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
> 
> ... you use (d)->vcpu[0] here (and, yes, you say so in the comment)
> and ...
> 
>>   #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
> 
> ... you don't alter this at all.
> 
> Also (re-ordering context) this ...
> 
>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>> physical IRQ to event channel mappings.
> 
> ... isn't really true - it's is_hvm_pv_evtchn_domain() which controls
> this (which in turn is why above you need to make the assumption I've
> put under question). With that assumption I think is_hvm_pv_evtchn_vcpu()
> would better go away.
Here only is_hvm_pv_evtchn_domain() should have been mentioned. The 
"per-VCPU" was in regard to the vector callback for evthcn delivery 
setup not being global but now done on a per-vCPU basis, in any case, I 
will amend the description and title. Thanks for the feedback.

Jane.

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 18/05/2022 10:09, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 13.05.2022 17:39, Roger Pau Monné wrote:
>> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>>> evtchn delivery set up on a per-vCPU basis via
>>> HVMOP_set_evtchn_upcall_vector.
>>>
>>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>>> physical IRQ to event channel mappings.
>>
>> I would add something like:
>>
>> The naming of the CPUID bit is a bit generic about upcall support
>> being available.  That's done so that the define name doesn't get
>> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
>> such.
> 
> On top of this at least half a sentence wants saying on why a new
> CPUID bit is introduced in the first place. This doesn't derive in
> any way from title or description. It would be only then when it
> is additionally explained why the name was chosen like this.Indeed it is 
> incomplete, thanks for pointing that out.

I could add:
"A CPUID bit is added so that guests know whether the check
in is_hvm_pv_evtchn_domain() will fail when using
HVMOP_set_evtchn_upcall_vector. This matters for guests that route
PIRQs over event channels since is_hvm_pv_evtchn_domain() is a
condition in physdev_map_pirq()."

Would this be enough clarification?

Thank you,

Jane.

[PATCH v2] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
evtchn delivery set up on a per-vCPU basis via
HVMOP_set_evtchn_upcall_vector.

is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
physical IRQ to event channel mappings.

The naming of the CPUID bit is quite generic about upcall support
being available. That's done so that the define name doesn't become
overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
such.

A guest that doesn't care about physical interrupts routed over event
channels can just test for the availability of the hypercall directly
(HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.

Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 

v2:
 * Since the naming of the CPUID bit is quite generic, better explain
   when it should be checked for, in code comments and commit message.
---
 xen/arch/x86/include/asm/domain.h   | 8 +++-
 xen/arch/x86/traps.c| 6 ++
 xen/include/public/arch-x86/cpuid.h | 5 +
 3 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/include/asm/domain.h 
b/xen/arch/x86/include/asm/domain.h
index 35898d725f..f044e0a492 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -14,8 +14,14 @@
 
 #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
 
+/*
+ * Set to true if either the global vector-type callback or per-vCPU
+ * LAPIC vectors are used. Assume all vCPUs will use
+ * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
+ */
 #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
-(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
+((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
+ (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
 #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
 #define is_domain_direct_mapped(d) ((void)(d), 0)
 
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 25bffe47d7..1a7f9df067 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1152,6 +1152,12 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
uint32_t leaf,
 res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
 res->c = d->domain_id;
 
+/*
+ * Per-vCPU event channel upcalls are implemented and work
+ * correctly with PIRQs routed over event channels.
+ */
+res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
+
 break;
 
 case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h 
b/xen/include/public/arch-x86/cpuid.h
index f2b2b3632c..c49eefeaf8 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -109,6 +109,11 @@
  * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/*
+ * Per-vCPU event channel upcalls work correctly with physical IRQs
+ * bound to event channels.
+ */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
-- 
2.11.0

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 13/05/2022 16:39, Roger Pau Monné wrote:
> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>> evtchn delivery set up on a per-vCPU basis via
>> HVMOP_set_evtchn_upcall_vector.
>>
>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>> physical IRQ to event channel mappings.
> 
> I would add something like:
> 
> The naming of the CPUID bit is a bit generic about upcall support
> being available.  That's done so that the define name doesn't get
> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
> such.
> 
> Guests that don't care about physical interrupts routed over event
> channels can just test for the availability of the hypercall directly
> (HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.
> 
>>
>> Signed-off-by: Jane Malalane 
>> ---
>> CC: Jan Beulich 
>> CC: Andrew Cooper 
>> CC: "Roger Pau Monné" 
>> CC: Wei Liu 
>> ---
>>   xen/arch/x86/include/asm/domain.h   | 8 +++-
>>   xen/arch/x86/traps.c| 3 +++
>>   xen/include/public/arch-x86/cpuid.h | 2 ++
>>   3 files changed, 12 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/x86/include/asm/domain.h 
>> b/xen/arch/x86/include/asm/domain.h
>> index 35898d725f..f044e0a492 100644
>> --- a/xen/arch/x86/include/asm/domain.h
>> +++ b/xen/arch/x86/include/asm/domain.h
>> @@ -14,8 +14,14 @@
>>   
>>   #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
>>   
>> +/*
>> + * Set to true if either the global vector-type callback or per-vCPU
>> + * LAPIC vectors are used. Assume all vCPUs will use
> 
> I think you should remove LAPIC here.  There's no such thing as 'LAPIC
> vectors', it's just that the old mechanism was bypassing the lapic
> EOI.
> 
>> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
>> + */
>>   #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
>> -(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
>> +((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
>> + (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
>>   #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
>>   #define is_domain_direct_mapped(d) ((void)(d), 0)
>>   
>> diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
>> index 25bffe47d7..2c51faab2c 100644
>> --- a/xen/arch/x86/traps.c
>> +++ b/xen/arch/x86/traps.c
>> @@ -1152,6 +1152,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>   res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
>>   res->c = d->domain_id;
>>   
>> +/* Per-vCPU event channel upcalls are implemented. */
> 
> ... are implemented and work correctly with PIRQs routed over event
> channels.
> 
>> +res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
>> +
>>   break;
>>   
>>   case 5: /* PV-specific parameters */
>> diff --git a/xen/include/public/arch-x86/cpuid.h 
>> b/xen/include/public/arch-x86/cpuid.h
>> index f2b2b3632c..1760e2c405 100644
>> --- a/xen/include/public/arch-x86/cpuid.h
>> +++ b/xen/include/public/arch-x86/cpuid.h
>> @@ -109,6 +109,8 @@
>>* field from 8 to 15 bits, allowing to target APIC IDs up 32768.
>>*/
>>   #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
>> +/* Per-vCPU event channel upcalls. */
> 
> I would maybe expand the message to:
> 
> "Per-vCPU event channel upcalls work correctly with physical IRQs bound
> to event channels."

Thanks. Yes, if others agree that the CPUID bit can still take this name 
if better explained, despite it being, as you say, quite generic, I will 
add these comments in a v2.

Thanks,

Jane.

[PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
evtchn delivery set up on a per-vCPU basis via
HVMOP_set_evtchn_upcall_vector.

is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
physical IRQ to event channel mappings.

Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 
---
 xen/arch/x86/include/asm/domain.h   | 8 +++-
 xen/arch/x86/traps.c| 3 +++
 xen/include/public/arch-x86/cpuid.h | 2 ++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/include/asm/domain.h 
b/xen/arch/x86/include/asm/domain.h
index 35898d725f..f044e0a492 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -14,8 +14,14 @@
 
 #define has_32bit_shinfo(d)((d)->arch.has_32bit_shinfo)
 
+/*
+ * Set to true if either the global vector-type callback or per-vCPU
+ * LAPIC vectors are used. Assume all vCPUs will use
+ * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
+ */
 #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
-(d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
+((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
+ (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
 #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
 #define is_domain_direct_mapped(d) ((void)(d), 0)
 
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 25bffe47d7..2c51faab2c 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1152,6 +1152,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
uint32_t leaf,
 res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
 res->c = d->domain_id;
 
+/* Per-vCPU event channel upcalls are implemented. */
+res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
+
 break;
 
 case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h 
b/xen/include/public/arch-x86/cpuid.h
index f2b2b3632c..1760e2c405 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -109,6 +109,8 @@
  * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID  (1u << 5)
+/* Per-vCPU event channel upcalls. */
+#define XEN_HVM_CPUID_UPCALL_VECTOR(1u << 6)
 
 /*
  * Leaf 6 (0x4x05)
-- 
2.11.0

[PATCH v10 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note:

- vmx_install_vlapic_mapping doesn't require modifications regardless
of whether the guest has "Virtualize APIC accesses" enabled or not,
i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long as
virtualize_apic_accesses is supported by the CPU.

- Both per-domain and global assisted_x{2}apic options are not part of
the migration stream, unless explicitly set in the respective
configuration files. Default settings of assisted_x{2}apic done
internally by the toolstack, based on host capabilities at create
time, are not migrated.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Reviewed-by: "Roger Pau Monné" 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v10:
 * Improve commit message note on migration

v9:
 * Fix style issues
 * Fix exit() logic for assisted_x{2}apic parsing
 * Add and use XEN_X86_MISC_FLAGS_MAX for ABI checking instead of
   using XEN_X86_ASSISTED_X2APIC directly
 * Expand commit message to mention migration is safe

v8:
 * Widen assisted_x{2}apic parsing to PVH guests in
   parse_config_data()

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  9 ++---
 tools/libs/light/libxl_create

[PATCH v10 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
Reviewed-by: "Roger Pau Monné" 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v10:
 * Make assisted_x{2}apic_available conditional upon _vmx_cpu_up()

v9:
 * Move assisted_x{2}apic_available to vmx_vmcs_init() so they get
   declared at boot time, after vmx_secondary_exec_control is set

v8:
 * Improve commit message

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  7 +++
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 94 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gntt

[PATCH v10 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 15 ++
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 14 ++---
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 17 ---
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 19 +
 xen/arch/x86/domain.c | 29 +-
 xen/arch/x86/hvm/hvm.c|  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c   | 11 ++
 xen/arch/x86/hvm/vmx/vmx.c| 13 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h| 10 +
 xen/arch/x86/sysctl.c |  4 
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  5 +
 xen/include/public/sysctl.h   | 11 +-
 28 files changed, 281 insertions(+), 34 deletions(-)

-- 
2.11.0

Re: [PATCH v9 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 06/04/2022 14:44, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 01.04.2022 12:47, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted virtualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by decoding the
>> APIC access and providing a VM exit with a more specific exit reason
>> than a regular EPT fault or by altogether avoiding a VM exit.
>>
>> On the other hand, being able to disable x{2}APIC hardware assisted
>> virtualization can be useful for testing and debugging purposes.
>>
>> Note:
>>
>> - vmx_install_vlapic_mapping doesn't require modifications regardless
>> of whether the guest has "Virtualize APIC accesses" enabled or not,
>> i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long as
>> virtualize_apic_accesses is supported by the CPU.
>>
>> - Both per-domain and global assisted_x{2}apic options are not part of
>> the migration stream, unless explicitly set in the configuration file,
>> so it is safe to migrate a guest that doesn't have assisted_x{2}apic
>> set in its config file between hosts that have different support for
>> hardware assisted x{2}APIC virtualization.
> 
> While I can agree this is the case as far as your changes are concerned,
> it reads as if this was a broader statement. Since hypervisor CPUID leaf
> contents can change across such a migration (which is pre-existing
> behavior), I'm not convinced the statement is generally applicable.
I see, I will make this clearer to only reference the toolstack setting.

Thanks,

Jane.

[PATCH v9 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v9:
 * Move assisted_x{2}apic_available to vmx_vmcs_init() so they get
   declared at boot time, after vmx_secondary_exec_control is set

v8:
 * Improve commit message

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 93 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssiste

[PATCH v9 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 15 ++
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 14 ++---
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 17 ---
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 19 +
 xen/arch/x86/domain.c | 29 +-
 xen/arch/x86/hvm/hvm.c|  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c   | 10 +
 xen/arch/x86/hvm/vmx/vmx.c| 13 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h| 10 +
 xen/arch/x86/sysctl.c |  4 
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  5 +
 xen/include/public/sysctl.h   | 11 +-
 28 files changed, 280 insertions(+), 34 deletions(-)

-- 
2.11.0

[PATCH v9 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note:

- vmx_install_vlapic_mapping doesn't require modifications regardless
of whether the guest has "Virtualize APIC accesses" enabled or not,
i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long as
virtualize_apic_accesses is supported by the CPU.

- Both per-domain and global assisted_x{2}apic options are not part of
the migration stream, unless explicitly set in the configuration file,
so it is safe to migrate a guest that doesn't have assisted_x{2}apic
set in its config file between hosts that have different support for
hardware assisted x{2}APIC virtualization.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v9:
 * Fix style issues
 * Fix exit() logic for assisted_x{2}apic parsing
 * Add and use XEN_X86_MISC_FLAGS_MAX for ABI checking instead of
   using XEN_X86_ASSISTED_X2APIC directly
 * Expand commit message to mention migration is safe

v8:
 * Widen assisted_x{2}apic parsing to PVH guests in
   parse_config_data()

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  9 ++---
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/li

Re: [PATCH v8 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 23/03/2022 11:30, Roger Pau Monné wrote:
> On Wed, Mar 16, 2022 at 09:13:14AM +0000, Jane Malalane wrote:
>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
>> index e1e1fa14e6..77ce0b2121 100644
>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>   MSR_IA32_VMX_PROCBASED_CTLS2, );
>>   }
>>   
>> +/* Check whether hardware supports accelerated xapic and x2apic. */
>> +if ( bsp )
>> +{
>> +assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
>> +assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode &&
>> +(cpu_has_vmx_apic_reg_virt ||
>> + cpu_has_vmx_virtual_intr_delivery);
>> +}
> 
> I'm afraid using cpu_has_vmx_* is not correct here. The
> vmx_secondary_exec_control variable hasn't been set here, so you will
> need to move those checks to the end of the function, after
> vmx_secondary_exec_control has been set.

Sorry I missed that. As Jan previously suggested, maybe I could also 
just move this to vmx_vmcs_init() and thus drop the "if ( bsp )" ?

Thank you,

Jane.

[PATCH v8 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has "Virtualize APIC accesses" enabled
or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v8:
 * Widen assisted_x{2}apic parsing to PVH guests in
   parse_config_data()

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  9 ++---
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 28 ++--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 18 ++
 xen/arch/x86/domain.c | 29 -
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c

[PATCH v8 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xAPIC and
x2APIC, on x86 hardware. This is so that xAPIC and x2APIC virtualization
can subsequently be enabled on a per-domain basis.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v8:
 * Improve commit message

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/gola

[PATCH v8 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 15 ++
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 14 ++---
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 17 ---
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 18 
 xen/arch/x86/domain.c | 29 +-
 xen/arch/x86/hvm/hvm.c|  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c   | 13 
 xen/arch/x86/hvm/vmx/vmx.c| 13 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h| 10 +
 xen/arch/x86/sysctl.c |  4 
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  2 ++
 xen/include/public/sysctl.h   | 11 +-
 28 files changed, 279 insertions(+), 34 deletions(-)

-- 
2.11.0

[PATCH v7 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 15 ++
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 14 ++---
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 17 ---
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 16 ++
 xen/arch/x86/domain.c | 29 +-
 xen/arch/x86/hvm/hvm.c|  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c   | 13 
 xen/arch/x86/hvm/vmx/vmx.c| 13 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h| 10 +
 xen/arch/x86/sysctl.c |  4 
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  2 ++
 xen/include/public/sysctl.h   | 11 +-
 28 files changed, 277 insertions(+), 34 deletions(-)

-- 
2.11.0

[PATCH v7 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has "Virtualize APIC accesses" enabled
or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  9 ++---
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 28 ++--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 16 
 xen/arch/x86/domain.c | 29 -
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c| 13 -
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 

[PATCH v7 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight

Re: [PATCH v6 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 09/03/2022 10:36, Roger Pau Monné wrote:
> On Tue, Mar 08, 2022 at 05:31:17PM +0000, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
>> XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> HW assisted xAPIC virtualization will be reported if HW, at the
>> minimum, supports virtualize_apic_accesses as this feature alone means
>> that an access to the APIC page will cause an APIC-access VM exit. An
>> APIC-access VM exit provides a VMM with information about the access
>> causing the VM exit, unlike a regular EPT fault, thus simplifying some
>> internal handling.
>>
>> HW assisted x2APIC virtualization will be reported if HW supports
>> virtualize_x2apic_mode and, at least, either apic_reg_virt or
>> virtual_intr_delivery. This also means that
>> sysctl follows the conditionals in vmx_vlapic_msr_changed().
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Note that this interface is intended to be compatible with AMD so that
>> AVIC support can be introduced in a future patch. Unlike Intel that
>> has multiple controls for APIC Virtualization, AMD has one global
>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>> control cannot be done on a common interface.
>>
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
> 
> Overall LGTM, just one question and one nit.
> 
>> diff --git a/tools/ocaml/libs/xc/xenctrl_stubs.c 
>> b/tools/ocaml/libs/xc/xenctrl_stubs.c
>> index 5b4fe72c8d..7e9c32ad1b 100644
>> --- a/tools/ocaml/libs/xc/xenctrl_stubs.c
>> +++ b/tools/ocaml/libs/xc/xenctrl_stubs.c
>> @@ -712,7 +712,7 @@ CAMLprim value stub_xc_send_debug_keys(value xch, value 
>> keys)
>>   CAMLprim value stub_xc_physinfo(value xch)
>>   {
>>  CAMLparam1(xch);
>> -CAMLlocal2(physinfo, cap_list);
>> +CAMLlocal3(physinfo, cap_list, arch_cap_list);
>>  xc_physinfo_t c_physinfo;
>>  int r;
>>   
>> @@ -731,7 +731,7 @@ CAMLprim value stub_xc_physinfo(value xch)
>>  /* ! XEN_SYSCTL_PHYSCAP_ XEN_SYSCTL_PHYSCAP_MAX max */
>>  (c_physinfo.capabilities);
>>   
>> -physinfo = caml_alloc_tuple(10);
>> +physinfo = caml_alloc_tuple(11);
>>  Store_field(physinfo, 0, Val_int(c_physinfo.threads_per_core));
>>  Store_field(physinfo, 1, Val_int(c_physinfo.cores_per_socket));
>>  Store_field(physinfo, 2, Val_int(c_physinfo.nr_cpus));
>> @@ -743,6 +743,17 @@ CAMLprim value stub_xc_physinfo(value xch)
>>  Store_field(physinfo, 8, cap_list);
>>  Store_field(physinfo, 9, Val_int(c_physinfo.max_cpu_id + 1));
>>   
>> +#if defined(__i386__) || defined(__x86_64__)
>> +/*
>> + * arch_capabilities: physinfo_arch_cap_flag list;
>> + */
>> +arch_cap_list = c_bitmap_to_ocaml_list
>> +/* ! physinfo_arch_cap_flag CAP_ none */
>> +/* ! XEN_SYSCTL_PHYSCAP_ XEN_SYSCTL_PHYSCAP_X86_MAX max */
>> +(c_physinfo.arch_capabilities);
>> +Store_field(physinfo, 10, arch_cap_list);
>> +#endif
> 
> Have you tried to build this on Arm? I wonder whether the compiler
> will complain about arch_cap_list being unused there?
Built it - no error.

Thank you,

Jane.

[PATCH v7 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has "Virtualize APIC accesses" enabled
or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v7:
 * Fix void return in libxl__arch_domain_build_info_setdefault
 * Fix style issues
 * Use EINVAL when rejecting assisted_x{2}apic for PV guests and
   ENODEV otherwise, when assisted_x{2}apic isn't supported
 * Define has_assisted_x{2}apic macros for when !CONFIG_HVM
 * Replace "EPT" fault reference with "p2m" fault since the former is
   Intel-specific

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  9 ++---
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 28 ++--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 16 
 xen/arch/x86/domain.c | 29 -
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c| 13 -
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 

[PATCH v7 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v7:
 * Make sure assisted_x{2}apic_available evaluates to false, to ensure
   Xen builds, when !CONFIG_HVM
 * Fix coding style issues

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/hvm.c   |  3 +++
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/hvm/hvm.h   |  5 +
 xen/arch/x86/sysctl.c|  4 
 xen/include/public/sysctl.h  | 11 ++-
 17 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 09/03/2022 16:51, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 09.03.2022 16:56, Jane Malalane wrote:
>> On 08/03/2022 14:41, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 08.03.2022 15:31, Jane Malalane wrote:
>>>> On 08/03/2022 12:33, Roger Pau Monné wrote:
>>>>> On Tue, Mar 08, 2022 at 01:24:23PM +0100, Jan Beulich wrote:
>>>>>> On 08.03.2022 12:38, Roger Pau Monné wrote:
>>>>>>> On Mon, Mar 07, 2022 at 03:06:09PM +, Jane Malalane wrote:
>>>>>>>> @@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct 
>>>>>>>> xen_domctl_createdomain *config)
>>>>>>>> }
>>>>>>>> }
>>>>>>>> 
>>>>>>>> -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
>>>>>>>> +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED |
>>>>>>>> + XEN_X86_ASSISTED_XAPIC |
>>>>>>>> + XEN_X86_ASSISTED_X2APIC) )
>>>>>>>> {
>>>>>>>> dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
>>>>>>>> config->arch.misc_flags);
>>>>>>>> return -EINVAL;
>>>>>>>> }
>>>>>>>> 
>>>>>>>> +if ( (assisted_xapic || assisted_x2apic) && !hvm )
>>>>>>>> +{
>>>>>>>> +dprintk(XENLOG_INFO,
>>>>>>>> +"Interrupt Controller Virtualization not supported 
>>>>>>>> for PV\n");
>>>>>>>> +return -EINVAL;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> +if ( (assisted_xapic && !assisted_xapic_available) ||
>>>>>>>> + (assisted_x2apic && !assisted_x2apic_available) )
>>>>>>>> +{
>>>>>>>> +dprintk(XENLOG_INFO,
>>>>>>>> +"Hardware assisted x%sAPIC requested but not 
>>>>>>>> available\n",
>>>>>>>> +assisted_xapic && !assisted_xapic_available ? "" : 
>>>>>>>> "2");
>>>>>>>> +return -EINVAL;
>>>>>>>
>>>>>>> I think for those two you could return -ENODEV if others agree.
>>>>>>
>>>>>> If by "two" you mean the xAPIC and x2APIC aspects here (and not e.g. this
>>>>>> and the earlier if()), then I agree. I'm always in favor of using 
>>>>>> distinct
>>>>>> error codes when possible and at least halfway sensible.
>>>>>
>>>>> I would be fine by using it for the !hvm if also. IMO it makes sense
>>>>> as PV doesn't have an APIC 'device' at all, so ENODEV would seem
>>>>> fitting. EINVAL is also fine as the caller shouldn't even attempt that
>>>>> in the first place.
>>>>>
>>>>> So let's use it for the last if only.
>>>> Wouldn't it make more sense to use -ENODEV particularly for the first? I
>>>> agree that -ENODEV should be reported in the first case because it
>>>> doesn't make sense to request acceleration of something that doesn't
>>>> exist and I should have put that. But having a look at the hap code
>>>> (since it resembles the second case), it returns -EINVAL when it is not
>>>> available, unless you deem this to be different or, in retrospective,
>>>> that the hap code should too have been coded to return -ENODEV.
>>>>
>>>> if ( hap && !hvm_hap_supported() )
>>>>{
>>>>dprintk(XENLOG_INFO, "HAP requested but not available\n");
>>>>return -EINVAL;
>>>>}
>>>
>>> This is just one of the examples where using -ENODEV as you suggest
>>> would introduce an inconsistency. We use -EINVAL also for other
>>> purely guest-type dependent checks.
>>>
>>> Jan
>> Hi Jan, so here I was comparing the hap implementation with the second
>> case, i.e.
>>
>> if ( (assisted_xapic && !assisted_xapic_available) ||
>>(assisted_x2apic && !assisted_x2apic_available) )
>>
>> and you seem to agree that using -ENODEV would be inconsistent? Have I
>> misinterpreted this?
> 
> Not exactly. I'm comparing existing hap / hvm / !hap / !hvm uses with
> what you add.
> 
> Jan
> 
Okay, I wil swap the error codes then, thank you.

Jane.

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/03/2022 14:41, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 08.03.2022 15:31, Jane Malalane wrote:
>> On 08/03/2022 12:33, Roger Pau Monné wrote:
>>> On Tue, Mar 08, 2022 at 01:24:23PM +0100, Jan Beulich wrote:
>>>> On 08.03.2022 12:38, Roger Pau Monné wrote:
>>>>> On Mon, Mar 07, 2022 at 03:06:09PM +, Jane Malalane wrote:
>>>>>> @@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct 
>>>>>> xen_domctl_createdomain *config)
>>>>>>}
>>>>>>}
>>>>>>
>>>>>> -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
>>>>>> +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED |
>>>>>> + XEN_X86_ASSISTED_XAPIC |
>>>>>> + XEN_X86_ASSISTED_X2APIC) )
>>>>>>{
>>>>>>dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
>>>>>>config->arch.misc_flags);
>>>>>>return -EINVAL;
>>>>>>}
>>>>>>
>>>>>> +if ( (assisted_xapic || assisted_x2apic) && !hvm )
>>>>>> +{
>>>>>> +dprintk(XENLOG_INFO,
>>>>>> +"Interrupt Controller Virtualization not supported for 
>>>>>> PV\n");
>>>>>> +return -EINVAL;
>>>>>> +}
>>>>>> +
>>>>>> +if ( (assisted_xapic && !assisted_xapic_available) ||
>>>>>> + (assisted_x2apic && !assisted_x2apic_available) )
>>>>>> +{
>>>>>> +dprintk(XENLOG_INFO,
>>>>>> +"Hardware assisted x%sAPIC requested but not 
>>>>>> available\n",
>>>>>> +assisted_xapic && !assisted_xapic_available ? "" : "2");
>>>>>> +return -EINVAL;
>>>>>
>>>>> I think for those two you could return -ENODEV if others agree.
>>>>
>>>> If by "two" you mean the xAPIC and x2APIC aspects here (and not e.g. this
>>>> and the earlier if()), then I agree. I'm always in favor of using distinct
>>>> error codes when possible and at least halfway sensible.
>>>
>>> I would be fine by using it for the !hvm if also. IMO it makes sense
>>> as PV doesn't have an APIC 'device' at all, so ENODEV would seem
>>> fitting. EINVAL is also fine as the caller shouldn't even attempt that
>>> in the first place.
>>>
>>> So let's use it for the last if only.
>> Wouldn't it make more sense to use -ENODEV particularly for the first? I
>> agree that -ENODEV should be reported in the first case because it
>> doesn't make sense to request acceleration of something that doesn't
>> exist and I should have put that. But having a look at the hap code
>> (since it resembles the second case), it returns -EINVAL when it is not
>> available, unless you deem this to be different or, in retrospective,
>> that the hap code should too have been coded to return -ENODEV.
>>
>> if ( hap && !hvm_hap_supported() )
>>   {
>>   dprintk(XENLOG_INFO, "HAP requested but not available\n");
>>   return -EINVAL;
>>   }
> 
> This is just one of the examples where using -ENODEV as you suggest
> would introduce an inconsistency. We use -EINVAL also for other
> purely guest-type dependent checks.
> 
> Jan
Hi Jan, so here I was comparing the hap implementation with the second 
case, i.e.

if ( (assisted_xapic && !assisted_xapic_available) ||
  (assisted_x2apic && !assisted_x2apic_available) )

and you seem to agree that using -ENODEV would be inconsistent? Have I 
misinterpreted this?

Thanks,

Jane.

[PATCH v6 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has "Virtualize APIC accesses" enabled
or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v6:
 * Use ENODEV instead of EINVAL when rejecting assisted_x{2}apic
   for PV guests
 * Move has_assisted_x{2}apic macros out of an Intel specific header
 * Remove references to Intel specific features in documentation

v5:
 * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
   the has_assisted_x{2}apic macros
 * Following changes in assisted_x{2}apic_available definitions in
   patch 1, retighten conditionals for setting
   XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
   cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 15 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  7 +--
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 27 +--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 16 
 xen/arch/x86/domain.c | 28 +++-
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c| 13 -
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/include/asm/hvm/hvm.h|  6 ++
 xen/arch/x86/traps.c  |  5 +++--
 xen/include/public/arch-x86/xen.h |  2 ++
 23 files changed, 179 insertions(+), 28 deletions(-)

diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..b4239fcc5e 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1862,6 +1862,21 @@ firmwa

[PATCH v6 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC and
XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v6:
 * Limit abi check to x86
 * Fix coding style issue

v5:
 * Have assisted_xapic_available solely depend on
   cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
   depend on cpu_has_vmx_virtualize_x2apic_mode and
   cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 15 +--
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/domain.h|  3 +++
 xen/arch/x86/sysctl.c|  7 +++
 xen/include/public/sysctl.h  | 11 ++-
 16 files changed, 94 insertions(+), 5 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -1014,6 +1014,8 @@ CapVmtrace bool
 CapVpmu bool
 CapGnttabV1 bool
 CapGnttabV2 bool
+CapAssistedXapic bool
+CapAssistedX2Apic bool
 }
 
 type Connectorinfo struct 

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/03/2022 16:02, Roger Pau Monné wrote:
> On Tue, Mar 08, 2022 at 03:44:18PM +0000, Jane Malalane wrote:
>> On 08/03/2022 11:38, Roger Pau Monné wrote:
>>> On Mon, Mar 07, 2022 at 03:06:09PM +0000, Jane Malalane wrote:
>>>> diff --git a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h 
>>>> b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> index 9119aa8536..5b7d662ed7 100644
>>>> --- a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> +++ b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> @@ -220,6 +220,9 @@ void vmx_vmcs_reload(struct vcpu *v);
>>>>#define CPU_BASED_ACTIVATE_SECONDARY_CONTROLS 0x8000
>>>>extern u32 vmx_cpu_based_exec_control;
>>>>
>>>> +#define has_assisted_xapic(d)   ((d)->arch.hvm.assisted_xapic)
>>>> +#define has_assisted_x2apic(d)  ((d)->arch.hvm.assisted_x2apic)
>>>
>>> Those macros should not be in an Intel specific header,
>>> arch/x86/include/asm/hvm/domain.h is likely a better place.
>>
>> Actually do you think hvm.h could be better?
> 
> I guess that's also fine, I did suggest hvm/domain.h because that's
> where the fields get declared. I guess you prefer hvm.h because there
> are other HVM related helpers in there?

Yeah, that is why - tsc_scaling_ratio also gets defined in domain.h for e.g.

Thanks again,

Jane.

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/03/2022 16:02, Roger Pau Monné wrote:
> On Tue, Mar 08, 2022 at 03:44:18PM +0000, Jane Malalane wrote:
>> On 08/03/2022 11:38, Roger Pau Monné wrote:
>>> On Mon, Mar 07, 2022 at 03:06:09PM +0000, Jane Malalane wrote:
>>>> diff --git a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h 
>>>> b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> index 9119aa8536..5b7d662ed7 100644
>>>> --- a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> +++ b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h
>>>> @@ -220,6 +220,9 @@ void vmx_vmcs_reload(struct vcpu *v);
>>>>#define CPU_BASED_ACTIVATE_SECONDARY_CONTROLS 0x8000
>>>>extern u32 vmx_cpu_based_exec_control;
>>>>
>>>> +#define has_assisted_xapic(d)   ((d)->arch.hvm.assisted_xapic)
>>>> +#define has_assisted_x2apic(d)  ((d)->arch.hvm.assisted_x2apic)
>>>
>>> Those macros should not be in an Intel specific header,
>>> arch/x86/include/asm/hvm/domain.h is likely a better place.
>>
>> Actually do you think hvm.h could be better?
> 
> I guess that's also fine, I did suggest hvm/domain.h because that's
> where the fields get declared. I guess you prefer hvm.h because there
> are other HVM related helpers in there?

Yeah, that is why - tsc_scaling_ratio also gets defined in domain.h, for 
e.g.
Thank you for pointing this out,

Jane.

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/03/2022 11:38, Roger Pau Monné wrote:
> On Mon, Mar 07, 2022 at 03:06:09PM +0000, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted virtualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by decoding the
>> APIC access and providing a VM exit with a more specific exit reason
>> than a regular EPT fault or by altogether avoiding a VM exit.
>>
>> On the other hand, being able to disable x{2}APIC hardware assisted
>> virtualization can be useful for testing and debugging purposes.
>>
>> Note: vmx_install_vlapic_mapping doesn't require modifications
>> regardless of whether the guest has "Virtualize APIC accesses" enabled
>> or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
>> as virtualize_apic_accesses is supported by the CPU.
>>
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
>> ---
>> CC: Wei Liu 
>> CC: Anthony PERARD 
>> CC: Juergen Gross 
>> CC: Andrew Cooper 
>> CC: George Dunlap 
>> CC: Jan Beulich 
>> CC: Julien Grall 
>> CC: Stefano Stabellini 
>> CC: Christian Lindig 
>> CC: David Scott 
>> CC: Volodymyr Babchuk 
>> CC: "Roger Pau Monné" 
>>
>> v5:
>> * Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
>>the has_assisted_x{2}apic macros
>> * Following changes in assisted_x{2}apic_available definitions in
>>patch 1, retighten conditionals for setting
>>XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
>>cpuid_hypervisor_leaves()
>>
>> v4:
>>   * Add has_assisted_x{2}apic macros and use them where appropriate
>>   * Replace CPU checks with per-domain assisted_x{2}apic control
>> options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
>> following edits to assisted_x{2}apic_available definitions in
>> patch 1
>> Note: new assisted_x{2}apic_available definitions make later
>> cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
>> checks redundant in vmx_vlapic_msr_changed()
>>
>> v3:
>>   * Change info in xl.cfg to better express reality and fix
>> capitalization of x{2}apic
>>   * Move "physinfo" variable definition to the beggining of
>> libxl__domain_build_info_setdefault()
>>   * Reposition brackets in if statement to match libxl coding style
>>   * Shorten logic in libxl__arch_domain_build_info_setdefault()
>>   * Correct dprintk message in arch_sanitise_domain_config()
>>   * Make appropriate changes in vmx_vlapic_msr_changed() and
>> cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
>>   * Remove unneeded parantheses
>>
>> v2:
>>   * Add a LIBXL_HAVE_ASSISTED_APIC macro
>>   * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
>>   * Add a return statement in now "int"
>> libxl__arch_domain_build_info_setdefault()
>>   * Preserve libxl__arch_domain_build_info_setdefault 's location in
>> libxl_create.c
>>   * Correct x{2}apic default setting logic in
>> libxl__arch_domain_prepare_config()
>>   * Correct logic for parsing assisted_x{2}apic host/guest options in
>> xl_parse.c and initialize them to -1 in xl.c
>>   * Use guest options directly in vmx_vlapic_msr_changed
>>   * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
>>   * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
>> ---
>>   docs/man/xl.cfg.5.pod.in| 19 +++
>>   docs/man/xl.conf.5.pod.in   | 12 
>>   tools/golang/xenlight/helpers.gen.go| 12 
>>   tools/golang/xenlight/types.gen.go  |  2 ++
>>   tools/include/libxl.h   |  7 +++
>>   tools/libs/light/libxl_arch.h   |  5 +++--
>>   tools/libs/light/libxl_arm.c|  7 +--
>>   tools/libs/light/libxl_create.c | 22 +-
>>   tools/libs/light/libxl_types.idl|  2 ++
>>   tools/libs/light/libxl_x86.c| 28 ++--
>>   tools/ocaml/libs/xc/xenctrl.ml  |  2 ++
>>   tools/ocaml/libs/xc/xenctrl.mli |  2 ++
>>   tools/ocaml/libs/xc/xe

Re: [PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/03/2022 12:33, Roger Pau Monné wrote:
> On Tue, Mar 08, 2022 at 01:24:23PM +0100, Jan Beulich wrote:
>> On 08.03.2022 12:38, Roger Pau Monné wrote:
>>> On Mon, Mar 07, 2022 at 03:06:09PM +0000, Jane Malalane wrote:
>>>> @@ -685,13 +687,31 @@ int arch_sanitise_domain_config(struct 
>>>> xen_domctl_createdomain *config)
>>>>   }
>>>>   }
>>>>   
>>>> -if ( config->arch.misc_flags & ~XEN_X86_MSR_RELAXED )
>>>> +if ( config->arch.misc_flags & ~(XEN_X86_MSR_RELAXED |
>>>> + XEN_X86_ASSISTED_XAPIC |
>>>> + XEN_X86_ASSISTED_X2APIC) )
>>>>   {
>>>>   dprintk(XENLOG_INFO, "Invalid arch misc flags %#x\n",
>>>>   config->arch.misc_flags);
>>>>   return -EINVAL;
>>>>   }
>>>>   
>>>> +if ( (assisted_xapic || assisted_x2apic) && !hvm )
>>>> +{
>>>> +dprintk(XENLOG_INFO,
>>>> +"Interrupt Controller Virtualization not supported for 
>>>> PV\n");
>>>> +return -EINVAL;
>>>> +}
>>>> +
>>>> +if ( (assisted_xapic && !assisted_xapic_available) ||
>>>> + (assisted_x2apic && !assisted_x2apic_available) )
>>>> +{
>>>> +dprintk(XENLOG_INFO,
>>>> +"Hardware assisted x%sAPIC requested but not available\n",
>>>> +assisted_xapic && !assisted_xapic_available ? "" : "2");
>>>> +return -EINVAL;
>>>
>>> I think for those two you could return -ENODEV if others agree.
>>
>> If by "two" you mean the xAPIC and x2APIC aspects here (and not e.g. this
>> and the earlier if()), then I agree. I'm always in favor of using distinct
>> error codes when possible and at least halfway sensible.
> 
> I would be fine by using it for the !hvm if also. IMO it makes sense
> as PV doesn't have an APIC 'device' at all, so ENODEV would seem
> fitting. EINVAL is also fine as the caller shouldn't even attempt that
> in the first place.
> 
> So let's use it for the last if only.
Wouldn't it make more sense to use -ENODEV particularly for the first? I 
agree that -ENODEV should be reported in the first case because it 
doesn't make sense to request acceleration of something that doesn't 
exist and I should have put that. But having a look at the hap code 
(since it resembles the second case), it returns -EINVAL when it is not 
available, unless you deem this to be different or, in retrospective, 
that the hap code should too have been coded to return -ENODEV.

if ( hap && !hvm_hap_supported() )
 {
 dprintk(XENLOG_INFO, "HAP requested but not available\n");
 return -EINVAL;
 }

I agree with all the other comments and have made the approp changes for 
v6. Thanks a lot.

Jane.

[PATCH v5 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted virtualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
virtualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has "Virtualize APIC accesses" enabled
or not, i.e., setting the APIC_ACCESS_ADDR VMCS field is fine so long
as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v5:
* Revert v4 changes in vmx_vlapic_msr_changed(), preserving the use of
  the has_assisted_x{2}apic macros
* Following changes in assisted_x{2}apic_available definitions in
  patch 1, retighten conditionals for setting
  XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT in
  cpuid_hypervisor_leaves()

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in| 19 +++
 docs/man/xl.conf.5.pod.in   | 12 
 tools/golang/xenlight/helpers.gen.go| 12 
 tools/golang/xenlight/types.gen.go  |  2 ++
 tools/include/libxl.h   |  7 +++
 tools/libs/light/libxl_arch.h   |  5 +++--
 tools/libs/light/libxl_arm.c|  7 +--
 tools/libs/light/libxl_create.c | 22 +-
 tools/libs/light/libxl_types.idl|  2 ++
 tools/libs/light/libxl_x86.c| 28 ++--
 tools/ocaml/libs/xc/xenctrl.ml  |  2 ++
 tools/ocaml/libs/xc/xenctrl.mli |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c |  2 +-
 tools/xl/xl.c   |  8 
 tools/xl/xl.h   |  2 ++
 tools/xl/xl_parse.c | 16 
 xen/arch/x86/domain.c   | 28 +++-
 xen/arch/x86/hvm/vmx/vmcs.c |  4 
 xen/arch/x86/hvm/vmx/vmx.c  | 13 -
 xen/arch/x86/include/asm/hvm/domain.h   |  6 ++
 xen/arch/x86/include/asm/hvm/vmx/vmcs.h |  3 +++
 xen/arch/x86/traps.c|  9 +
 xen/include/public/arch-x86/xen.h   |  2 ++
 23 files changed, 183 insertions(+), 30 deletions(-)

diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..dcca564a23 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1862,6 +1862,25 @@ firmware tables when using certain older guest Operating
 Systems. These tables have been superseded by newer constructs within
 the ACPI tables.
 
+=item B
+
+B<(x86 only)&

[PATCH v5 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the
minimum, supports virtualize_apic_accesses as this feature alone means
that an access to the APIC page will cause an APIC-access VM exit. An
APIC-access VM exit provides a VMM with information about the access
causing the VM exit, unlike a regular EPT fault, thus simplifying some
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports
virtualize_x2apic_mode and, at least, either apic_reg_virt or
virtual_intr_delivery. This is due to apic_reg_virt and
virtual_intr_delivery preventing a VM exit from occuring or at least
replacing a regular EPT fault VM-exit with an APIC-access VM-exit on
read and write APIC accesses, respectively. This also means that
sysctl follows the conditionals in vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v5:
* Have assisted_xapic_available solely depend on
  cpu_has_vmx_virtualize_apic_accesses and assisted_x2apic_available
  depend on cpu_has_vmx_virtualize_x2apic_mode and
  cpu_has_vmx_apic_reg_virt OR cpu_has_vmx_virtual_intr_delivery.

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID.

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 14 +++---
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/domain.h|  3 +++
 xen/arch/x86/sysctl.c|  7 +++
 xen/include/public/sysctl.h  | 11 ++-
 16 files changed, 92 insertions(+), 6 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -1014,

[PATCH v5 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in| 19 
 docs/man/xl.conf.5.pod.in   | 12 ++
 tools/golang/xenlight/helpers.gen.go| 16 ++
 tools/golang/xenlight/types.gen.go  |  4 
 tools/include/libxl.h   | 14 
 tools/libs/light/libxl.c|  3 +++
 tools/libs/light/libxl_arch.h   |  9 ++--
 tools/libs/light/libxl_arm.c| 12 --
 tools/libs/light/libxl_create.c | 22 +++
 tools/libs/light/libxl_types.idl|  4 
 tools/libs/light/libxl_x86.c| 39 +++--
 tools/ocaml/libs/xc/xenctrl.ml  |  7 ++
 tools/ocaml/libs/xc/xenctrl.mli |  7 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c | 16 ++
 tools/xl/xl.c   |  8 +++
 tools/xl/xl.h   |  2 ++
 tools/xl/xl_info.c  |  6 +++--
 tools/xl/xl_parse.c | 16 ++
 xen/arch/x86/domain.c   | 28 ++-
 xen/arch/x86/hvm/vmx/vmcs.c | 13 +++
 xen/arch/x86/hvm/vmx/vmx.c  | 13 ---
 xen/arch/x86/include/asm/domain.h   |  3 +++
 xen/arch/x86/include/asm/hvm/domain.h   |  6 +
 xen/arch/x86/include/asm/hvm/vmx/vmcs.h |  3 +++
 xen/arch/x86/sysctl.c   |  7 ++
 xen/arch/x86/traps.c|  9 
 xen/include/public/arch-x86/xen.h   |  2 ++
 xen/include/public/sysctl.h | 11 +-
 28 files changed, 275 insertions(+), 36 deletions(-)

-- 
2.11.0

Re: [PATCH v4 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 07/03/2022 12:31, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 07.03.2022 13:17, Jane Malalane wrote:
>> On 04/03/2022 08:17, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 03.03.2022 17:37, Jane Malalane wrote:
>>>> On 03/03/2022 11:37, Jan Beulich wrote:
>>>>> On 02.03.2022 16:00, Jane Malalane wrote:
>>>>>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>>>>>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>>>>>> and x2apic, on x86 hardware.
>>>>>> No such features are currently implemented on AMD hardware.
>>>>>>
>>>>>> For that purpose, also add an arch-specific "capabilities" parameter
>>>>>> to struct xen_sysctl_physinfo.
>>>>>>
>>>>>> Note that this interface is intended to be compatible with AMD so that
>>>>>> AVIC support can be introduced in a future patch. Unlike Intel that
>>>>>> has multiple controls for APIC Virtualization, AMD has one global
>>>>>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>>>>>> control cannot be done on a common interface. Therefore, for xAPIC HW
>>>>>> assisted virtualization support to be reported, HW must support
>>>>>> virtualize_apic_accesses as well as apic_reg_virt.
>>>>>
>>>>> Okay, here you now describe _what_ is being implemented, but I'm
>>>>> afraid it still lacks justification (beyond making this re-usable for
>>>>> AVIC, which imo can only be a secondary goal). You actually say ...
>> Is the following any better...?
>>
>> "Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> HW assisted xAPIC virtualization will be reported if HW, at the minimum,
>>supports virtualize_apic_accesses as this feature alone means that an
>> access to the APIC page will cause an APIC-access VM exit. An
>> APIC-access VM exit provides a VMM with information about the access
>> causing the VM exit, unlike a regular EPT fault, thus simplifying some
>> internal handling.
>>
>> HW assisted x2APIC virtualization will be reported if HW supports
>> virtualize_x2apic_mode and, at least, either apic_reg_virt or
>> virtual_intr_delivery. This is due to apic_reg_virt and
>> virtual_intr_delivery preventing a VM exit from occuring or at least
>> replacing a regular EPT fault VM-exit with an APIC-access VM-exit on
>> read and write APIC accesses, respectively.
>> This also means that sysctl follows the conditionals in
>> vmx_vlapic_msr_changed().
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Note that this interface is intended to be compatible with AMD so that
>> AVIC support can be introduced in a future patch. Unlike Intel that
>> has multiple controls for APIC Virtualization, AMD has one global
>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>> control cannot be done on a common interface."
> 
> Yes, this looks quite a bit better, thanks. Assuming, of course, it's
> in sync with the code in v5 ...
Yes, ofc.

Just one thing, since vmx_vlapic_msr_changed() uses 
has_assisted_x{2}apic anyway do you think it's still necessary to add a 
comment pointing to this function in vmx_init_vmcs_config() when setting 
asissted_x{2}apic_available and v.v. ?

Thanks,

Jane.

Re: [PATCH v4 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 04/03/2022 08:17, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 03.03.2022 17:37, Jane Malalane wrote:
>> On 03/03/2022 11:37, Jan Beulich wrote:
>>> On 02.03.2022 16:00, Jane Malalane wrote:
>>>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>>>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>>>> and x2apic, on x86 hardware.
>>>> No such features are currently implemented on AMD hardware.
>>>>
>>>> For that purpose, also add an arch-specific "capabilities" parameter
>>>> to struct xen_sysctl_physinfo.
>>>>
>>>> Note that this interface is intended to be compatible with AMD so that
>>>> AVIC support can be introduced in a future patch. Unlike Intel that
>>>> has multiple controls for APIC Virtualization, AMD has one global
>>>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>>>> control cannot be done on a common interface. Therefore, for xAPIC HW
>>>> assisted virtualization support to be reported, HW must support
>>>> virtualize_apic_accesses as well as apic_reg_virt.
>>>
>>> Okay, here you now describe _what_ is being implemented, but I'm
>>> afraid it still lacks justification (beyond making this re-usable for
>>> AVIC, which imo can only be a secondary goal). You actually say ...
Is the following any better...?

"Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

HW assisted xAPIC virtualization will be reported if HW, at the minimum, 
  supports virtualize_apic_accesses as this feature alone means that an 
access to the APIC page will cause an APIC-access VM exit. An 
APIC-access VM exit provides a VMM with information about the access 
causing the VM exit, unlike a regular EPT fault, thus simplifying some 
internal handling.

HW assisted x2APIC virtualization will be reported if HW supports 
virtualize_x2apic_mode and, at least, either apic_reg_virt or 
virtual_intr_delivery. This is due to apic_reg_virt and 
virtual_intr_delivery preventing a VM exit from occuring or at least 
replacing a regular EPT fault VM-exit with an APIC-access VM-exit on 
read and write APIC accesses, respectively.
This also means that sysctl follows the conditionals in 
vmx_vlapic_msr_changed().

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface."

I previously didn't add here any info about the assistance that each CPU 
bit provides to avoid repitition, as I talk about that in patch 2, but I 
interpreted from your comment that it might be helpful to add that here 
too.
>>>
>>>> For x2APIC HW
>>>> assisted virtualization reporting, virtualize_x2apic_mode must be
>>>> supported alongside apic_reg_virt and virtual_intr_delivery.
>>>>
>>>> Suggested-by: Andrew Cooper 
>>>> Signed-off-by: Jane Malalane 
>>>>
>>>> v4:
>>>>* Fallback to the original v2/v1 conditions for setting
>>>>  assisted_xapic_available and assisted_x2apic_available so that in
>>>>  the future APIC virtualization can be exposed on AMD hardware
>>>>  since fine-graining of "AVIC" is not supported, i.e., AMD solely
>>>>  uses "AVIC Enable". This also means that sysctl mimics what's
>>>>  exposed in CPUID.
>>>
>>> ... more here: You claim similarity with CPUID. That's a possible route,
>>> but we need to be clear that these CPUID flags are optimization hints
>>> for the guest to use, while the new control is intended to be a functional
>>> one. Hence it's not obvious that CPUID wants following, and not instead
>>> the conditionals used in vmx_vlapic_msr_changed() (or yet something else).
>>>
>>> What's worse though: What you say is true for x2APIC, but not for xAPIC.
>>> Which effectively is in line with vmx_vlapic_msr_changed() and CPUID
>>> handling also agreeing as far as x2APIC is concerned, but disagreeing on
>>> the xAPIC side. I can only once again try to express that it may well be
>&g

Re: [PATCH v4 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 03/03/2022 11:37, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 02.03.2022 16:00, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Note that this interface is intended to be compatible with AMD so that
>> AVIC support can be introduced in a future patch. Unlike Intel that
>> has multiple controls for APIC Virtualization, AMD has one global
>> 'AVIC Enable' control bit, so fine-graining of APIC virtualization
>> control cannot be done on a common interface. Therefore, for xAPIC HW
>> assisted virtualization support to be reported, HW must support
>> virtualize_apic_accesses as well as apic_reg_virt.
> 
> Okay, here you now describe _what_ is being implemented, but I'm
> afraid it still lacks justification (beyond making this re-usable for
> AVIC, which imo can only be a secondary goal). You actually say ...
> 
>> For x2APIC HW
>> assisted virtualization reporting, virtualize_x2apic_mode must be
>> supported alongside apic_reg_virt and virtual_intr_delivery.
>>
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
>>
>> v4:
>>   * Fallback to the original v2/v1 conditions for setting
>> assisted_xapic_available and assisted_x2apic_available so that in
>> the future APIC virtualization can be exposed on AMD hardware
>> since fine-graining of "AVIC" is not supported, i.e., AMD solely
>> uses "AVIC Enable". This also means that sysctl mimics what's
>> exposed in CPUID.
> 
> ... more here: You claim similarity with CPUID. That's a possible route,
> but we need to be clear that these CPUID flags are optimization hints
> for the guest to use, while the new control is intended to be a functional
> one. Hence it's not obvious that CPUID wants following, and not instead
> the conditionals used in vmx_vlapic_msr_changed() (or yet something else).
> 
> What's worse though: What you say is true for x2APIC, but not for xAPIC.
> Which effectively is in line with vmx_vlapic_msr_changed() and CPUID
> handling also agreeing as far as x2APIC is concerned, but disagreeing on
> the xAPIC side. I can only once again try to express that it may well be
> that pre-existing code wants adjusting before actually making the changes
> you're after.


I've been thinking about this. Considering what you say, I propose:

- having assisted_x2apic_available = cpu_has_vmx_virtualize_x2apic_mode 
&& (cpu_has_vmx_apic_reg_virt || cpu_has_vmx_virtual_intr_delivery). 
This would mean that on Intel CPUs has_assisted_x2apic==1 would signify 
that there is at least "some" assistance*, whereas on AMD it would 
signify that there is full assistance (assistance here meaning no VM-exits).
* apic_reg_virt prevents VM exits on execution of RDMSR and 
virtual_intr_delivery prevents VM exits on execution of RDMSR, from what 
I've gathered.
- having assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses 
&& cpu_has_vmx_apic_reg_virt because apic_reg_virt is neccessary for 
"any" assistance.

- Currently, the code only sets SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE if 
"some" assistance is guaranteed but sets 
SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES even if no assistance is 
guaranteed. So the adjustment to the pre-existing code that I propose is
adding cpu_has_vmx_apic_reg_virt to the initial check in 
vmx_vlapic_msr_changed():

  void vmx_vlapic_msr_changed(struct vcpu *v)
  {
  int virtualize_x2apic_mode;
  struct vlapic *vlapic = vcpu_vlapic(v);
  unsigned int msr;

  virtualize_x2apic_mode = ((cpu_has_vmx_apic_reg_virt ||
 cpu_has_vmx_virtual_intr_delivery) &&
cpu_has_vmx_virtualize_x2apic_mode);

  if ( !cpu_has_vmx_virtualize_apic_accesses &&
+ !cpu_has_vmx_apic_reg_virt &&
   !virtualize_x2apic_mode )
  return;


which would then eventually just be what I currently have:
+if ( !has_assisted_xapic(v->domain) &&
+ !has_assisted_x2apic(v->domain) )
  return;

So, essentially, the only difference from v4 would be 
assisted_x2apic_available = (cpu_has_vmx_virtualize_x2apic_mode &&
 (cpu_has_vmx_apic_reg_virt ||
  cpu_has_vmx_virtual_intr_delivery));  

sysctl would now coincide with CPUID for xAPIC but not x2APIC (for CPUID 
the condition is the AND of all features unlike the 
assisted_x2apic_available proposed). IOW, it would follow the 
conditionals used in vmx_vlapic_msr_changed(), if we take the change to 
vmx_vlapic_msr_changed() above.

Thank you,

Jane.

[PATCH v4 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in| 19 
 docs/man/xl.conf.5.pod.in   | 12 ++
 tools/golang/xenlight/helpers.gen.go| 16 +
 tools/golang/xenlight/types.gen.go  |  4 
 tools/include/libxl.h   | 14 
 tools/libs/light/libxl.c|  3 +++
 tools/libs/light/libxl_arch.h   |  9 ++--
 tools/libs/light/libxl_arm.c| 12 --
 tools/libs/light/libxl_create.c | 22 ++
 tools/libs/light/libxl_types.idl|  4 
 tools/libs/light/libxl_x86.c| 39 ++--
 tools/ocaml/libs/xc/xenctrl.ml  |  7 ++
 tools/ocaml/libs/xc/xenctrl.mli |  7 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c | 16 +
 tools/xl/xl.c   |  8 +++
 tools/xl/xl.h   |  2 ++
 tools/xl/xl_info.c  |  6 +++--
 tools/xl/xl_parse.c | 16 +
 xen/arch/x86/domain.c   | 28 ++-
 xen/arch/x86/hvm/vmx/vmcs.c | 14 
 xen/arch/x86/hvm/vmx/vmx.c  | 40 +
 xen/arch/x86/include/asm/domain.h   |  3 +++
 xen/arch/x86/include/asm/hvm/domain.h   |  6 +
 xen/arch/x86/include/asm/hvm/vmx/vmcs.h |  3 +++
 xen/arch/x86/sysctl.c   |  7 ++
 xen/arch/x86/traps.c|  6 ++---
 xen/include/public/arch-x86/xen.h   |  2 ++
 xen/include/public/sysctl.h | 11 -
 28 files changed, 285 insertions(+), 51 deletions(-)

-- 
2.11.0

[PATCH v4 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Note that this interface is intended to be compatible with AMD so that
AVIC support can be introduced in a future patch. Unlike Intel that
has multiple controls for APIC Virtualization, AMD has one global
'AVIC Enable' control bit, so fine-graining of APIC virtualization
control cannot be done on a common interface. Therefore, for xAPIC HW
assisted virtualization support to be reported, HW must support
virtualize_apic_accesses as well as apic_reg_virt. For x2APIC HW
assisted virtualization reporting, virtualize_x2apic_mode must be
supported alongside apic_reg_virt and virtual_intr_delivery.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v4:
 * Fallback to the original v2/v1 conditions for setting
   assisted_xapic_available and assisted_x2apic_available so that in
   the future APIC virtualization can be exposed on AMD hardware
   since fine-graining of "AVIC" is not supported, i.e., AMD solely
   uses "AVIC Enable". This also means that sysctl mimics what's
   exposed in CPUID.

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set "arch_capbilities", via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   _X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 14 +++---
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/vmx/vmcs.c  | 10 ++
 xen/arch/x86/include/asm/domain.h|  3 +++
 xen/arch/x86/sysctl.c|  7 +++
 xen/include/public/sysctl.h  | 11 ++-
 16 files changed, 93 insertions(+), 6 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -1014,6 +1014,8 @@ CapVmtrace bool
 CapVpmu bool
 CapGnttabV1 bool
 CapGnttabV2 bool
+CapAssistedXapic bool
+CapAssistedX2Apic bool
 }
 
 type Connectorinfo struct {
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 51a9b6cfac..94e6355822 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -528,6 +528,13 @@
 #define LIBXL_HAVE_MAX_GRANT_VERSION 1
 
 /*
+ * LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has
+ * cap_assisted_xapic and cap_assisted_x2apic fields, which indicates
+ * the availability of x{2}APIC hardware assisted virtualization.
+ */
+#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
diff --git a/tools/li

[PATCH v4 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted vitualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
vitualization can be useful for testing and debugging purposes.

Note:

- vmx_install_vlapic_mapping doesn't require modifications regardless
  of whether the guest has "Virtualize APIC accesses" enabled or not,
  i.e., setting the the APIC_ACCESS_ADDR VMCS field is fine so long as
  virtualize_apic_accesses is supported by the CPU.

- XEN_HVM_CPUID_XAPIC_VIRT will now also depend on HW supporting
  virtualize_apic_accesses but this addition has no effect since
  apic_reg_virt support, which is currently a requirement for setting
  this CPUID bit, without virtualize_apic_accesses has no effect on
  xAPIC accesses anyway.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v4:
 * Add has_assisted_x{2}apic macros and use them where appropriate
 * Replace CPU checks with per-domain assisted_x{2}apic control
   options in vmx_vlapic_msr_changed() and cpuid_hypervisor_leaves(),
   following edits to assisted_x{2}apic_available definitions in
   patch 1
   Note: new assisted_x{2}apic_available definitions make later
   cpu_has_vmx_apic_reg_virt and cpu_has_vmx_virtual_intr_delivery
   checks redundant in vmx_vlapic_msr_changed()

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in| 19 
 docs/man/xl.conf.5.pod.in   | 12 ++
 tools/golang/xenlight/helpers.gen.go| 12 ++
 tools/golang/xenlight/types.gen.go  |  2 ++
 tools/include/libxl.h   |  7 ++
 tools/libs/light/libxl_arch.h   |  5 +++--
 tools/libs/light/libxl_arm.c|  7 --
 tools/libs/light/libxl_create.c | 22 +++
 tools/libs/light/libxl_types.idl|  2 ++
 tools/libs/light/libxl_x86.c| 28 +--
 tools/ocaml/libs/xc/xenctrl.ml  |  2 ++
 tools/ocaml/libs/xc/xenctrl.mli |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c |  2 +-
 tools/xl/xl.c   |  8 +++
 tools/xl/xl.h   |  2 ++
 tools/xl/xl_parse.c | 16 ++
 xen/arch/x86/domain.c   | 28 ++-
 xen/arch/x86/hvm/vmx/vmcs.c |  4 
 xen/arch/x86/hvm/vmx/vmx.c  | 39 +
 xen/arch/x86/include/asm/hvm/domain.h   |  6 +
 xen/arch/x86/include/asm/hvm/vmx/vmcs.h |  3 +++
 xen/arch/x86/traps.c|  6 ++---
 xen/include/public/arch-x86/xen.h   |  2 ++
 23 files changed, 191 insertions(+), 45 deletions(-)

diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..dcca564a23 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1862,6 +1862,25 @@ firmware tables when using certain older guest Operating
 Systems. These tables have been superseded by newer constructs within
 the ACPI tables.
 
+=item B
+
+B<(x86 only)> Enables or disables hardw

Re: [PATCH v3 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 28/02/2022 07:32, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 25.02.2022 17:02, Jane Malalane wrote:
>> On 24/02/2022 14:08, Jan Beulich wrote:
>>> On 18.02.2022 18:29, Jane Malalane wrote:
>>>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>>>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>>>> and x2apic, on x86 hardware.
>>>> No such features are currently implemented on AMD hardware.
>>>>
>>>> For that purpose, also add an arch-specific "capabilities" parameter
>>>> to struct xen_sysctl_physinfo.
>>>>
>>>> Suggested-by: Andrew Cooper 
>>>> Signed-off-by: Jane Malalane 
>>>> ---
>>>> v3:
>>>>* Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
>>>>  set arch_capbilities, via a call to c_bitmap_to_ocaml_list()
>>>>* Have assisted_x2apic_available only depend on
>>>>  cpu_has_vmx_virtualize_x2apic_mode
>>>
>>> I understand this was the result from previous discussion, but this
>>> needs justifying in the description. Not the least because it differs
>>> from when XEN_HVM_CPUID_X2APIC_VIRT would be set as well as from what
>>> vmx_vlapic_msr_changed() does. The difference between those two is
>>> probably intended (judging from a comment there), but the further
>>> difference to what you add isn't obvious.
>>
>> Okay, I will make that explicit.
>>
>>> Which raises another thought: If that hypervisor leaf was part of the
>>> HVM feature set, the tool stack could be able to obtain the wanted
>>> information without altering sysctl (assuming the conditions to set
>>> the respective bits were the same). And I would view it as generally
>>> reasonable for there to be a way for tool stacks to know what
>>> hypervisor leaves guests are going to get to see (at the maximum and
>>> by default).
>>
>> Like the "cpuid" xtf test allows us to?
> 
> I don't think I understand the question. That xtf test is concerned
> about checking the CPUID output it gets to see itself. It doesn't care
> about what other guests might get to see, nor the maximum and default.
> 
>> Makes sense to me. I'm happy to take that up after.
> 
> "After" what?
So I meant to say that I could add the Xen CPUID leaves (4x...) to 
the policy so that toolstacks could know what hypervisor leaves guests 
are going to see - in a future patch, as this wouldn't just expose 
XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT 
(0x4x04) but other features too.

But, at the same time, w.r.t. this patch in particular, using 
XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT to detect 
assisted APIC gives us less flexibility to add more fine grained 
controls in the future.

Thanks,

Jane.

Re: [PATCH v3 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 28/02/2022 07:32, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 25.02.2022 17:02, Jane Malalane wrote:
>> On 24/02/2022 14:08, Jan Beulich wrote:
>>> On 18.02.2022 18:29, Jane Malalane wrote:
>>>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>>>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>>>> and x2apic, on x86 hardware.
>>>> No such features are currently implemented on AMD hardware.
>>>>
>>>> For that purpose, also add an arch-specific "capabilities" parameter
>>>> to struct xen_sysctl_physinfo.
>>>>
>>>> Suggested-by: Andrew Cooper 
>>>> Signed-off-by: Jane Malalane 
>>>> ---
>>>> v3:
>>>>* Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
>>>>  set arch_capbilities, via a call to c_bitmap_to_ocaml_list()
>>>>* Have assisted_x2apic_available only depend on
>>>>  cpu_has_vmx_virtualize_x2apic_mode
>>>
>>> I understand this was the result from previous discussion, but this
>>> needs justifying in the description. Not the least because it differs
>>> from when XEN_HVM_CPUID_X2APIC_VIRT would be set as well as from what
>>> vmx_vlapic_msr_changed() does. The difference between those two is
>>> probably intended (judging from a comment there), but the further
>>> difference to what you add isn't obvious.
>>
>> Okay, I will make that explicit.
>>
>>> Which raises another thought: If that hypervisor leaf was part of the
>>> HVM feature set, the tool stack could be able to obtain the wanted
>>> information without altering sysctl (assuming the conditions to set
>>> the respective bits were the same). And I would view it as generally
>>> reasonable for there to be a way for tool stacks to know what
>>> hypervisor leaves guests are going to get to see (at the maximum and
>>> by default).
>>
>> Like the "cpuid" xtf test allows us to?
> 
> I don't think I understand the question. That xtf test is concerned
> about checking the CPUID output it gets to see itself. It doesn't care
> about what other guests might get to see, nor the maximum and default.
> 
>> Makes sense to me. I'm happy to take that up after.
> 
> "After" what?
So I meant to say that I could add the Xen CPUID leaves (4x...) to 
the policy so that toolstacks could know what hypervisor leaves guests 
are going to see - in a future patch, as this wouldn't just expose 
XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT 
(0x4x04) but other features too.

But, at the same time, w.r.t. this patch in particular, using 
XEN_HVM_CPUID_APIC_ACCESS_VIRT and XEN_HVM_CPUID_X2APIC_VIRT to detect 
assisted APIC gives us less flexibility to add more fine grained 
controls in the future.

Thanks,

Jane.

Re: [PATCH v3 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 24/02/2022 14:08, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 18.02.2022 18:29, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Suggested-by: Andrew Cooper 
>> Signed-off-by: Jane Malalane 
>> ---
>> v3:
>>   * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
>> set arch_capbilities, via a call to c_bitmap_to_ocaml_list()
>>   * Have assisted_x2apic_available only depend on
>> cpu_has_vmx_virtualize_x2apic_mode
> 
> I understand this was the result from previous discussion, but this
> needs justifying in the description. Not the least because it differs
> from when XEN_HVM_CPUID_X2APIC_VIRT would be set as well as from what
> vmx_vlapic_msr_changed() does. The difference between those two is
> probably intended (judging from a comment there), but the further
> difference to what you add isn't obvious.

Okay, I will make that explicit.

> Which raises another thought: If that hypervisor leaf was part of the
> HVM feature set, the tool stack could be able to obtain the wanted
> information without altering sysctl (assuming the conditions to set
> the respective bits were the same). And I would view it as generally
> reasonable for there to be a way for tool stacks to know what
> hypervisor leaves guests are going to get to see (at the maximum and
> by default).

Like the "cpuid" xtf test allows us to?
Makes sense to me. I'm happy to take that up after.

> 
>> --- a/xen/include/public/sysctl.h
>> +++ b/xen/include/public/sysctl.h
>> @@ -35,7 +35,7 @@
>>   #include "domctl.h"
>>   #include "physdev.h"
>>   
>> -#define XEN_SYSCTL_INTERFACE_VERSION 0x0014
>> +#define XEN_SYSCTL_INTERFACE_VERSION 0x0015
>>   
>>   /*
>>* Read console content from Xen buffer ring.
>> @@ -111,6 +111,13 @@ struct xen_sysctl_tbuf_op {
>>   /* Max XEN_SYSCTL_PHYSCAP_* constant.  Used for ABI checking. */
>>   #define XEN_SYSCTL_PHYSCAP_MAX XEN_SYSCTL_PHYSCAP_gnttab_v2
>>   
>> +/* The platform supports x{2}apic hardware assisted emulation. */
>> +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_XAPIC  (1u << 0)
>> +#define XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC (1u << 1)
>> +
>> +/* Max XEN_SYSCTL_PHYSCAP_X86{ARM}__* constant. Used for ABI checking. */
>> +#define XEN_SYSCTL_PHYSCAP_ARCH_MAX XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X2APIC
> 
> Doesn't this then need to be a per-arch constant? The ABIs would differ
> unless we required that every bit may only be used for a single purpose.
> IOW it would want to be named XEN_SYSCTL_PHYSCAP_X86_MAX.

Okay.

> 
>> @@ -120,6 +127,8 @@ struct xen_sysctl_physinfo {
>>   uint32_t max_node_id; /* Largest possible node ID on this host */
>>   uint32_t cpu_khz;
>>   uint32_t capabilities;/* XEN_SYSCTL_PHYSCAP_??? */
>> +uint32_t arch_capabilities;/* XEN_SYSCTL_PHYSCAP_X86{ARM}_??? */
>> +uint32_t pad; /* Must be zero. */
> 
> If this was an input field (or could potentially become one), the
> comment would be applicable. But the whole struct is OUT-only, so
> either omit the comment or use e.g. "will" or better "reserved" (as
> people shouldn't make themselves dependent on the field being zero).

Will ommit.

Thank you,

Jane.

Re: [PATCH v3 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 25/02/2022 13:13, Anthony PERARD wrote:
> On Fri, Feb 18, 2022 at 05:29:43PM +0000, Jane Malalane wrote:
>> diff --git a/tools/include/libxl.h b/tools/include/libxl.h
>> index 333ffad38d..1c83cae711 100644
>> --- a/tools/include/libxl.h
>> +++ b/tools/include/libxl.h
>> @@ -535,6 +535,13 @@
>>   #define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1
>>   
>>   /*
>> + * LIBXL_HAVE_ASSISTED_APIC indicates that libxl_domain_build_info has
>> + * assisted_x{2}apic fields, for enabling hardware assisted virtualization 
>> for
> 
> Could you spell out both "assisted_xapic and assisted_x2apic" as that
> would allow for grep to find both string.
Will do (for both cases).
> 
>> + * x{2}apic per domain.
>> + */
>> +#define LIBXL_HAVE_ASSISTED_APIC 1
>> +
>> +/*
>> diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c
>> index 39fdca1b49..ba5b8f433f 100644
>> --- a/tools/libs/light/libxl_arm.c
>> +++ b/tools/libs/light/libxl_arm.c
>> @@ -1384,8 +1384,9 @@ void 
>> libxl__arch_domain_create_info_setdefault(libxl__gc *gc,
>>   }
>>   }
>>   
>> -void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>> -  libxl_domain_build_info 
>> *b_info)
>> +int libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>> + libxl_domain_build_info 
>> *b_info,
>> + const libxl_physinfo *physinfo)
>>   {
>>   /* ACPI is disabled by default */
>>   libxl_defbool_setdefault(_info->acpi, false);
>> @@ -1399,6 +1400,8 @@ void 
>> libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
> 
> There is another return in this function, which want to return 0 rather
> than void.
> 
>>   memset(_info->u, '\0', sizeof(b_info->u));
>>   b_info->type = LIBXL_DOMAIN_TYPE_INVALID;
>>   libxl_domain_build_info_init_type(b_info, LIBXL_DOMAIN_TYPE_PVH);
>> +
>> +return 0;
>>   }
>>   
>>   int libxl__arch_passthrough_mode_setdefault(libxl__gc *gc,
>> diff --git a/tools/libs/light/libxl_x86.c b/tools/libs/light/libxl_x86.c
>> index e0a06ecfe3..c377d13b19 100644
>> --- a/tools/libs/light/libxl_x86.c
>> +++ b/tools/libs/light/libxl_x86.c
>> @@ -819,11 +827,27 @@ void 
>> libxl__arch_domain_create_info_setdefault(libxl__gc *gc,
>>   {
>>   }
>>   
>> -void libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>> -  libxl_domain_build_info 
>> *b_info)
>> +int libxl__arch_domain_build_info_setdefault(libxl__gc *gc,
>> + libxl_domain_build_info 
>> *b_info,
>> + const libxl_physinfo *physinfo)
>>   {
>>   libxl_defbool_setdefault(_info->acpi, true);
>>   libxl_defbool_setdefault(_info->arch_x86.msr_relaxed, false);
>> +
>> +if (b_info->type != LIBXL_DOMAIN_TYPE_PV) {
>> +libxl_defbool_setdefault(_info->arch_x86.assisted_xapic,
>> + physinfo->cap_assisted_xapic);
>> +libxl_defbool_setdefault(_info->arch_x86.assisted_x2apic,
>> + physinfo->cap_assisted_x2apic);
>> +}
>> +
>> +else if (!libxl_defbool_is_default(b_info->arch_x86.assisted_xapic) ||
> 
> This "else" needs to be on the same line as the "}" 2 lines above.
Okay.

Thank you,

Jane.

Re: [PATCH v3 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 24/02/2022 17:04, Jan Beulich wrote:
> On 24.02.2022 17:59, Jane Malalane wrote:
>> On 24/02/2022 14:16, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 18.02.2022 18:29, Jane Malalane wrote:
>>>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>>>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>>>> @@ -,15 +,15 @@ static void vmx_install_vlapic_mapping(struct vcpu 
>>>> *v)
>>>>
>>>>void vmx_vlapic_msr_changed(struct vcpu *v)
>>>>{
>>>> -int virtualize_x2apic_mode;
>>>> +bool virtualize_x2apic_mode;
>>>>struct vlapic *vlapic = vcpu_vlapic(v);
>>>>unsigned int msr;
>>>>
>>>>virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt ||
>>>>cpu_has_vmx_virtual_intr_delivery) &&
>>>> -   cpu_has_vmx_virtualize_x2apic_mode );
>>>> +   v->domain->arch.hvm.assisted_x2apic );
>>>
>>> Following from my comment on patch 1, I'd expect this to become a simple
>>> assignment of v->domain->arch.hvm.assisted_x2apic (at which point the
>>> local variable could go away), just like ...
>>>
>>>> -if ( !cpu_has_vmx_virtualize_apic_accesses &&
>>>> +if ( !v->domain->arch.hvm.assisted_xapic &&
>>>> !virtualize_x2apic_mode )
>>>>return;
>>>
>>> ... here.
>> Previosuly we discussed setting v->domain->arch.hvm.assisted_xapic equal
>> to only cpu_has_vmx_virtualize_x2apic_mode, that's why I have those
>> additional checks as at least apic_reg_virt or virtual_intr_delivery are
>> needed for the subsequent parts of this function. I might be
>> misunderstanding your question.
> 
> My expectation would have been that assisted_x2apic_available is assigned
> what is (in context above) assigned to virtualize_x2apic_mode (in patch 1).
> Anything deviating from this needs, I think, explaining there.

Oh, sorry, I kept thinking you meant cpu_has_... instead of the local 
variable and it was just all confusing me. Would it help if I didn't use 
a local variable at all, or changed the name then. This is really just a 
check that is done before executing the code below in the function.

Thanks,

Jane.

Re: [PATCH v3 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 24/02/2022 14:16, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 18.02.2022 18:29, Jane Malalane wrote:
>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> @@ -,15 +,15 @@ static void vmx_install_vlapic_mapping(struct vcpu 
>> *v)
>>   
>>   void vmx_vlapic_msr_changed(struct vcpu *v)
>>   {
>> -int virtualize_x2apic_mode;
>> +bool virtualize_x2apic_mode;
>>   struct vlapic *vlapic = vcpu_vlapic(v);
>>   unsigned int msr;
>>   
>>   virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt ||
>>   cpu_has_vmx_virtual_intr_delivery) &&
>> -   cpu_has_vmx_virtualize_x2apic_mode );
>> +   v->domain->arch.hvm.assisted_x2apic );
> 
> Following from my comment on patch 1, I'd expect this to become a simple
> assignment of v->domain->arch.hvm.assisted_x2apic (at which point the
> local variable could go away), just like ...
> 
>> -if ( !cpu_has_vmx_virtualize_apic_accesses &&
>> +if ( !v->domain->arch.hvm.assisted_xapic &&
>>!virtualize_x2apic_mode )
>>   return;
> 
> ... here.
Previosuly we discussed setting v->domain->arch.hvm.assisted_xapic equal 
to only cpu_has_vmx_virtualize_x2apic_mode, that's why I have those 
additional checks as at least apic_reg_virt or virtual_intr_delivery are 
needed for the subsequent parts of this function. I might be 
misunderstanding your question.

Unless you mean that we should fallback to having 
v->domain->arch.hvm.assisted_xapic depend on those other features...?
> 
>> @@ -1124,9 +1125,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>* and wrmsr in the guest will run without VMEXITs (see
>>* vmx_vlapic_msr_changed()).
>>*/
>> -if ( cpu_has_vmx_virtualize_x2apic_mode &&
>> - cpu_has_vmx_apic_reg_virt &&
>> - cpu_has_vmx_virtual_intr_delivery )
>> +if ( cpu_has_vmx_apic_reg_virt && cpu_has_vmx_virtual_intr_delivery 
>> &&
>> + v->domain->arch.hvm.assisted_x2apic )
>>   res->a |= XEN_HVM_CPUID_X2APIC_VIRT;
> 
> While within the 80 cols limit, I think it would help readability if you
> kept it at one sub-condition per line.
Sure.

Thank you,

Jane.

[PATCH v3 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v3:
 * Define XEN_SYSCTL_PHYSCAP_ARCH_MAX for ABI checking and actually
   set arch_capbilities, via a call to c_bitmap_to_ocaml_list()
 * Have assisted_x2apic_available only depend on
   cpu_has_vmx_virtualize_x2apic_mode
v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   (...)_X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  2 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/ocaml/libs/xc/xenctrl_stubs.c  | 14 +++---
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/vmx/vmcs.c  |  7 +++
 xen/arch/x86/include/asm/domain.h|  3 +++
 xen/arch/x86/sysctl.c|  7 +++
 xen/include/public/sysctl.h  | 11 ++-
 16 files changed, 90 insertions(+), 6 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..87be46c745 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -1014,6 +1014,8 @@ CapVmtrace bool
 CapVpmu bool
 CapGnttabV1 bool
 CapGnttabV2 bool
+CapAssistedXapic bool
+CapAssistedX2Apic bool
 }
 
 type Connectorinfo struct {
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 51a9b6cfac..333ffad38d 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -528,6 +528,13 @@
 #define LIBXL_HAVE_MAX_GRANT_VERSION 1
 
 /*
+ * LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has
+ * cap_assisted_x{2}apic fields, which indicates the availability of x{2}APIC
+ * hardware assisted virtualization.
+ */
+#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
diff --git a/tools/libs/light/libxl.c b/tools/libs/light/libxl.c
index a0bf7d186f..6d699951e2 100644
--- a/tools/libs/light/libxl.c
+++ b/tools/libs/light/libxl.c
@@ -15,6 +15,7 @@
 #include "libxl_osdeps.h"
 
 #include "libxl_internal.h"
+#include "libxl_arch.h"
 
 int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 unsigned flags, xentoollog_logger * lg)
@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo 
*physinfo)
 physinfo->cap_gnttab_v2 =
 !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2);
 
+libxl__arch_get_physinfo(physinfo, );
+
 GC_FREE;
 return 0;
 }
diff --git a/tools/libs/light/libxl_arch.h b/tools/libs/light/libxl_arch.h
index 1522ecb97f..207ceac6a1 100644
--- a/tools/libs/light/libxl_arch.h
+++ b/tools/libs/light/libxl_arch.h
@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc,
  uint64_t *out);
 
 _hidden
+void libxl__arch_get_physinfo(libxl_phy

[PATCH v3 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted vitualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by decoding the
APIC access and providing a VM exit with a more specific exit reason
than a regular EPT fault or by altogether avoiding a VM exit.

On the other hand, being able to disable x{2}APIC hardware assisted
vitualization can be useful for testing and debugging purposes.

Note: vmx_install_vlapic_mapping doesn't require modifications
regardless of whether the guest has virtualize_apic_accesses enabled
or not, i.e., setting the the APIC_ACCESS_ADDR VMCS field is fine so
long as virtualize_apic_accesses is supported by the CPU.

Suggested-by: Andrew Cooper 
Signed-off-by: Jane Malalane 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v3:
 * Change info in xl.cfg to better express reality and fix
   capitalization of x{2}apic
 * Move "physinfo" variable definition to the beggining of
   libxl__domain_build_info_setdefault()
 * Reposition brackets in if statement to match libxl coding style
 * Shorten logic in libxl__arch_domain_build_info_setdefault()
 * Correct dprintk message in arch_sanitise_domain_config()
 * Make appropriate changes in vmx_vlapic_msr_changed() and
   cpuid_hypervisor_leaves() for amended "assisted_x2apic" bit
 * Remove unneeded parantheses

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault()
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config()
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 19 +++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/golang/xenlight/types.gen.go|  2 ++
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  7 +--
 tools/libs/light/libxl_create.c   | 22 +-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 28 ++--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 16 
 xen/arch/x86/domain.c | 28 +++-
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c|  8 
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/traps.c  |  8 
 xen/include/public/arch-x86/xen.h |  2 ++
 22 files changed, 179 insertions(+), 25 deletions(-)

diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..dcca564a23 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1862,6 +1862,25 @@ firmware tables when using certain older guest Operating
 Systems. These tables have been superseded by newer constructs within
 the ACPI tables.
 
+=item B
+
+B<(x86 only)> Enables or disables hardware assisted virtualization for
+xAPIC. With this option enabled, a memory-mapped APIC access will be
+decoded by hardware and either issue a more specific VM exit than just
+an EPT fault, or altogether avoid a VM exit. Notice full
+virtualization for xAPIC can only be achieved if hardware supports
+“APIC-register virtualization” and “virtual-interrupt delivery”. The
+default is settable via L.
+
+=item B
+
+B<(x86 only)> Enables or disables hardware assisted virtualization for
+x2APIC. With this option enabled, an MSR-Based APIC access will
+either issue a VM exit or altogether avoid one. Notice full
+virtualization for x2APIC can only be achieved if hardware supports
+“APIC-register virtualization” and “virtual-interrupt delivery”. The
+default is settable via L.
+
 =item B
 
 B<(x86 only)> Hides or exposes the No-eXecute capability. This al

[PATCH v3 0/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 19 +
 docs/man/xl.conf.5.pod.in | 12 +++
 tools/golang/xenlight/helpers.gen.go  | 16 ++
 tools/golang/xenlight/types.gen.go|  4 
 tools/include/libxl.h | 14 +
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 12 +--
 tools/libs/light/libxl_create.c   | 22 
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 39 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 +++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 +++
 tools/ocaml/libs/xc/xenctrl_stubs.c   | 16 ++
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 --
 tools/xl/xl_parse.c   | 16 ++
 xen/arch/x86/domain.c | 28 -
 xen/arch/x86/hvm/vmx/vmcs.c   | 11 ++
 xen/arch/x86/hvm/vmx/vmx.c|  8 +++
 xen/arch/x86/include/asm/domain.h |  3 +++
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/sysctl.c |  7 +++
 xen/arch/x86/traps.c  |  8 +++
 xen/include/public/arch-x86/xen.h |  2 ++
 xen/include/public/sysctl.h   | 11 +-
 27 files changed, 269 insertions(+), 31 deletions(-)

-- 
2.11.0

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 15/02/2022 15:21, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 15.02.2022 16:10, Jane Malalane wrote:
>> On 15/02/2022 10:19, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 15.02.2022 11:14, Jane Malalane wrote:
>>>> On 15/02/2022 07:09, Jan Beulich wrote:
>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>>>> unless you have verified the sender and know the content is safe.
>>>>>
>>>>> On 14.02.2022 18:09, Jane Malalane wrote:
>>>>>> On 14/02/2022 13:18, Jan Beulich wrote:
>>>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
>>>>>>> attachments unless you have verified the sender and know the content is 
>>>>>>> safe.
>>>>>>>
>>>>>>> On 14.02.2022 14:11, Jane Malalane wrote:
>>>>>>>> On 11/02/2022 11:46, Jan Beulich wrote:
>>>>>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
>>>>>>>>> attachments unless you have verified the sender and know the content 
>>>>>>>>> is safe.
>>>>>>>>>
>>>>>>>>> On 11.02.2022 12:29, Roger Pau Monné wrote:
>>>>>>>>>> On Fri, Feb 11, 2022 at 10:06:48AM +, Jane Malalane wrote:
>>>>>>>>>>> On 10/02/2022 10:03, Roger Pau Monné wrote:
>>>>>>>>>>>> On Mon, Feb 07, 2022 at 06:21:00PM +, Jane Malalane wrote:
>>>>>>>>>>>>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c 
>>>>>>>>>>>>> b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>>>> index 7ab15e07a0..4060aef1bd 100644
>>>>>>>>>>>>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>>>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>>>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>>>>>>>>>>>>MSR_IA32_VMX_PROCBASED_CTLS2, );
>>>>>>>>>>>>>}
>>>>>>>>>>>>>
>>>>>>>>>>>>> +/* Check whether hardware supports accelerated xapic and 
>>>>>>>>>>>>> x2apic. */
>>>>>>>>>>>>> +if ( bsp )
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> +assisted_xapic_available = 
>>>>>>>>>>>>> cpu_has_vmx_virtualize_apic_accesses;
>>>>>>>>>>>>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>>>>>>>>>>>>> + 
>>>>>>>>>>>>> cpu_has_vmx_virtual_intr_delivery) &&
>>>>>>>>>>>>> +
>>>>>>>>>>>>> cpu_has_vmx_virtualize_x2apic_mode;
>>>>>>>>>>>>
>>>>>>>>>>>> I've been think about this, and it seems kind of asymmetric that 
>>>>>>>>>>>> for
>>>>>>>>>>>> xAPIC mode we report hw assisted support only with
>>>>>>>>>>>> virtualize_apic_accesses available, while for x2APIC we require
>>>>>>>>>>>> virtualize_x2apic_mode plus either apic_reg_virt or
>>>>>>>>>>>> virtual_intr_delivery.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we likely need to be more consistent here, and report hw
>>>>>>>>>>>> assisted x2APIC support as long as virtualize_x2apic_mode is
>>>>>>>>>>>> available.
>>>>>>>>>>>>
>>>>>>>>>>>> This will likely have some effect on patch 2 also, as you will 
>>>>>>>>>>>> have to
>>>>>>>>>>>> adjust vmx_vlapic_ms

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 15/02/2022 10:19, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 15.02.2022 11:14, Jane Malalane wrote:
>> On 15/02/2022 07:09, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 14.02.2022 18:09, Jane Malalane wrote:
>>>> On 14/02/2022 13:18, Jan Beulich wrote:
>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>>>> unless you have verified the sender and know the content is safe.
>>>>>
>>>>> On 14.02.2022 14:11, Jane Malalane wrote:
>>>>>> On 11/02/2022 11:46, Jan Beulich wrote:
>>>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open 
>>>>>>> attachments unless you have verified the sender and know the content is 
>>>>>>> safe.
>>>>>>>
>>>>>>> On 11.02.2022 12:29, Roger Pau Monné wrote:
>>>>>>>> On Fri, Feb 11, 2022 at 10:06:48AM +, Jane Malalane wrote:
>>>>>>>>> On 10/02/2022 10:03, Roger Pau Monné wrote:
>>>>>>>>>> On Mon, Feb 07, 2022 at 06:21:00PM +, Jane Malalane wrote:
>>>>>>>>>>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c 
>>>>>>>>>>> b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>> index 7ab15e07a0..4060aef1bd 100644
>>>>>>>>>>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>>>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>>>>>>>>>>   MSR_IA32_VMX_PROCBASED_CTLS2, );
>>>>>>>>>>>   }
>>>>>>>>>>>   
>>>>>>>>>>> +/* Check whether hardware supports accelerated xapic and 
>>>>>>>>>>> x2apic. */
>>>>>>>>>>> +if ( bsp )
>>>>>>>>>>> +{
>>>>>>>>>>> +assisted_xapic_available = 
>>>>>>>>>>> cpu_has_vmx_virtualize_apic_accesses;
>>>>>>>>>>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>>>>>>>>>>> + 
>>>>>>>>>>> cpu_has_vmx_virtual_intr_delivery) &&
>>>>>>>>>>> +
>>>>>>>>>>> cpu_has_vmx_virtualize_x2apic_mode;
>>>>>>>>>>
>>>>>>>>>> I've been think about this, and it seems kind of asymmetric that for
>>>>>>>>>> xAPIC mode we report hw assisted support only with
>>>>>>>>>> virtualize_apic_accesses available, while for x2APIC we require
>>>>>>>>>> virtualize_x2apic_mode plus either apic_reg_virt or
>>>>>>>>>> virtual_intr_delivery.
>>>>>>>>>>
>>>>>>>>>> I think we likely need to be more consistent here, and report hw
>>>>>>>>>> assisted x2APIC support as long as virtualize_x2apic_mode is
>>>>>>>>>> available.
>>>>>>>>>>
>>>>>>>>>> This will likely have some effect on patch 2 also, as you will have 
>>>>>>>>>> to
>>>>>>>>>> adjust vmx_vlapic_msr_changed.
>>>>>>>>>>
>>>>>>>>>> Thanks, Roger.
>>>>>>>>>
>>>>>>>>> Any other thoughts on this? As on one hand it is asymmetric but also
>>>>>>>>> there isn't much assistance with only virtualize_x2apic_mode set as, 
>>>>>>>>> in
>>>>>>>>> this case, a VM exit will be avoided only when trying to access the 
>>>>>>>>> TPR
>>>>>>>>> register.
>>>>>>>>
>>>>>>>> I've been thinking about this, and reporting hardware assisted
>>>>>>>> x{2}APIC virtualization with just
>>&g

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 15/02/2022 07:09, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 14.02.2022 18:09, Jane Malalane wrote:
>> On 14/02/2022 13:18, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 14.02.2022 14:11, Jane Malalane wrote:
>>>> On 11/02/2022 11:46, Jan Beulich wrote:
>>>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>>>> unless you have verified the sender and know the content is safe.
>>>>>
>>>>> On 11.02.2022 12:29, Roger Pau Monné wrote:
>>>>>> On Fri, Feb 11, 2022 at 10:06:48AM +, Jane Malalane wrote:
>>>>>>> On 10/02/2022 10:03, Roger Pau Monné wrote:
>>>>>>>> On Mon, Feb 07, 2022 at 06:21:00PM +, Jane Malalane wrote:
>>>>>>>>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>> index 7ab15e07a0..4060aef1bd 100644
>>>>>>>>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>>>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>>>>>>>>  MSR_IA32_VMX_PROCBASED_CTLS2, );
>>>>>>>>>  }
>>>>>>>>>  
>>>>>>>>> +/* Check whether hardware supports accelerated xapic and x2apic. 
>>>>>>>>> */
>>>>>>>>> +if ( bsp )
>>>>>>>>> +{
>>>>>>>>> +assisted_xapic_available = 
>>>>>>>>> cpu_has_vmx_virtualize_apic_accesses;
>>>>>>>>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>>>>>>>>> + 
>>>>>>>>> cpu_has_vmx_virtual_intr_delivery) &&
>>>>>>>>> +
>>>>>>>>> cpu_has_vmx_virtualize_x2apic_mode;
>>>>>>>>
>>>>>>>> I've been think about this, and it seems kind of asymmetric that for
>>>>>>>> xAPIC mode we report hw assisted support only with
>>>>>>>> virtualize_apic_accesses available, while for x2APIC we require
>>>>>>>> virtualize_x2apic_mode plus either apic_reg_virt or
>>>>>>>> virtual_intr_delivery.
>>>>>>>>
>>>>>>>> I think we likely need to be more consistent here, and report hw
>>>>>>>> assisted x2APIC support as long as virtualize_x2apic_mode is
>>>>>>>> available.
>>>>>>>>
>>>>>>>> This will likely have some effect on patch 2 also, as you will have to
>>>>>>>> adjust vmx_vlapic_msr_changed.
>>>>>>>>
>>>>>>>> Thanks, Roger.
>>>>>>>
>>>>>>> Any other thoughts on this? As on one hand it is asymmetric but also
>>>>>>> there isn't much assistance with only virtualize_x2apic_mode set as, in
>>>>>>> this case, a VM exit will be avoided only when trying to access the TPR
>>>>>>> register.
>>>>>>
>>>>>> I've been thinking about this, and reporting hardware assisted
>>>>>> x{2}APIC virtualization with just
>>>>>> SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES or
>>>>>> SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE doesn't seem very helpful. While
>>>>>> those provide some assistance to the VMM in order to handle APIC
>>>>>> accesses, it will still require a trap into the hypervisor to handle
>>>>>> most of the accesses.
>>>>>>
>>>>>> So maybe we should only report hardware assisted support when the
>>>>>> mentioned features are present together with
>>>>>> SECONDARY_EXEC_APIC_REGISTER_VIRT?
>>>>>
>>>>> Not sure - "some assistance" seems still a little better than none at all.
>>>>> Which route to go depends on what exactly we intend the bit to be used 
>>>>> for.
>>>>>
>>>> True. 

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 14/02/2022 13:18, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 14.02.2022 14:11, Jane Malalane wrote:
>> On 11/02/2022 11:46, Jan Beulich wrote:
>>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
>>> unless you have verified the sender and know the content is safe.
>>>
>>> On 11.02.2022 12:29, Roger Pau Monné wrote:
>>>> On Fri, Feb 11, 2022 at 10:06:48AM +, Jane Malalane wrote:
>>>>> On 10/02/2022 10:03, Roger Pau Monné wrote:
>>>>>> On Mon, Feb 07, 2022 at 06:21:00PM +, Jane Malalane wrote:
>>>>>>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>> index 7ab15e07a0..4060aef1bd 100644
>>>>>>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>>>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>>>>>> MSR_IA32_VMX_PROCBASED_CTLS2, );
>>>>>>> }
>>>>>>> 
>>>>>>> +/* Check whether hardware supports accelerated xapic and x2apic. */
>>>>>>> +if ( bsp )
>>>>>>> +{
>>>>>>> +assisted_xapic_available = 
>>>>>>> cpu_has_vmx_virtualize_apic_accesses;
>>>>>>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>>>>>>> + 
>>>>>>> cpu_has_vmx_virtual_intr_delivery) &&
>>>>>>> +cpu_has_vmx_virtualize_x2apic_mode;
>>>>>>
>>>>>> I've been think about this, and it seems kind of asymmetric that for
>>>>>> xAPIC mode we report hw assisted support only with
>>>>>> virtualize_apic_accesses available, while for x2APIC we require
>>>>>> virtualize_x2apic_mode plus either apic_reg_virt or
>>>>>> virtual_intr_delivery.
>>>>>>
>>>>>> I think we likely need to be more consistent here, and report hw
>>>>>> assisted x2APIC support as long as virtualize_x2apic_mode is
>>>>>> available.
>>>>>>
>>>>>> This will likely have some effect on patch 2 also, as you will have to
>>>>>> adjust vmx_vlapic_msr_changed.
>>>>>>
>>>>>> Thanks, Roger.
>>>>>
>>>>> Any other thoughts on this? As on one hand it is asymmetric but also
>>>>> there isn't much assistance with only virtualize_x2apic_mode set as, in
>>>>> this case, a VM exit will be avoided only when trying to access the TPR
>>>>> register.
>>>>
>>>> I've been thinking about this, and reporting hardware assisted
>>>> x{2}APIC virtualization with just
>>>> SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES or
>>>> SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE doesn't seem very helpful. While
>>>> those provide some assistance to the VMM in order to handle APIC
>>>> accesses, it will still require a trap into the hypervisor to handle
>>>> most of the accesses.
>>>>
>>>> So maybe we should only report hardware assisted support when the
>>>> mentioned features are present together with
>>>> SECONDARY_EXEC_APIC_REGISTER_VIRT?
>>>
>>> Not sure - "some assistance" seems still a little better than none at all.
>>> Which route to go depends on what exactly we intend the bit to be used for.
>>>
>> True. I intended this bit to be specifically for enabling
>> assisted_x{2}apic. So, would it be inconsistent to report hardware
>> assistance with just VIRTUALIZE_APIC_ACCESSES or VIRTUALIZE_X2APIC_MODE
>> but still claim that x{2}apic is virtualized if no MSR accesses are
>> intercepted with XEN_HVM_CPUID_X2APIC_VIRT (in traps.c) so that, as you
>> say, the guest gets at least "some assistance" instead of none but we
>> still claim x{2}apic virtualization when it is actually complete? Maybe
>> I could also add a comment alluding to this in the xl documentation.
> 
> To rephrase my earlier point: Which kind of decisions are the consumer(s)
> of us reporting hardware assistance going to take? In how far is there a
> risk that "some assistance" is overall going to lead to a loss of
> performance? I guess I'd need to see comment and actual code all in one
> place ...
> 
So, I was thinking of adding something along the lines of:

+=item B B<(x86 only)>
+Enables or disables hardware assisted virtualization for xAPIC. This
+allows accessing APIC registers without a VM-exit. Notice enabling
+this does not guarantee full virtualization for xAPIC, as this can
+only be achieved if hardware supports “APIC-register virtualization”
+and “virtual-interrupt delivery”. The default is settable via
+L.

and going for assisted_x2apic_available = 
cpu_has_vmx_virtualize_x2apic_mode.

This would prevent the customer from expecting full acceleration when 
apic_register_virt and/or virtual_intr_delivery aren't available whilst 
still offering some if they are not available as Xen currently does. In 
a future patch, we could also expose and add config options for these 
controls if we wanted to.

Thank you for your help,

Jane.

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 11/02/2022 11:46, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 11.02.2022 12:29, Roger Pau Monné wrote:
>> On Fri, Feb 11, 2022 at 10:06:48AM +0000, Jane Malalane wrote:
>>> On 10/02/2022 10:03, Roger Pau Monné wrote:
>>>> On Mon, Feb 07, 2022 at 06:21:00PM +, Jane Malalane wrote:
>>>>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>> index 7ab15e07a0..4060aef1bd 100644
>>>>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>>>>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>>>>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>>>>MSR_IA32_VMX_PROCBASED_CTLS2, );
>>>>>}
>>>>>
>>>>> +/* Check whether hardware supports accelerated xapic and x2apic. */
>>>>> +if ( bsp )
>>>>> +{
>>>>> +assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
>>>>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>>>>> + cpu_has_vmx_virtual_intr_delivery) 
>>>>> &&
>>>>> +cpu_has_vmx_virtualize_x2apic_mode;
>>>>
>>>> I've been think about this, and it seems kind of asymmetric that for
>>>> xAPIC mode we report hw assisted support only with
>>>> virtualize_apic_accesses available, while for x2APIC we require
>>>> virtualize_x2apic_mode plus either apic_reg_virt or
>>>> virtual_intr_delivery.
>>>>
>>>> I think we likely need to be more consistent here, and report hw
>>>> assisted x2APIC support as long as virtualize_x2apic_mode is
>>>> available.
>>>>
>>>> This will likely have some effect on patch 2 also, as you will have to
>>>> adjust vmx_vlapic_msr_changed.
>>>>
>>>> Thanks, Roger.
>>>
>>> Any other thoughts on this? As on one hand it is asymmetric but also
>>> there isn't much assistance with only virtualize_x2apic_mode set as, in
>>> this case, a VM exit will be avoided only when trying to access the TPR
>>> register.
>>
>> I've been thinking about this, and reporting hardware assisted
>> x{2}APIC virtualization with just
>> SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES or
>> SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE doesn't seem very helpful. While
>> those provide some assistance to the VMM in order to handle APIC
>> accesses, it will still require a trap into the hypervisor to handle
>> most of the accesses.
>>
>> So maybe we should only report hardware assisted support when the
>> mentioned features are present together with
>> SECONDARY_EXEC_APIC_REGISTER_VIRT?
> 
> Not sure - "some assistance" seems still a little better than none at all.
> Which route to go depends on what exactly we intend the bit to be used for.
> 
> Jan
> 
True. I intended this bit to be specifically for enabling 
assisted_x{2}apic. So, would it be inconsistent to report hardware 
assistance with just VIRTUALIZE_APIC_ACCESSES or VIRTUALIZE_X2APIC_MODE 
but still claim that x{2}apic is virtualized if no MSR accesses are 
intercepted with XEN_HVM_CPUID_X2APIC_VIRT (in traps.c) so that, as you 
say, the guest gets at least "some assistance" instead of none but we 
still claim x{2}apic virtualization when it is actually complete? Maybe 
I could also add a comment alluding to this in the xl documentation.

Thanks,

Jane.

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 10/02/2022 10:03, Roger Pau Monné wrote:
> On Mon, Feb 07, 2022 at 06:21:00PM +0000, Jane Malalane wrote:
>> diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
>> index 7ab15e07a0..4060aef1bd 100644
>> --- a/xen/arch/x86/hvm/vmx/vmcs.c
>> +++ b/xen/arch/x86/hvm/vmx/vmcs.c
>> @@ -343,6 +343,15 @@ static int vmx_init_vmcs_config(bool bsp)
>>   MSR_IA32_VMX_PROCBASED_CTLS2, );
>>   }
>>   
>> +/* Check whether hardware supports accelerated xapic and x2apic. */
>> +if ( bsp )
>> +{
>> +assisted_xapic_available = cpu_has_vmx_virtualize_apic_accesses;
>> +assisted_x2apic_available = (cpu_has_vmx_apic_reg_virt ||
>> + cpu_has_vmx_virtual_intr_delivery) &&
>> +cpu_has_vmx_virtualize_x2apic_mode;
> 
> I've been think about this, and it seems kind of asymmetric that for
> xAPIC mode we report hw assisted support only with
> virtualize_apic_accesses available, while for x2APIC we require
> virtualize_x2apic_mode plus either apic_reg_virt or
> virtual_intr_delivery.
> 
> I think we likely need to be more consistent here, and report hw
> assisted x2APIC support as long as virtualize_x2apic_mode is
> available.
> 
> This will likely have some effect on patch 2 also, as you will have to
> adjust vmx_vlapic_msr_changed.
> 
> Thanks, Roger.

Any other thoughts on this? As on one hand it is asymmetric but also 
there isn't much assistance with only virtualize_x2apic_mode set as, in 
this case, a VM exit will be avoided only when trying to access the TPR 
register.

Thanks,

Jane.

Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 10/02/2022 10:09, Roger Pau Monné wrote:
> On Mon, Feb 07, 2022 at 06:21:01PM +0000, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted vitualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>> read/write accesses without taking a VM exit.
>>
>> Being able to disable x{2}APIC hardware assisted vitualization can be
>> useful for testing and debugging purposes.
> 
> Might be worth adding a note to the commit log in order to note that
> vmx_install_vlapic_mapping doesn't require modifications regardless of
> whether the guest has virtualize_apic_accesses enabled or not.
> 
> Setting the APIC_ACCESS_ADDR VMCS field is fine even if
> virtualize_apic_accesses is not enabled for the guest: as long as the
> feature is supported by the CPU the field will exist.
Oh right. Will add these two points.

Thanks,

Jane.

Re: [PATCH] x86/dom0: Add log for dom0_nodes and dom0_max_vcpus_max conflict

[Jane Malalane]

On 09/02/2022 11:37, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 09.02.2022 11:31, Jane Malalane wrote:
>> This is not a bug. The xen cmdline can request both a NUMA restriction
>> and a vcpu count restriction for Dom0. The node restriction wil always
>> be respected which might mean either using dom0_max_vcpus <
>> opt_dom0_max_vcpus_max
> 
> This is quite normal a case if a range was specified, or did you mean
> opt_dom0_max_vcpus_min? But min and max get applied last anyway, so
> those always override what was derived from dom0_nr_pxms.
Yes, I was just giving context here for what I say in the following 
sentence. Maybe this became more confusing than helpful.
> 
>> or using more vCPUs than pCPUs on a node. In
>> the case where dom0_max_vcpus gets capped at the maximum number of
>> pCPUs for the number of nodes chosen, it can be useful particularly
>> for debugging to print a message in the serial log.
> > The number of vCPU-s Dom0 gets is logged in all cases. And the
> reasons why a certain value is uses depends on more than just
> the number-of-nodes restriction. 
Maybe I should have said 'Dom0 "receiving" %d vCPUS' instead of "using" 
in the serial log, in which case I can amend that to make it clearer 
(that ofc if we still want this change)?
I therefor wonder whether the
> wording as you've chosen it is potentially misleading, and
> properly expressing everything in a single message is going to
> be quite a bit too noisy. Furthermore ...
> 
>> --- a/xen/arch/x86/dom0_build.c
>> +++ b/xen/arch/x86/dom0_build.c
>> @@ -240,6 +240,11 @@ unsigned int __init dom0_max_vcpus(void)
>>   if ( max_vcpus > limit )
>>   max_vcpus = limit;
>>   
>> +if ( max_vcpus < opt_dom0_max_vcpus_max && max_vcpus > 
>> opt_dom0_max_vcpus_min )
>> +printk(XENLOG_INFO "Dom0 using %d vCPUs conflicts with request to 
>> use"
>> +   " %d node(s), using up to %d vCPUs\n", 
>> opt_dom0_max_vcpus_max,
>> +   dom0_nr_pxms, max_vcpus);
> 
> ... the function can be called more than once, whereas such a
> message (if we really want it) would better be issued just once.
Yes, that is true and this code would have to live outside of dom0_build.c.
> 
> To answer your later reply to yourself: I think printk() is fine
> here (again assuming we want such a message in the first place);
> it's a boot-time-only message after all.
> 
Okay.

Thank you,

Jane.

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 09/02/2022 13:48, Anthony PERARD wrote:
> On Wed, Feb 09, 2022 at 12:26:05PM +0000, Jane Malalane wrote:
>> On 08/02/2022 15:26, Roger Pau Monné wrote:
>>> On Mon, Feb 07, 2022 at 06:21:00PM +0000, Jane Malalane wrote:
>>>> diff --git a/tools/golang/xenlight/types.gen.go 
>>>> b/tools/golang/xenlight/types.gen.go
>>>> index b1e84d5258..5f384b767c 100644
>>>> --- a/tools/golang/xenlight/types.gen.go
>>>> +++ b/tools/golang/xenlight/types.gen.go
>>>> @@ -389,6 +389,10 @@ RunHotplugScripts Defbool
>>>>DriverDomain Defbool
>>>>Passthrough Passthrough
>>>>XendSuspendEvtchnCompat Defbool
>>>> +ArchX86 struct {
>>>> +AssistedXapic Defbool
>>>> +AssistedX2Apic Defbool
>>>
>>> Don't you need some indentation here?
>> I hadn't realized it appeared like this here (and the same happens for
>> other parts of my code as I'm seeing now) because the git output is
>> correct. I will fix it.
>>>
>>> Also name would better be Assistedx{2}APIC IMO if possible. Having a
>>> capital 'X' and lowercase 'apic' looks really strange.
>> Okay.
> 
> 
> This is a generated file, you can't change indentation or fields names.
> It would be rebuilt automatically if you had golang installed and where
> rebuilding all the tools.
> 
> There's two ways to generate it, you could install golang and build all
> the tools. Or without golang: just
> `cd tools/golang/xenlight; make types.gen.go`. Both should regenerate
> both "helpers.gen.go" "types.gen.go" files.
> 
> There's an even easier way, tell the committer to regen the files when
> committing :-).

Oh, yeah I forgot they were automatically generated. Right, thank you.

Jane.

Re: [PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 08/02/2022 15:26, Roger Pau Monné wrote:
> On Mon, Feb 07, 2022 at 06:21:00PM +0000, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Signed-off-by: Jane Malalane 
>> Suggested-by: Andrew Cooper 
> 
> Tag order should be inverted, first Suggested-by, then SoB.
> 
>> ---
>> CC: Wei Liu 
>> CC: Anthony PERARD 
>> CC: Juergen Gross 
>> CC: Andrew Cooper 
>> CC: George Dunlap 
>> CC: Jan Beulich 
>> CC: Julien Grall 
>> CC: Stefano Stabellini 
>> CC: Volodymyr Babchuk 
>> CC: Bertrand Marquis 
>> CC: Jun Nakajima 
>> CC: Kevin Tian 
>> CC: "Roger Pau Monné" 
>>
>> v2:
>>   * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
>>   * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
>>   * Set assisted_x{2}apic_available to be conditional upon "bsp" and
>> annotate it with __ro_after_init
>>   * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
>> .._X86_ASSISTED_X{2}APIC
>>   * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
>> sysctl.h
>>   * Fix padding introduced in struct xen_sysctl_physinfo and bump
>> XEN_SYSCTL_INTERFACE_VERSION
>> ---
>>   tools/golang/xenlight/helpers.gen.go |  4 
>>   tools/golang/xenlight/types.gen.go   |  6 ++
>>   tools/include/libxl.h|  7 +++
>>   tools/libs/light/libxl.c |  3 +++
>>   tools/libs/light/libxl_arch.h|  4 
>>   tools/libs/light/libxl_arm.c |  5 +
>>   tools/libs/light/libxl_types.idl |  2 ++
>>   tools/libs/light/libxl_x86.c | 11 +++
>>   tools/ocaml/libs/xc/xenctrl.ml   |  5 +
>>   tools/ocaml/libs/xc/xenctrl.mli  |  5 +
>>   tools/xl/xl_info.c   |  6 --
>>   xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
>>   xen/arch/x86/include/asm/domain.h|  3 +++
>>   xen/arch/x86/sysctl.c|  7 +++
>>   xen/include/public/sysctl.h  |  8 +++-
>>   15 files changed, 82 insertions(+), 3 deletions(-)
>>
>> diff --git a/tools/golang/xenlight/helpers.gen.go 
>> b/tools/golang/xenlight/helpers.gen.go
>> index b746ff1081..dd4e6c9f14 100644
>> --- a/tools/golang/xenlight/helpers.gen.go
>> +++ b/tools/golang/xenlight/helpers.gen.go
>> @@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
>>   x.CapVpmu = bool(xc.cap_vpmu)
>>   x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
>>   x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
>> +x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
>> +x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
>>   
>>return nil}
>>   
>> @@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
>>   xc.cap_vpmu = C.bool(x.CapVpmu)
>>   xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
>>   xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
>> +xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
>> +xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
>>   
>>return nil
>>}
>> diff --git a/tools/golang/xenlight/types.gen.go 
>> b/tools/golang/xenlight/types.gen.go
>> index b1e84d5258..5f384b767c 100644
>> --- a/tools/golang/xenlight/types.gen.go
>> +++ b/tools/golang/xenlight/types.gen.go
>> @@ -389,6 +389,10 @@ RunHotplugScripts Defbool
>>   DriverDomain Defbool
>>   Passthrough Passthrough
>>   XendSuspendEvtchnCompat Defbool
>> +ArchX86 struct {
>> +AssistedXapic Defbool
>> +AssistedX2Apic Defbool
> 
> Don't you need some indentation here?
I hadn't realized it appeared like this here (and the same happens for 
other parts of my code as I'm seeing now) because the git output is 
correct. I will fix it.
> 
> Also name would better be Assistedx{2}APIC IMO if possible. Having a
> capital 'X' and lowercase 'apic' looks really strange.
Okay.
> 
>> +}
>>   }
>>   
>>   type DomainRestoreParams struct {
>> @@ -1014,6 +1018,8 @@ CapVmtrace bool
>>   CapVpmu bool
>>   CapGnttabV1 bool
>>   CapGnttabV2 bool
>> +CapAssistedXApic bool
>> +CapAssistedX2apic bool
>>   }
>>   
>>   type Connectorinfo struct {
>> diff --git a/tools/include/libxl.h b/tools/include/libxl.h
>> index 2bbbd21f0b..924e142628 100644
>> 

Re: [PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/02/2022 16:17, Roger Pau Monné wrote:
> On Mon, Feb 07, 2022 at 06:21:01PM +0000, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted vitualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>> read/write accesses without taking a VM exit.
>>
>> Being able to disable x{2}APIC hardware assisted vitualization can be
>> useful for testing and debugging purposes.
> 
> I think you have agreed with Jan some changes to the description
> regarding the purpose of the commit.
Yes.
> 
>>
>> Signed-off-by: Jane Malalane 
>> Suggested-by: Andrew Cooper 
Will swap the ordering, forgot to do this previously, apologies.
>> ---
>> CC: Wei Liu 
>> CC: Anthony PERARD 
>> CC: Juergen Gross 
>> CC: Andrew Cooper 
>> CC: George Dunlap 
>> CC: Jan Beulich 
>> CC: Julien Grall 
>> CC: Stefano Stabellini 
>> CC: Christian Lindig 
>> CC: David Scott 
>> CC: Volodymyr Babchuk 
>> CC: "Roger Pau Monné" 
>>
>> v2:
>>   * Add a LIBXL_HAVE_ASSISTED_APIC macro
>>   * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
>>   * Add a return statement in now "int"
>> libxl__arch_domain_build_info_setdefault
>>   * Preserve libxl__arch_domain_build_info_setdefault 's location in
>> libxl_create.c
>>   * Correct x{2}apic default setting logic in
>> libxl__arch_domain_prepare_config
>>   * Correct logic for parsing assisted_x{2}apic host/guest options in
>> xl_parse.c and initialize them to -1 in xl.c
>>   * Use guest options directly in vmx_vlapic_msr_changed
>>   * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
>>   * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
>> ---
>>   docs/man/xl.cfg.5.pod.in  | 10 ++
>>   docs/man/xl.conf.5.pod.in | 12 
>>   tools/golang/xenlight/helpers.gen.go  | 12 
>>   tools/include/libxl.h |  7 +++
>>   tools/libs/light/libxl_arch.h |  5 +++--
>>   tools/libs/light/libxl_arm.c  |  7 +--
>>   tools/libs/light/libxl_create.c   | 23 ++-
>>   tools/libs/light/libxl_types.idl  |  2 ++
>>   tools/libs/light/libxl_x86.c  | 31 +--
>>   tools/ocaml/libs/xc/xenctrl.ml|  2 ++
>>   tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
>>   tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
>>   tools/xl/xl.c |  8 
>>   tools/xl/xl.h |  2 ++
>>   tools/xl/xl_parse.c   | 16 
>>   xen/arch/x86/domain.c | 28 +++-
>>   xen/arch/x86/hvm/vmx/vmcs.c   |  4 
>>   xen/arch/x86/hvm/vmx/vmx.c| 14 +-
>>   xen/arch/x86/include/asm/hvm/domain.h |  6 ++
>>   xen/arch/x86/traps.c  |  8 
>>   xen/include/public/arch-x86/xen.h |  2 ++
>>   21 files changed, 173 insertions(+), 30 deletions(-)
>>
>> diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
>> index b98d161398..1d98bbd182 100644
>> --- a/docs/man/xl.cfg.5.pod.in
>> +++ b/docs/man/xl.cfg.5.pod.in
>> @@ -1862,6 +1862,16 @@ firmware tables when using certain older guest 
>> Operating
>>   Systems. These tables have been superseded by newer constructs within
>>   the ACPI tables.
>>   
>> +=item B
>> +B<(x86 only)> Enables or disables hardware assisted virtualization for 
>> xapic.
>> +This allows accessing APIC registers without a VM-exit.
>> +The default is settable via L.
>> +
>> +=item B
>> +B<(x86 only)> Enables or disables hardware assisted virtualization for 
>> x2apic.
>> +This allows accessing APIC registers without a VM-exit.
>> +The default is settable via L.
>> +
>>   =item B
>>   
>>   B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a 
>> guest
>> diff --git a/docs/man/xl.conf.5.pod.in b/docs/man/xl.conf.5.pod.in
>> index df20c08137..30993827e5 100644
>> --- a/docs/man/xl.conf.5.pod.in
>> +++ b/docs/man/xl.conf.5.pod.in
>> @@ -107,6 +107,18 @@ Se

Re: [PATCH] x86/dom0: Add log for dom0_nodes and dom0_max_vcpus_max conflict

[Jane Malalane]

On 09/02/2022 10:31, Jane Malalane wrote:
> This is not a bug. The xen cmdline can request both a NUMA restriction
> and a vcpu count restriction for Dom0. The node restriction wil always
> be respected which might mean either using dom0_max_vcpus <
> opt_dom0_max_vcpus_max or using more vCPUs than pCPUs on a node. In
> the case where dom0_max_vcpus gets capped at the maximum number of
> pCPUs for the number of nodes chosen, it can be useful particularly
> for debugging to print a message in the serial log.
> 
> Suggested-by: Edwin Torok 
> Signed-off-by: Jane Malalane 
> ---
> CC: Jan Beulich 
> CC: Andrew Cooper 
> CC: "Roger Pau Monné" 
> CC: Wei Liu 
> ---
>   xen/arch/x86/dom0_build.c | 5 +
>   1 file changed, 5 insertions(+)
> 
> diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
> index fe24e11b37..e57cc80ef0 100644
> --- a/xen/arch/x86/dom0_build.c
> +++ b/xen/arch/x86/dom0_build.c
> @@ -240,6 +240,11 @@ unsigned int __init dom0_max_vcpus(void)
>   if ( max_vcpus > limit )
>   max_vcpus = limit;
>   
> +if ( max_vcpus < opt_dom0_max_vcpus_max && max_vcpus > 
> opt_dom0_max_vcpus_min )
> +printk(XENLOG_INFO "Dom0 using %d vCPUs conflicts with request to 
> use"
> +   " %d node(s), using up to %d vCPUs\n", opt_dom0_max_vcpus_max,
> +   dom0_nr_pxms, max_vcpus);
> +
>   return max_vcpus;
>   }
>   
Here I was debating whether to use a printk or a dprintk, as although 
this would be useful for debugging, it gives general info on dom0 vCPU 
topology, as does for e.g. 'Dom0 has maximimum ... vCPUS'. However, I 
defer to the maintainers, as I'm still getting acquainted with Xen (I 
also understand this patch is of minor importance).

Thanks,

Jane.

[PATCH] x86/dom0: Add log for dom0_nodes and dom0_max_vcpus_max conflict

[Jane Malalane]

This is not a bug. The xen cmdline can request both a NUMA restriction
and a vcpu count restriction for Dom0. The node restriction wil always
be respected which might mean either using dom0_max_vcpus <
opt_dom0_max_vcpus_max or using more vCPUs than pCPUs on a node. In
the case where dom0_max_vcpus gets capped at the maximum number of
pCPUs for the number of nodes chosen, it can be useful particularly
for debugging to print a message in the serial log.

Suggested-by: Edwin Torok 
Signed-off-by: Jane Malalane 
---
CC: Jan Beulich 
CC: Andrew Cooper 
CC: "Roger Pau Monné" 
CC: Wei Liu 
---
 xen/arch/x86/dom0_build.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index fe24e11b37..e57cc80ef0 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -240,6 +240,11 @@ unsigned int __init dom0_max_vcpus(void)
 if ( max_vcpus > limit )
 max_vcpus = limit;
 
+if ( max_vcpus < opt_dom0_max_vcpus_max && max_vcpus > 
opt_dom0_max_vcpus_min )
+printk(XENLOG_INFO "Dom0 using %d vCPUs conflicts with request to use"
+   " %d node(s), using up to %d vCPUs\n", opt_dom0_max_vcpus_max,
+   dom0_nr_pxms, max_vcpus);
+
 return max_vcpus;
 }
 
-- 
2.11.0

Re: [PATCH 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 08/02/2022 13:52, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments 
> unless you have verified the sender and know the content is safe.
> 
> On 08.02.2022 14:27, Jane Malalane wrote:
>> On 31/01/2022 12:05, Jan Beulich wrote:
>>> On 27.01.2022 17:01, Jane Malalane wrote:
>>>> Introduce a new per-domain creation x86 specific flag to
>>>> select whether hardware assisted virtualization should be used for
>>>> x{2}APIC.
>>>>
>>>> A per-domain option is added to xl in order to select the usage of
>>>> x{2}APIC hardware assisted vitualization, as well as a global
>>>> configuration option.
>>>>
>>>> Having all APIC interaction exit to Xen for emulation is slow and can
>>>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>>>> read/write accesses without taking a VM exit.
>>>
>>> This is odd to read for a patch which makes it possible to _turn off_
>>> acceleration. Instead it would be interesting to know what problems
>>> you have encountered making it desirable to have a way to turn this off.
>>
>> After speaking to Andrew he told me of a performance regression that was
>> reported some time ago when enabling apicv related to the pass-through
>> LAPIC timer of a HVM guest causing Xen to intercept the LAPIC timer MSR,
>> making anything that uses the LAPIC timer end up slower than it was
>> before. So, adressing your comment here, other than mentioning how being
>> able to disable acceleration for apicv can be useful when testing and
>> debugging, do you think it's worth mentioning (in more detail) that this
>> perf problem exists, in the commit message.
> 
> Yes, I think it would be worth mentioning, as then the purpose of this
> change is also to be a workaround there, not just testing/debugging. In
> fact this workaround may then be viewed as the primary purpose.
> 
Okay I will add this in a v3 (alongside other changes I'll have to make).
Thank you,

Jane.

Re: [PATCH 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 31/01/2022 12:05, Jan Beulich wrote:
> On 27.01.2022 17:01, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted vitualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>> read/write accesses without taking a VM exit.
> 
> This is odd to read for a patch which makes it possible to _turn off_
> acceleration. Instead it would be interesting to know what problems
> you have encountered making it desirable to have a way to turn this off.

Hi Jan,

After speaking to Andrew he told me of a performance regression that was 
reported some time ago when enabling apicv related to the pass-through 
LAPIC timer of a HVM guest causing Xen to intercept the LAPIC timer MSR, 
making anything that uses the LAPIC timer end up slower than it was 
before. So, adressing your comment here, other than mentioning how being 
able to disable acceleration for apicv can be useful when testing and 
debugging, do you think it's worth mentioning (in more detail) that this 
perf problem exists, in the commit message.

Thanks,

Jane.

> 
>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> @@ -3342,16 +3342,19 @@ static void vmx_install_vlapic_mapping(struct vcpu 
>> *v)
>>   
>>   void vmx_vlapic_msr_changed(struct vcpu *v)
>>   {
>> -int virtualize_x2apic_mode;
>> +int virtualize_xapic_mode, virtualize_x2apic_mode;
> 
> Please switch to bool as you touch and extend this.
> 
>>   struct vlapic *vlapic = vcpu_vlapic(v);
>>   unsigned int msr;
>>   
>> +virtualize_xapic_mode = ( cpu_has_vmx_virtualize_apic_accesses &&
>> +  v->domain->arch.hvm.assisted_xapic );
> 
> Please don't clone the bad use of blanks immediately inside parentheses
> here; instead, ...
> 
>>   virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt ||
>>   cpu_has_vmx_virtual_intr_delivery) &&
>> -   cpu_has_vmx_virtualize_x2apic_mode );
>> +   cpu_has_vmx_virtualize_x2apic_mode &&
>> +   v->domain->arch.hvm.assisted_x2apic );
> 
> ... since you're touching this anyway, please consider correcting
> the style violation here.
> 
> However - do you need these expressions anymore? The per-domain fields
> can only be set if the respective CPU capabilities exit.
> 
>> --- a/xen/arch/x86/include/asm/hvm/domain.h
>> +++ b/xen/arch/x86/include/asm/hvm/domain.h
>> @@ -117,6 +117,12 @@ struct hvm_domain {
>>   
>>   bool   is_s3_suspended;
>>   
>> +/* xAPIC hardware assisted emulation. */
>> +bool assisted_xapic;
>> +
>> +/* x2APIC hardware assisted emulation. */
>> +bool assisted_x2apic;
>> +
>>   /* hypervisor intercepted msix table */
>>   struct list_head   msixtbl_list;
> 
> Please follow how adjacent code pads types / names here.
> 
>> --- a/xen/arch/x86/traps.c
>> +++ b/xen/arch/x86/traps.c
>> @@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>   if ( !is_hvm_domain(d) || subleaf != 0 )
>>   break;
>>   
>> -if ( cpu_has_vmx_apic_reg_virt )
>> +if ( cpu_has_vmx_apic_reg_virt &&
>> + v->domain->arch.hvm.assisted_xapic )
>>   res->a |= XEN_HVM_CPUID_APIC_ACCESS_VIRT;
>>   
>>   /*
>> @@ -1126,7 +1127,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>*/
>>   if ( cpu_has_vmx_virtualize_x2apic_mode &&
>>cpu_has_vmx_apic_reg_virt &&
>> - cpu_has_vmx_virtual_intr_delivery )
>> + cpu_has_vmx_virtual_intr_delivery &&
>> + v->domain->arch.hvm.assisted_x2apic )
>>   res->a |= XEN_HVM_CPUID_X2APIC_VIRT;
> 
> Same remark as above - can't you now use _just_ the per-domain field?
> In this latter of the two cases this would then also mean bringing
> the CPU feature checks in line with what vmx_vlapic_msr_changed()
> does (as also pointed out for patch 1). Albeit it might be best to
> have a prereq patch fixing the issue, which could then be backported.
> 
> Jan
> 
> 

[PATCH v2 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Introduce a new per-domain creation x86 specific flag to
select whether hardware assisted virtualization should be used for
x{2}APIC.

A per-domain option is added to xl in order to select the usage of
x{2}APIC hardware assisted vitualization, as well as a global
configuration option.

Having all APIC interaction exit to Xen for emulation is slow and can
induce much overhead. Hardware can speed up x{2}APIC by running APIC
read/write accesses without taking a VM exit.

Being able to disable x{2}APIC hardware assisted vitualization can be
useful for testing and debugging purposes.

Signed-off-by: Jane Malalane 
Suggested-by: Andrew Cooper 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Christian Lindig 
CC: David Scott 
CC: Volodymyr Babchuk 
CC: "Roger Pau Monné" 

v2:
 * Add a LIBXL_HAVE_ASSISTED_APIC macro
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Add a return statement in now "int"
   libxl__arch_domain_build_info_setdefault
 * Preserve libxl__arch_domain_build_info_setdefault 's location in
   libxl_create.c
 * Correct x{2}apic default setting logic in
   libxl__arch_domain_prepare_config
 * Correct logic for parsing assisted_x{2}apic host/guest options in
   xl_parse.c and initialize them to -1 in xl.c
 * Use guest options directly in vmx_vlapic_msr_changed
 * Fix indentation of bool assisted_x{2}apic in struct hvm_domain
 * Add a change in xenctrl_stubs.c to pass xenctrl ABI checks
---
 docs/man/xl.cfg.5.pod.in  | 10 ++
 docs/man/xl.conf.5.pod.in | 12 
 tools/golang/xenlight/helpers.gen.go  | 12 
 tools/include/libxl.h |  7 +++
 tools/libs/light/libxl_arch.h |  5 +++--
 tools/libs/light/libxl_arm.c  |  7 +--
 tools/libs/light/libxl_create.c   | 23 ++-
 tools/libs/light/libxl_types.idl  |  2 ++
 tools/libs/light/libxl_x86.c  | 31 +--
 tools/ocaml/libs/xc/xenctrl.ml|  2 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 
 tools/xl/xl.h |  2 ++
 tools/xl/xl_parse.c   | 16 
 xen/arch/x86/domain.c | 28 +++-
 xen/arch/x86/hvm/vmx/vmcs.c   |  4 
 xen/arch/x86/hvm/vmx/vmx.c| 14 +-
 xen/arch/x86/include/asm/hvm/domain.h |  6 ++
 xen/arch/x86/traps.c  |  8 
 xen/include/public/arch-x86/xen.h |  2 ++
 21 files changed, 173 insertions(+), 30 deletions(-)

diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index b98d161398..1d98bbd182 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1862,6 +1862,16 @@ firmware tables when using certain older guest Operating
 Systems. These tables have been superseded by newer constructs within
 the ACPI tables.
 
+=item B
+B<(x86 only)> Enables or disables hardware assisted virtualization for xapic.
+This allows accessing APIC registers without a VM-exit.
+The default is settable via L.
+
+=item B
+B<(x86 only)> Enables or disables hardware assisted virtualization for x2apic.
+This allows accessing APIC registers without a VM-exit.
+The default is settable via L.
+
 =item B
 
 B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a guest
diff --git a/docs/man/xl.conf.5.pod.in b/docs/man/xl.conf.5.pod.in
index df20c08137..30993827e5 100644
--- a/docs/man/xl.conf.5.pod.in
+++ b/docs/man/xl.conf.5.pod.in
@@ -107,6 +107,18 @@ Sets the default value for the C domain 
config value.
 
 Default: maximum grant version supported by the hypervisor.
 
+=item B
+
+If enabled, domains will use xAPIC hardware assisted virtualization by default.
+
+Default: enabled if supported.
+
+=item B
+
+If enabled, domains will use x2APIC hardware assisted virtualization by 
default.
+
+Default: enabled if supported.
+
 =item B
 
 Configures the default hotplug script used by virtual network devices.
diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index dd4e6c9f14..90e7b9b205 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -636,6 +636,12 @@ x.Passthrough = Passthrough(xc.passthrough)
 if err := x.XendSuspendEvtchnCompat.fromC(_suspend_evtchn_compat);err 
!= nil {
 return fmt.Errorf("converting field XendSuspendEvtchnCompat: %v", err)
 }
+if err := x.ArchX86.AssistedXapic.fromC(_x86.assisted_xapic);err != 
nil {
+return fmt.Errorf("converting field ArchX86.AssistedXapic: %v", err)
+}
+if err := x.ArchX86.AssistedX2Apic.fromC(_x86.assisted_x2apic);err != 
nil {
+return fmt.Errorf("converting field ArchX86.AssistedX2Apic: %v", err)
+}
 
  ret

[PATCH v2 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
and x2apic, on x86 hardware.
No such features are currently implemented on AMD hardware.

For that purpose, also add an arch-specific "capabilities" parameter
to struct xen_sysctl_physinfo.

Signed-off-by: Jane Malalane 
Suggested-by: Andrew Cooper 
---
CC: Wei Liu 
CC: Anthony PERARD 
CC: Juergen Gross 
CC: Andrew Cooper 
CC: George Dunlap 
CC: Jan Beulich 
CC: Julien Grall 
CC: Stefano Stabellini 
CC: Volodymyr Babchuk 
CC: Bertrand Marquis 
CC: Jun Nakajima 
CC: Kevin Tian 
CC: "Roger Pau Monné" 

v2:
 * Use one macro LIBXL_HAVE_PHYSINFO_ASSISTED_APIC instead of two
 * Pass xcpyshinfo as a pointer in libxl__arch_get_physinfo
 * Set assisted_x{2}apic_available to be conditional upon "bsp" and
   annotate it with __ro_after_init
 * Change XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_X{2}APIC to
   .._X86_ASSISTED_X{2}APIC
 * Keep XEN_SYSCTL_PHYSCAP_X86_ASSISTED_X{2}APIC contained within
   sysctl.h
 * Fix padding introduced in struct xen_sysctl_physinfo and bump
   XEN_SYSCTL_INTERFACE_VERSION
---
 tools/golang/xenlight/helpers.gen.go |  4 
 tools/golang/xenlight/types.gen.go   |  6 ++
 tools/include/libxl.h|  7 +++
 tools/libs/light/libxl.c |  3 +++
 tools/libs/light/libxl_arch.h|  4 
 tools/libs/light/libxl_arm.c |  5 +
 tools/libs/light/libxl_types.idl |  2 ++
 tools/libs/light/libxl_x86.c | 11 +++
 tools/ocaml/libs/xc/xenctrl.ml   |  5 +
 tools/ocaml/libs/xc/xenctrl.mli  |  5 +
 tools/xl/xl_info.c   |  6 --
 xen/arch/x86/hvm/vmx/vmcs.c  |  9 +
 xen/arch/x86/include/asm/domain.h|  3 +++
 xen/arch/x86/sysctl.c|  7 +++
 xen/include/public/sysctl.h  |  8 +++-
 15 files changed, 82 insertions(+), 3 deletions(-)

diff --git a/tools/golang/xenlight/helpers.gen.go 
b/tools/golang/xenlight/helpers.gen.go
index b746ff1081..dd4e6c9f14 100644
--- a/tools/golang/xenlight/helpers.gen.go
+++ b/tools/golang/xenlight/helpers.gen.go
@@ -3373,6 +3373,8 @@ x.CapVmtrace = bool(xc.cap_vmtrace)
 x.CapVpmu = bool(xc.cap_vpmu)
 x.CapGnttabV1 = bool(xc.cap_gnttab_v1)
 x.CapGnttabV2 = bool(xc.cap_gnttab_v2)
+x.CapAssistedXapic = bool(xc.cap_assisted_xapic)
+x.CapAssistedX2Apic = bool(xc.cap_assisted_x2apic)
 
  return nil}
 
@@ -3407,6 +3409,8 @@ xc.cap_vmtrace = C.bool(x.CapVmtrace)
 xc.cap_vpmu = C.bool(x.CapVpmu)
 xc.cap_gnttab_v1 = C.bool(x.CapGnttabV1)
 xc.cap_gnttab_v2 = C.bool(x.CapGnttabV2)
+xc.cap_assisted_xapic = C.bool(x.CapAssistedXapic)
+xc.cap_assisted_x2apic = C.bool(x.CapAssistedX2Apic)
 
  return nil
  }
diff --git a/tools/golang/xenlight/types.gen.go 
b/tools/golang/xenlight/types.gen.go
index b1e84d5258..5f384b767c 100644
--- a/tools/golang/xenlight/types.gen.go
+++ b/tools/golang/xenlight/types.gen.go
@@ -389,6 +389,10 @@ RunHotplugScripts Defbool
 DriverDomain Defbool
 Passthrough Passthrough
 XendSuspendEvtchnCompat Defbool
+ArchX86 struct {
+AssistedXapic Defbool
+AssistedX2Apic Defbool
+}
 }
 
 type DomainRestoreParams struct {
@@ -1014,6 +1018,8 @@ CapVmtrace bool
 CapVpmu bool
 CapGnttabV1 bool
 CapGnttabV2 bool
+CapAssistedXApic bool
+CapAssistedX2apic bool
 }
 
 type Connectorinfo struct {
diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 2bbbd21f0b..924e142628 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -528,6 +528,13 @@
 #define LIBXL_HAVE_MAX_GRANT_VERSION 1
 
 /*
+ * LIBXL_HAVE_PHYSINFO_ASSISTED_APIC indicates that libxl_physinfo has
+ * cap_assisted_x{2}apic fields, which indicates the availability of x{2}APIC
+ * hardware assisted virtualization.
+ */
+#define LIBXL_HAVE_PHYSINFO_ASSISTED_APIC 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
diff --git a/tools/libs/light/libxl.c b/tools/libs/light/libxl.c
index 667ae6409b..fabb474221 100644
--- a/tools/libs/light/libxl.c
+++ b/tools/libs/light/libxl.c
@@ -15,6 +15,7 @@
 #include "libxl_osdeps.h"
 
 #include "libxl_internal.h"
+#include "libxl_arch.h"
 
 int libxl_ctx_alloc(libxl_ctx **pctx, int version,
 unsigned flags, xentoollog_logger * lg)
@@ -410,6 +411,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo 
*physinfo)
 physinfo->cap_gnttab_v2 =
 !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_gnttab_v2);
 
+libxl__arch_get_physinfo(physinfo, );
+
 GC_FREE;
 return 0;
 }
diff --git a/tools/libs/light/libxl_arch.h b/tools/libs/light/libxl_arch.h
index 1522ecb97f..207ceac6a1 100644
--- a/tools/libs/light/libxl_arch.h
+++ b/tools/libs/light/libxl_arch.h
@@ -86,6 +86,10 @@ int libxl__arch_extra_memory(libxl__gc *gc,
  uint64_t *out);
 
 _hidden
+void libxl__arch_get_physinfo(libxl_physinfo *physinfo,
+ 

[PATCH v2 0/2] xen: Report and use hardware APIC virtualization capabilities

[Jane Malalane]

Jane Malalane (2):
  xen+tools: Report Interrupt Controller Virtualization capabilities on
x86
  x86/xen: Allow per-domain usage of hardware virtualized APIC

 docs/man/xl.cfg.5.pod.in  | 10 +
 docs/man/xl.conf.5.pod.in | 12 ++
 tools/golang/xenlight/helpers.gen.go  | 16 +
 tools/golang/xenlight/types.gen.go|  6 +
 tools/include/libxl.h | 14 
 tools/libs/light/libxl.c  |  3 +++
 tools/libs/light/libxl_arch.h |  9 ++--
 tools/libs/light/libxl_arm.c  | 12 --
 tools/libs/light/libxl_create.c   | 23 +++
 tools/libs/light/libxl_types.idl  |  4 
 tools/libs/light/libxl_x86.c  | 42 +--
 tools/ocaml/libs/xc/xenctrl.ml|  7 ++
 tools/ocaml/libs/xc/xenctrl.mli   |  7 ++
 tools/ocaml/libs/xc/xenctrl_stubs.c   |  2 +-
 tools/xl/xl.c |  8 +++
 tools/xl/xl.h |  2 ++
 tools/xl/xl_info.c|  6 +++--
 tools/xl/xl_parse.c   | 16 +
 xen/arch/x86/domain.c | 28 ++-
 xen/arch/x86/hvm/vmx/vmcs.c   | 13 +++
 xen/arch/x86/hvm/vmx/vmx.c| 14 +---
 xen/arch/x86/include/asm/domain.h |  3 +++
 xen/arch/x86/include/asm/hvm/domain.h |  6 +
 xen/arch/x86/sysctl.c |  7 ++
 xen/arch/x86/traps.c  |  8 +++
 xen/include/public/arch-x86/xen.h |  2 ++
 xen/include/public/sysctl.h   |  8 ++-
 27 files changed, 255 insertions(+), 33 deletions(-)

-- 
2.11.0

Re: [PATCH 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

On 01/02/2022 10:02, Roger Pau Monné wrote:
> On Thu, Jan 27, 2022 at 04:01:33PM +0000, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted vitualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>> read/write accesses without taking a VM exit.
>>
>> Signed-off-by: Jane Malalane 
>> Suggested-by: Andrew Cooper 
>> ---
>> CC: Wei Liu 
>> CC: Anthony PERARD 
>> CC: Juergen Gross 
>> CC: Andrew Cooper 
>> CC: George Dunlap 
>> CC: Jan Beulich 
>> CC: Julien Grall 
>> CC: Stefano Stabellini 
>> CC: Christian Lindig 
>> CC: David Scott 
>> CC: Volodymyr Babchuk 
>> CC: "Roger Pau Monné" 
>> ---
>>   docs/man/xl.cfg.5.pod.in  | 10 
>>   docs/man/xl.conf.5.pod.in | 12 ++
>>   tools/golang/xenlight/helpers.gen.go  | 12 ++
>>   tools/libs/light/libxl_arch.h |  5 ++--
>>   tools/libs/light/libxl_arm.c  |  5 ++--
>>   tools/libs/light/libxl_create.c   | 21 ++---
>>   tools/libs/light/libxl_types.idl  |  2 ++
>>   tools/libs/light/libxl_x86.c  | 43 
>> +--
>>   tools/ocaml/libs/xc/xenctrl.ml|  2 ++
>>   tools/ocaml/libs/xc/xenctrl.mli   |  2 ++
>>   tools/xl/xl.c |  8 +++
>>   tools/xl/xl.h |  2 ++
>>   tools/xl/xl_parse.c   | 14 
>>   xen/arch/x86/domain.c | 27 +-
>>   xen/arch/x86/hvm/vmx/vmcs.c   |  4 
>>   xen/arch/x86/hvm/vmx/vmx.c| 13 +++
>>   xen/arch/x86/include/asm/hvm/domain.h |  6 +
>>   xen/arch/x86/traps.c  |  6 +++--
>>   xen/include/public/arch-x86/xen.h |  2 ++
>>   19 files changed, 174 insertions(+), 22 deletions(-)
>>
>> diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
>> index b98d161398..974fe7d2d8 100644
>> --- a/docs/man/xl.cfg.5.pod.in
>> +++ b/docs/man/xl.cfg.5.pod.in
>> @@ -1862,6 +1862,16 @@ firmware tables when using certain older guest 
>> Operating
>>   Systems. These tables have been superseded by newer constructs within
>>   the ACPI tables.
>>   
>> +=item B
>> +B<(x86 only)> Enables or disables hardware assisted virtualization for 
>> xapic.
>> +This allows accessing APIC registers without a VM-exit.
>> +The default is settable via L.
>> +
>> +=item B
>> +B<(x86 only)> Enables or disables hardware assisted virtualization for 
>> x2apic.
>> +This allows accessing APIC registers without a VM-exit.
>> +The default is settable via L.
> 
> Like you do below I would capitalize xAPIC and x2APIC in the option
> text.
> 
>> +
>>   =item B
>>   
>>   B<(x86 only)> Hides or exposes the No-eXecute capability. This allows a 
>> guest
>> diff --git a/docs/man/xl.conf.5.pod.in b/docs/man/xl.conf.5.pod.in
>> index df20c08137..2d0a59d019 100644
>> --- a/docs/man/xl.conf.5.pod.in
>> +++ b/docs/man/xl.conf.5.pod.in
>> @@ -107,6 +107,18 @@ Sets the default value for the C 
>> domain config value.
>>   
>>   Default: maximum grant version supported by the hypervisor.
>>   
>> +=item B
>> +
>> +If enabled, domains will use xAPIC hardware assisted emulation by default.
>> +
>> +Default: enabled.
>> +
>> +=item B
>> +
>> +If enabled, domains will use x2APIC hardware assisted emulation by default.
>> +
>> +Default: enabled.
> 
> I think for both options this should be:
> 
> Default: enabled if supported.
> 
>> +
>>   =item B
>>   
>>   Configures the default hotplug script used by virtual network devices.
>> diff --git a/tools/golang/xenlight/helpers.gen.go 
>> b/tools/golang/xenlight/helpers.gen.go
>> index dd4e6c9f14..90e7b9b205 100644
>> --- a/tools/golang/xenlight/helpers.gen.go
>> +++ b/tools/golang/xenlight/helpers.gen.go
>> @@ -636,6 +636,12 @@ x.Passthrough = Passthrough(xc.passthrough)
>>   if err := 
>> x.XendSuspendEvtchnCompat.fromC(_suspend_evtchn_compat);err != nil {
>>   return fmt.Errorf("converting field XendSuspen

Re: [PATCH 2/2] x86/xen: Allow per-domain usage of hardware virtualized APIC

[Jane Malalane]

Hi,
On 31/01/2022 12:05, Jan Beulich wrote:
> On 27.01.2022 17:01, Jane Malalane wrote:
>> Introduce a new per-domain creation x86 specific flag to
>> select whether hardware assisted virtualization should be used for
>> x{2}APIC.
>>
>> A per-domain option is added to xl in order to select the usage of
>> x{2}APIC hardware assisted vitualization, as well as a global
>> configuration option.
>>
>> Having all APIC interaction exit to Xen for emulation is slow and can
>> induce much overhead. Hardware can speed up x{2}APIC by running APIC
>> read/write accesses without taking a VM exit.
> 
> This is odd to read for a patch which makes it possible to _turn off_
> acceleration. Instead it would be interesting to know what problems
> you have encountered making it desirable to have a way to turn this off.
Makes sense.
> 
>> --- a/xen/arch/x86/hvm/vmx/vmx.c
>> +++ b/xen/arch/x86/hvm/vmx/vmx.c
>> @@ -3342,16 +3342,19 @@ static void vmx_install_vlapic_mapping(struct vcpu 
>> *v)
>>   
>>   void vmx_vlapic_msr_changed(struct vcpu *v)
>>   {
>> -int virtualize_x2apic_mode;
>> +int virtualize_xapic_mode, virtualize_x2apic_mode;
> 
> Please switch to bool as you touch and extend this.
> 
>>   struct vlapic *vlapic = vcpu_vlapic(v);
>>   unsigned int msr;
>>   
>> +virtualize_xapic_mode = ( cpu_has_vmx_virtualize_apic_accesses &&
>> +  v->domain->arch.hvm.assisted_xapic );
> 
> Please don't clone the bad use of blanks immediately inside parentheses
> here; instead, ...
> 
>>   virtualize_x2apic_mode = ( (cpu_has_vmx_apic_reg_virt ||
>>   cpu_has_vmx_virtual_intr_delivery) &&
>> -   cpu_has_vmx_virtualize_x2apic_mode );
>> +   cpu_has_vmx_virtualize_x2apic_mode &&
>> +   v->domain->arch.hvm.assisted_x2apic );
> 
> ... since you're touching this anyway, please consider correcting
> the style violation here.
> 
> However - do you need these expressions anymore? The per-domain fields
> can only be set if the respective CPU capabilities exit.
> 
>> --- a/xen/arch/x86/include/asm/hvm/domain.h
>> +++ b/xen/arch/x86/include/asm/hvm/domain.h
>> @@ -117,6 +117,12 @@ struct hvm_domain {
>>   
>>   bool   is_s3_suspended;
>>   
>> +/* xAPIC hardware assisted emulation. */
>> +bool assisted_xapic;
>> +
>> +/* x2APIC hardware assisted emulation. */
>> +bool assisted_x2apic;
>> +
>>   /* hypervisor intercepted msix table */
>>   struct list_head   msixtbl_list;
> 
> Please follow how adjacent code pads types / names here.
> 
>> --- a/xen/arch/x86/traps.c
>> +++ b/xen/arch/x86/traps.c
>> @@ -1115,7 +1115,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>   if ( !is_hvm_domain(d) || subleaf != 0 )
>>   break;
>>   
>> -if ( cpu_has_vmx_apic_reg_virt )
>> +if ( cpu_has_vmx_apic_reg_virt &&
>> + v->domain->arch.hvm.assisted_xapic )
>>   res->a |= XEN_HVM_CPUID_APIC_ACCESS_VIRT;
>>   
>>   /*
>> @@ -1126,7 +1127,8 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
>> uint32_t leaf,
>>*/
>>   if ( cpu_has_vmx_virtualize_x2apic_mode &&
>>cpu_has_vmx_apic_reg_virt &&
>> - cpu_has_vmx_virtual_intr_delivery )
>> + cpu_has_vmx_virtual_intr_delivery &&
>> + v->domain->arch.hvm.assisted_x2apic )
>>   res->a |= XEN_HVM_CPUID_X2APIC_VIRT;
> 
> Same remark as above - can't you now use _just_ the per-domain field?
> In this latter of the two cases this would then also mean bringing
> the CPU feature checks in line with what vmx_vlapic_msr_changed()
> does (as also pointed out for patch 1). Albeit it might be best to
> have a prereq patch fixing the issue, which could then be backported.
Indeed, that would be better.
Here I think the choice has been to only announce harwdare assisted 
x2apic support in CPUID if both read and write accesses are virtualized. 
I could either keep the x2apic_available check in Patch 1 as is and use 
the per-domain field here, and keep vmx_vlapic_msr_changed as is or vv.
> 
> Jan
> 
> 
Thank you,

Jane.

Re: [PATCH 1/2] xen+tools: Report Interrupt Controller Virtualization capabilities on x86

[Jane Malalane]

On 28/01/2022 17:04, Anthony PERARD wrote:
> On Thu, Jan 27, 2022 at 04:01:32PM +0000, Jane Malalane wrote:
>> Add XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_xapic and
>> XEN_SYSCTL_PHYSCAP_ARCH_ASSISTED_x2apic to report accelerated xapic
>> and x2apic, on x86 hardware.
>> No such features are currently implemented on AMD hardware.
>>
>> For that purpose, also add an arch-specific "capabilities" parameter
>> to struct xen_sysctl_physinfo.
>>
>> Signed-off-by: Jane Malalane 
>> Suggested-by: Andrew Cooper 
>> ---
>>   tools/golang/xenlight/helpers.gen.go |  4 
>>   tools/golang/xenlight/types.gen.go   |  6 ++
> 
> Note for committers: Please regenerate the go bindings, there are
> out-of-sync with libxl_types.idl at the moment.
> 
>> diff --git a/tools/libs/light/libxl_x86.c b/tools/libs/light/libxl_x86.c
>> index 1feadebb18..33da51fe89 100644
>> --- a/tools/libs/light/libxl_x86.c
>> +++ b/tools/libs/light/libxl_x86.c
>> @@ -866,6 +866,17 @@ int libxl__arch_passthrough_mode_setdefault(libxl__gc 
>> *gc,
>>   return rc;
>>   }
>>   
>> +void libxl__arch_get_physinfo(libxl_physinfo *physinfo,
>> +  xc_physinfo_t xcphysinfo)
> 
> It might be better to pass "xcphysinfo" as a pointer, otherwise I think
> a copy of the whole struct is made when calling this function.
> 
Will correct this, thanks.
> 
> In any case, the tool part of the patch looks good:
> Acked-by: Anthony PERARD 
> 
> Thanks,
> 

Jane.