From: Joerg Roedel
When a #VC exception is triggered by user-space the instruction decoder
needs to read the instruction bytes from user addresses. Enhance
vc_decode_insn() to safely fetch kernel and user instructions.
Signed-off-by: Joerg Roedel
Link:
From: Tom Lendacky
As part of the GHCB specification, the booting of APs under SEV-ES
requires an AP jump table when transitioning from one layer of code to
another (e.g. when going from UEFI to the OS). As a result, each layer
that parks an AP must provide the physical address of an AP jump
From: Tom Lendacky
Implement a handler for #VC exceptions caused by VMMCALL instructions.
This patch is only a starting point, VMMCALL emulation under SEV-ES
needs further hypervisor-specific changes to provide additional state.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC
From: Tom Lendacky
Implement a handler for #VC exceptions caused by MONITOR and MONITORX
instructions.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Tom Lendacky
Implement a handler for #VC exceptions caused by RDMSR/WRMSR
instructions.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Add a play_dead handler when running under SEV-ES. This is needed
because the hypervisor can't deliver an SIPI request to restart the AP.
Instead the kernel has to issue a VMGEXIT to halt the VCPU until the
hypervisor wakes it up again.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
The APs are not ready to handle exceptions when verify_cpu() is called
in secondary_startup_64.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-71-j...@8bytes.org
---
arch/x86/include/asm/realmode.h | 1 +
From: Tom Lendacky
Implement a handler for #VC exceptions caused by RDPMC instructions.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Doug Covelli
Add VMware specific handling for #VC faults caused by VMMCALL
instructions.
Signed-off-by: Doug Covelli
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: - Adapt to different paravirt interface ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Refactor the message printed to the kernel log which indicates whether
SEV or SME is active to print a list of enabled encryption features.
This will scale better in the future when more memory encryption
features might be added. Also add SEV-ES to the list of features.
From: Tom Lendacky
The runtime handler needs a GHCB per CPU. Set them up and map them
unencrypted.
Signed-off-by: Tom Lendacky
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-44-j...@8bytes.org
---
arch/x86/include/asm/mem_encrypt.h | 2 ++
From: Joerg Roedel
Add the infrastructure to handle #VC exceptions when the kernel runs
on virtual addresses and has a GHCB mapped. This handler will be used
until the runtime #VC handler takes over.
Signed-off-by: Joerg Roedel
Link:
From: Tom Lendacky
Add handler for VC exceptions caused by MMIO intercepts. These
intercepts come along as nested page faults on pages with reserved
bits set.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to VC handling framework ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg
From: Joerg Roedel
Implement a handler for #VC exceptions caused by #AC exceptions. The #AC
exception is just forwarded to do_alignment_check() and not pushed down
to the hypervisor, as requested by the SEV-ES GHCB Standardization
Specification.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Allocate and map an IST stack and an additional fall-back stack for
the #VC handler. The memory for the stacks is allocated only when
SEV-ES is active.
The #VC handler needs to use an IST stack because it could be raised
from kernel space with unsafe stack, e.g. in the
From: Joerg Roedel
Handle #VC exceptions caused by #DB exceptions in the guest. Those
must be handled outside of instrumentation_begin()/end() so that the
handler will not be raised recursivly.
Handle them by calling the kernels debug exception handler.
Signed-off-by: Joerg Roedel
Link:
From: Tom Lendacky
Add the handlers for #VC exceptions invoked at runtime.
Signed-off-by: Tom Lendacky
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-49-j...@8bytes.org
---
arch/x86/include/asm/idtentry.h | 6 +
arch/x86/kernel/idt.c | 11 +-
From: Joerg Roedel
Setup an early handler for #VC exceptions. There is no GHCB mapped
yet, so just re-use the vc_no_ghcb_handler. It can only handle CPUID
exit-codes, but that should be enough to get the kernel through
verify_cpu() and __startup_64() until it runs on virtual addresses.
From: Joerg Roedel
Re-use the handlers for CPUID and IOIO caused #VC exceptions in the
early boot handler.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-50-j...@8bytes.org
---
arch/x86/kernel/sev-es-shared.c | 7 +++
arch/x86/kernel/sev-es.c| 6
From: Tom Lendacky
Implement a handler for #VC exceptions caused by MWAIT and MWAITX
instructions.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Add handling for emulation the MOVS instruction on MMIO regions, as done
by the memcpy_toio() and memcpy_fromio() functions.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-53-j...@8bytes.org
---
arch/x86/kernel/sev-es.c | 77
From: Joerg Roedel
When running under SEV-ES the kernel has to tell the hypervisor when to
open the NMI window again after an NMI was injected. This is done with
an NMI-complete message to the hypervisor.
Add code to the kernels NMI handler to send this message right at the
beginning of
From: Joerg Roedel
For SEV-ES this entry point will be used for restarting APs after they
have been offlined. Remove the '0' from the name to reflect that.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-72-j...@8bytes.org
---
From: Joerg Roedel
Add two new paravirt callbacks to provide hypervisor specific processor
state in the GHCB and to copy state from the hypervisor back to the
processor.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-65-j...@8bytes.org
---
From: Tom Lendacky
Implement the callbacks to copy the processor state required by KVM to
the GHCB.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: - Split out of a larger patch
- Adapt to different callback functions ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg
From: Joerg Roedel
The get_stack_info functionality is needed in the entry code for the #VC
exception handler. Provide a version of it in the .text.noinstr
section which can be called safely from there.
Signed-off-by: Joerg Roedel
Link:
From: Tom Lendacky
Implement a handler for #VC exceptions caused by RDTSC and RDTSCP
instructions. Also make it available in the pre-decompression stage
because the KASLR code used RDTSC/RDTSCP to gather entropy and some
hypervisors intercept these instructions.
Signed-off-by: Tom Lendacky
[
From: Joerg Roedel
When an NMI hits in the #VC handler entry code before it switched to
another stack, any subsequent #VC exception in the NMI code-path will
overwrite the interrupted #VC handlers stack.
Make sure this doesn't happen by explicitly adjusting the #VC IST entry
in the NMI handler
From: Tom Lendacky
Implement a handler for #VC exceptions caused by INVD instructions.
Since Linux should never use INVD, just mark it as unsupported.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling infrastructure ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg
From: Joerg Roedel
The code at the trampoline entry point is executed in real-mode. In
real-mode #VC exceptions can't be handled, so anything that might cause
such an exception must be avoided.
In the standard trampoline entry code this is the WBINVD instruction and
the call to verify_cpu(),
From: Joerg Roedel
The #VC handler needs special entry code because:
1. It runs on an IST stack
2. It needs to be able to handle nested #VC exceptions
To make this work the entry code is implemented to pretend it doesn't
use an IST stack. When entered from user-mode or early
From: Tom Lendacky
Calling down to EFI runtime services can result in the firmware performing
VMGEXIT calls. The firmware is likely to use the GHCB of the OS (e.g., for
setting EFI variables), so each GHCB in the system needs to be identity
mapped in the EFI page tables, as unencrypted, to avoid
From: Tom Lendacky
Implement a handler for #VC exceptions caused by WBINVD instructions.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Set up the TSS for starting APs before they are kicked. This allows the
APs to use IST in early exception handling.
Also load the TSS early if the TSS entry in the GDT is present. This
makes sure a TSS is only loaded when it has been set up.
Signed-off-by: Joerg Roedel
From: Tom Lendacky
Add code to handle #VC exceptions on DR7 register reads and writes.
This is needed early because show_regs() reads DR7 to print it out.
Under SEV-ES there is currently no support for saving/restoring the
DRx registers, but software expects to be able to write to the DR7
From: Martin Radev
Make sure the machine supports RDRAND, otherwise there is no trusted
source of of randomness in the system.
To also check this in the pre-decompression stage, make has_cpuflag
not depend on CONFIG_RANDOMIZE_BASE anymore.
Signed-off-by: Martin Radev
Signed-off-by: Joerg
From: Joerg Roedel
Early exception handling will use rd/wrgsbase in paranoid_entry/exit.
Enable the feature to avoid #UD exceptions on boot APs.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-38-j...@8bytes.org
---
arch/x86/kernel/head_64.S | 7 +++
1 file
From: Joerg Roedel
Add code needed to setup an IDT in the early pre-decompression
boot-code. The IDT is loaded first in startup_64, which is after
EfiExitBootServices() has been called, and later reloaded when the
kernel image has been relocated to the end of the decompression area.
This allows
From: Joerg Roedel
The functions are needed to map the GHCB for SEV-ES guests. The GHCB is
used for communication with the hypervisor, so its content must not be
encrypted. After the GHCB is not needed anymore it must be mapped
encrypted again so that the running kernel image can safely re-use
From: Joerg Roedel
The file contains only code related to identity mapped page-tables.
Rename the file and compile it always in.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-14-j...@8bytes.org
---
arch/x86/boot/compressed/Makefile
From: Joerg Roedel
Building a correct GHCB for the hypervisor requires setting valid bits
in the GHCB. Simplify that process by providing accessor functions to
set values and to update the valid bitmap and to check the valid bitmap
in KVM.
Signed-off-by: Joerg Roedel
---
From: Joerg Roedel
Factor out the code used to decode an instruction with the correct
address and operand sizes to a helper function.
No functional changes.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-9-j...@8bytes.org
---
arch/x86/include/asm/insn-eval.h
From: Joerg Roedel
Make sure segments are properly set up before setting up an IDT and
doing anything that might cause a #VC exception. This is later needed
for early exception handling.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link:
From: Joerg Roedel
When stack-protector is enabled a valid GS_BASE is needed before
calling any C code function, because the stack canary is loaded from
per-cpu data.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-31-j...@8bytes.org
---
From: Joerg Roedel
When booted through startup_64 the kernel keeps running on the EFI
page-table until the KASLR code sets up its own page-table. Without
KASLR the pre-decompression boot code never switches off the EFI
page-table. Change that by unconditionally switching to a kernel
controlled
From: Tom Lendacky
Add support for decoding and handling #VC exceptions for IOIO events.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapted code to #VC handling framework ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
The xgetbv() function is needed in pre-decompression boot code, but
asm/fpu/internal.h can't be included there directly. Doing so opens
the door to include-hell due to various include-magic in
boot/compressed/misc.h.
Avoid that by moving xgetbv()/xsetbv() to a separate header
From: Joerg Roedel
Handling exceptions during boot requires a working GDT. The kernel GDT
can't be used on the direct mapping, so load a startup GDT and setup
segments.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-30-j...@8bytes.org
---
From: Joerg Roedel
With SEV-ES, exception handling is needed very early, even before the
kernel has cleared the bss segment. In order to prevent clearing the
currently used IDT, move the IDT to the data segment.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link:
From: Joerg Roedel
Add the sev_es_active function for checking whether SEV-ES is enabled.
Also cache the value of MSR_AMD64_SEV at boot to speed up the feature
checking in the running code.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link:
From: Tom Lendacky
Extend the vmcb_safe_area with SEV-ES fields and add a new
'struct ghcb' which will be used for guest-hypervisor communication.
Signed-off-by: Tom Lendacky
Signed-off-by: Joerg Roedel
---
arch/x86/include/asm/svm.h | 45 +-
From: Joerg Roedel
Move the definition of the x86 page-fault error code bits to the new
header file asm/trap_pf.h. This makes it easier to include them into
pre-decompression boot code. No functional changes.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Load the GDT right after switching to virtual addresses to make sure
there is a defined GDT for exception handling.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-32-j...@8bytes.org
---
arch/x86/kernel/head_64.S | 16
From: Joerg Roedel
The function can fail to create an identity mapping, check for that
and bail out if it happens.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-21-j...@8bytes.org
---
arch/x86/boot/compressed/ident_map_64.c | 7 +--
1 file changed, 5
From: Joerg Roedel
Make sure there is a stack once the kernel runs from virual addresses.
At this stage any secondary CPU which boots will have lost its stack
because the kernel switched to a new page-table which does not map the
real-mode stack anymore.
This is needed for handling early #VC
From: Joerg Roedel
Move the assembly coded dispatch between page-faults and all other
exceptions to C code to make it easier to maintain and extend.
Also change the return-type of early_make_pgtable() to bool and make it
static.
Signed-off-by: Joerg Roedel
Link:
From: Borislav Petkov
Use the shorthand to make it more readable.
No functional changes.
Signed-off-by: Borislav Petkov
Signed-off-by: Joerg Roedel
---
arch/x86/include/asm/svm.h | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/svm.h
From: Joerg Roedel
Factor out the code to fetch the instruction from user-space to a helper
function.
No functional changes.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-8-j...@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 2 ++
arch/x86/kernel/umip.c
From: Joerg Roedel
Also move it to a header file so that it can be used in the idt code
to setup the early IDT.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link: https://lore.kernel.org/r/20200724160336.5435-35-j...@8bytes.org
---
arch/x86/include/asm/setup.h | 10 ++
From: Joerg Roedel
With the page-fault handler in place the identity mapping can be built
on-demand. So remove the code which manually creates the mappings and
unexport/remove the functions used for it.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link:
From: Joerg Roedel
The code to setup idt_data is needed for early exception handling, but
set_intr_gate() can't be used that early because it has pv-ops in its
code path, which don't work that early.
Split out the idt_data initialization part from set_intr_gate() so
that it can be used
From: Tom Lendacky
Handle #VC exceptions caused by CPUID instructions. These happen in
early boot code when the KASLR code checks for RDTSC.
Signed-off-by: Tom Lendacky
[ jroe...@suse.de: Adapt to #VC handling framework ]
Co-developed-by: Joerg Roedel
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Hi,
here is the new version of the SEV-ES client enabling patch-set. It is
based on the latest tip/master branch and contains the necessary
changes. In particular those ar:
- Enabling CR4.FSGSBASE early on supported processors so that
early #VC exceptions
From: Joerg Roedel
Load the IDT right after switching to virtual addresses in head_64.S
so that the kernel can handle #VC exceptions.
Signed-off-by: Joerg Roedel
Link: https://lore.kernel.org/r/20200724160336.5435-36-j...@8bytes.org
---
arch/x86/include/asm/setup.h | 3 +++
On Mon, Aug 24, 2020 at 10:53:57AM +0200, Joerg Roedel wrote:
> static inline void __unused_size_checks(void)
> {
> - BUILD_BUG_ON(sizeof(struct vmcb_save_area) != 0x298);
> + BUILD_BUG_ON(sizeof(struct vmcb_save_area) != 1032);
> BUILD_BUG_ON(sizeof(struct vmcb_control_area) !=
From: Joerg Roedel
Add the first handler for #VC exceptions. At stage 1 there is no GHCB
yet becaue the kernel might still be running on the EFI page table.
The stage 1 handler is limited to the MSR based protocol to talk to
the hypervisor and can only support CPUID exit-codes, but that is
From: Joerg Roedel
Do not allocate a vmcb_control_area and a vmcb_save_area on the stack,
as these structures will become larger with future extenstions of
SVM and thus the svm_set_nested_state() function will become a too large
stack frame.
Signed-off-by: Joerg Roedel
---
From: Joerg Roedel
Force a page-fault on any further accesses to the GHCB page when they
shouldn't happen anymore. This will catch the bugs where a #VC exception
is raised when no one is expected anymore.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Add a function to the instruction decoder which returns the pt_regs
offset of the register specified in the reg field of the modrm byte.
Signed-off-by: Joerg Roedel
Acked-by: Masami Hiramatsu
Link: https://lore.kernel.org/r/20200724160336.5435-10-j...@8bytes.org
---
From: Joerg Roedel
Install a page-fault handler to add an identity mapping to addresses
not yet mapped. Also do some checking whether the error code is sane.
This makes non SEV-ES machines use the exception handling
infrastructure in the pre-decompressions boot code too, making it less
likely
From: Tom Lendacky
Add CPU feature detection for Secure Encrypted Virtualization with
Encrypted State. This feature enhances SEV by also encrypting the
guest register state, making it in-accessible to the hypervisor.
Signed-off-by: Tom Lendacky
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
The x86-64 ABI defines a red-zone on the stack:
The 128-byte area beyond the location pointed to by %rsp is considered
to be reserved and shall not be modified by signal or interrupt
handlers. Therefore, functions may use this area for temporary data
that is not
From: Joerg Roedel
Add a function to check whether an instruction has a REP prefix.
Signed-off-by: Joerg Roedel
Reviewed-by: Masami Hiramatsu
Link: https://lore.kernel.org/r/20200724160336.5435-11-j...@8bytes.org
---
arch/x86/include/asm/insn-eval.h | 1 +
arch/x86/lib/insn-eval.c |
From: Joerg Roedel
Install an exception handler for #VC exception that uses a GHCB. Also
add the infrastructure for handling different exit-codes by decoding
the instruction that caused the exception and error handling.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
The inat-tables.c file has some arrays in it that contain pointers to
other arrays. These pointers need to be relocated when the kernel
image is moved to a different location.
The pre-decompression boot-code has no support for applying ELF
relocations, so initialize these
From: Joerg Roedel
Changing the function to take start and end as parameters instead of
start and size simplifies the callers, which don't need to calculate
the size if they already have start and end.
Signed-off-by: Joerg Roedel
Reviewed-by: Kees Cook
Link:
From: Joerg Roedel
Call set_sev_encryption_mask() while still on the stage 1 #VC-handler,
because the stage 2 handler needs our own page-tables to be set up, to
which calling set_sev_encryption_mask() is a prerequisite.
Signed-off-by: Joerg Roedel
Link:
From: Joerg Roedel
Setup sev-es.c and include the code from the
pre-decompression stage to also build it into the image of the running
kernel. Temporarily add __maybe_unused annotations to avoid build
warnings until the functions get used.
Signed-off-by: Joerg Roedel
Link:
From: Xin He
[ Upstream commit 836b194d65782aaec4485a07d2aab52d3f698505 ]
Before setting shmem->pages to NULL, kfree() should
be called.
Signed-off-by: Xin He
Reviewed-by: Qi Liu
Link:
http://patchwork.freedesktop.org/patch/msgid/20200722051851.72662-1-hexin...@bytedance.com
Signed-off-by:
From: Xin He
[ Upstream commit 836b194d65782aaec4485a07d2aab52d3f698505 ]
Before setting shmem->pages to NULL, kfree() should
be called.
Signed-off-by: Xin He
Reviewed-by: Qi Liu
Link:
http://patchwork.freedesktop.org/patch/msgid/20200722051851.72662-1-hexin...@bytedance.com
Signed-off-by:
On Mon, Aug 24, 2020 at 11:06:51AM -0400, Konrad Rzeszutek Wilk wrote:
> So it fails at
>
> 683 dev_WARN_ONCE(dev, 1,
>
> 684 "swiotlb addr %pad+%zu overflow (mask %llx, bus
> limit %llx).\n",
> 685
On Thu, Aug 06, 2020 at 03:46:23AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit:47ec5303 Merge git://git.kernel.org/pub/scm/linux/kernel/g..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=16fe1dea90
>
On Mon, Aug 24, 2020 at 5:07 PM Konrad Rzeszutek Wilk
wrote:
>
> On Thu, Aug 06, 2020 at 03:46:23AM -0700, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit:47ec5303 Merge git://git.kernel.org/pub/scm/linux/kernel/g..
> > git tree: upstream
> >
83 matches
Mail list logo
