Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Fri, Oct 11, 2019 at 09:11:54AM -0700, Jim Mattson wrote: > On Fri, Oct 11, 2019 at 12:48 AM Yang Weijiang > wrote: > > > > On Thu, Oct 10, 2019 at 02:42:51PM -0700, Jim Mattson wrote: > > > On Tue, Sep 17, 2019 at 1:52 AM Yang Weijiang > > > wrote: > > > > > > > > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > > > > Virtual Machine Monitor(VMM) to specify write-permission for guest > > > > physical memory at a sub-page(128 byte) granularity. When this > > > > capability is enabled, the CPU enforces write-access check for sub-pages > > > > within a 4KB page. > > > > > > > > The feature is targeted to provide fine-grained memory protection for > > > > usages such as device virtualization, memory check-point and VM > > > > introspection etc. > > > > > > > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > > > > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > > > > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > > > > called Sub-Page Permission Table Pointer (SPPTP) which contains a > > > > 4K-aligned physical address. > > > > > > > > To enable SPP for certain physical page, the gfn should be first mapped > > > > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > > > > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > > > > physical address to find out the sub-page permissions at the leaf entry. > > > > If the corresponding bit is set, write to sub-page is permitted, > > > > otherwise, SPP induced EPT violation is generated. > > > > > > How do you handle sub-page permissions for instructions emulated by kvm? > > How about checking if the gpa is SPP protected, if it is, inject some > > exception to guest? > The SPP semantics are well-defined. If a kvm-emulated instruction > tries to write to a sub-page that is write-protected, then an > SPP-induced EPT violation should be synthesized. Hi, Jim, Regarding the emulated instructions in KVM, there're quite a few instructions can write guest memory, such as MOVS, XCHG, INS etc., check each destination against SPP protected area would be trivial if deals with them individually, and PIO/MMIO induced vmexit/page_fault also can link to a SPP protected page, e.g., a string instruction's the destination is SPP protected memory. Is there a good way to intercept these writes? emulate_ops.write_emulated() is called in most of the emulation cases to check and write guest memory, but not sure it's suitable. Do you have any suggestion? Thanks!
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Fri, Oct 11, 2019 at 12:48 AM Yang Weijiang wrote: > > On Thu, Oct 10, 2019 at 02:42:51PM -0700, Jim Mattson wrote: > > On Tue, Sep 17, 2019 at 1:52 AM Yang Weijiang > > wrote: > > > > > > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > > > Virtual Machine Monitor(VMM) to specify write-permission for guest > > > physical memory at a sub-page(128 byte) granularity. When this > > > capability is enabled, the CPU enforces write-access check for sub-pages > > > within a 4KB page. > > > > > > The feature is targeted to provide fine-grained memory protection for > > > usages such as device virtualization, memory check-point and VM > > > introspection etc. > > > > > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > > > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > > > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > > > called Sub-Page Permission Table Pointer (SPPTP) which contains a > > > 4K-aligned physical address. > > > > > > To enable SPP for certain physical page, the gfn should be first mapped > > > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > > > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > > > physical address to find out the sub-page permissions at the leaf entry. > > > If the corresponding bit is set, write to sub-page is permitted, > > > otherwise, SPP induced EPT violation is generated. > > > > How do you handle sub-page permissions for instructions emulated by kvm? > How about checking if the gpa is SPP protected, if it is, inject some > exception to guest? The SPP semantics are well-defined. If a kvm-emulated instruction tries to write to a sub-page that is write-protected, then an SPP-induced EPT violation should be synthesized.
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Thu, Oct 10, 2019 at 02:42:51PM -0700, Jim Mattson wrote: > On Tue, Sep 17, 2019 at 1:52 AM Yang Weijiang wrote: > > > > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > > Virtual Machine Monitor(VMM) to specify write-permission for guest > > physical memory at a sub-page(128 byte) granularity. When this > > capability is enabled, the CPU enforces write-access check for sub-pages > > within a 4KB page. > > > > The feature is targeted to provide fine-grained memory protection for > > usages such as device virtualization, memory check-point and VM > > introspection etc. > > > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > > called Sub-Page Permission Table Pointer (SPPTP) which contains a > > 4K-aligned physical address. > > > > To enable SPP for certain physical page, the gfn should be first mapped > > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > > physical address to find out the sub-page permissions at the leaf entry. > > If the corresponding bit is set, write to sub-page is permitted, > > otherwise, SPP induced EPT violation is generated. > > How do you handle sub-page permissions for instructions emulated by kvm? How about checking if the gpa is SPP protected, if it is, inject some exception to guest?
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Tue, Sep 17, 2019 at 1:52 AM Yang Weijiang wrote: > > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > Virtual Machine Monitor(VMM) to specify write-permission for guest > physical memory at a sub-page(128 byte) granularity. When this > capability is enabled, the CPU enforces write-access check for sub-pages > within a 4KB page. > > The feature is targeted to provide fine-grained memory protection for > usages such as device virtualization, memory check-point and VM > introspection etc. > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > called Sub-Page Permission Table Pointer (SPPTP) which contains a > 4K-aligned physical address. > > To enable SPP for certain physical page, the gfn should be first mapped > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > physical address to find out the sub-page permissions at the leaf entry. > If the corresponding bit is set, write to sub-page is permitted, > otherwise, SPP induced EPT violation is generated. How do you handle sub-page permissions for instructions emulated by kvm?
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Tue, Sep 17, 2019 at 04:52:55PM +0800, Yang, Weijiang wrote:
Hi, Paolo,
Could you review this v5 patch at your convenience?
Thanks a lot!
> EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows
> Virtual Machine Monitor(VMM) to specify write-permission for guest
> physical memory at a sub-page(128 byte) granularity. When this
> capability is enabled, the CPU enforces write-access check for sub-pages
> within a 4KB page.
>
> The feature is targeted to provide fine-grained memory protection for
> usages such as device virtualization, memory check-point and VM
> introspection etc.
>
> SPP is active when the "sub-page write protection" (bit 23) is 1 in
> Secondary VM-Execution Controls. The feature is backed with a Sub-Page
> Permission Table(SPPT), SPPT is referenced via a 64-bit control field
> called Sub-Page Permission Table Pointer (SPPTP) which contains a
> 4K-aligned physical address.
>
> To enable SPP for certain physical page, the gfn should be first mapped
> to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry.
> While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset
> physical address to find out the sub-page permissions at the leaf entry.
> If the corresponding bit is set, write to sub-page is permitted,
> otherwise, SPP induced EPT violation is generated.
>
> This patch serial passed SPP function test and selftest on Ice-Lake platform.
>
> Please refer to the SPP introduction document in this patch set and
> Intel SDM for details:
>
> Intel SDM:
> https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf
>
> SPP selftest patch:
> https://lkml.org/lkml/2019/6/18/1197
>
> Previous patch:
> https://lkml.org/lkml/2019/8/14/97
>
> Patch 1: Introduction to SPP.
> Patch 2: Add SPP related flags and control bits.
> Patch 3: Functions for SPPT setup.
> Patch 4: Add SPP access bitmaps for memslots.
> Patch 5: Introduce SPP {init,set,get} functions
> Patch 6: Implement User space access IOCTLs.
> Patch 7: Set up SPP paging table at vm-entry/exit.
> Patch 8: Enable lazy mode SPPT setup.
> Patch 9: Handle SPP protected pages when VM memory changes
>
>
> Change logs:
>
> V5 -> V4:
> 1. Enable SPP support for Hugepage(1GB/2MB) to extend application.
> 2. Make SPP miss vm-exit handler as the unified place to set up SPPT.
> 3. If SPP protected pages are access-tracked or dirty-page-tracked,
> store SPP flag in reserved address bit, restore it in
> fast_page_fault() handler.
> 4. Move SPP specific functions to vmx/spp.c and vmx/spp.h
> 5. Rebased code to kernel v5.3
> 6. Other change suggested by KVM community.
>
> V3 -> V4:
> 1. Modified documentation to make it consistent with patches.
> 2. Allocated SPPT root page in init_spp() instead of vmx_set_cr3() to
> avoid SPPT miss error.
> 3. Added back co-developers and sign-offs.
>
> V2 -> V3:
> 1. Rebased patches to kernel 5.1 release
> 2. Deferred SPPT setup to EPT fault handler if the page is not
> available while set_subpage() is being called.
> 3. Added init IOCTL to reduce extra cost if SPP is not used.
> 4. Refactored patch structure, cleaned up cross referenced functions.
> 5. Added code to deal with memory swapping/migration/shrinker cases.
>
> V2 -> V1:
> 1. Rebased to 4.20-rc1
> 2. Move VMCS change to a separated patch.
> 3. Code refine and Bug fix
>
>
> Yang Weijiang (9):
> Documentation: Introduce EPT based Subpage Protection
> vmx: spp: Add control flags for Sub-Page Protection(SPP)
> mmu: spp: Add SPP Table setup functions
> mmu: spp: Add functions to create/destroy SPP bitmap block
> mmu: spp: Introduce SPP {init,set,get} functions
> x86: spp: Introduce user-space SPP IOCTLs
> vmx: spp: Set up SPP paging table at vm-entry/exit
> mmu: spp: Enable Lazy mode SPP protection
> mmu: spp: Handle SPP protected pages when VM memory changes
>
> Documentation/virtual/kvm/spp_kvm.txt | 178 +++
> arch/x86/include/asm/cpufeatures.h| 1 +
> arch/x86/include/asm/kvm_host.h | 10 +-
> arch/x86/include/asm/vmx.h| 10 +
> arch/x86/include/uapi/asm/vmx.h | 2 +
> arch/x86/kernel/cpu/intel.c | 4 +
> arch/x86/kvm/mmu.c| 78 ++-
> arch/x86/kvm/mmu.h| 2 +
> arch/x86/kvm/vmx/capabilities.h | 5 +
> arch/x86/kvm/vmx/spp.c| 651 ++
> arch/x86/kvm/vmx/spp.h| 27 ++
> arch/x86/kvm/vmx/vmx.c| 99
> arch/x86/kvm/x86.c| 51 ++
> include/uapi/linux/kvm.h | 17 +
> 14 files changed, 1133 insertions(+), 2 deletions(-)
> create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
> create mode 100644 arch/x86/kvm/vmx/spp.c
> create mode 100644 arch/x86/kvm/vmx/spp.h
>
> --
> 2.17.2
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Tue, 17 Sep 2019 08:59:04 -0400, Konrad Rzeszutek Wilk wrote: > On Tue, Sep 17, 2019 at 04:52:55PM +0800, Yang Weijiang wrote: > > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > > Virtual Machine Monitor(VMM) to specify write-permission for guest > > physical memory at a sub-page(128 byte) granularity. When this > > capability is enabled, the CPU enforces write-access check for sub-pages > > within a 4KB page. > > > > The feature is targeted to provide fine-grained memory protection for > > usages such as device virtualization, memory check-point and VM > > introspection etc. > > > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > > called Sub-Page Permission Table Pointer (SPPTP) which contains a > > 4K-aligned physical address. > > > > To enable SPP for certain physical page, the gfn should be first mapped > > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > > physical address to find out the sub-page permissions at the leaf entry. > > If the corresponding bit is set, write to sub-page is permitted, > > otherwise, SPP induced EPT violation is generated. > > > > This patch serial passed SPP function test and selftest on Ice-Lake > > platform. > > > > Please refer to the SPP introduction document in this patch set and > > Intel SDM for details: > > > > Intel SDM: > > https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf > > > > SPP selftest patch: > > https://lkml.org/lkml/2019/6/18/1197 > > > > Previous patch: > > https://lkml.org/lkml/2019/8/14/97 > > I saw the patches as part of the introspection patch-set. > Are you all working together on this? Weijiang helped us to start using the SPP feature with the introspection API and tested the integration when we didn't had the hardware available. I've included the SPP patches in the introspection patch series in order to "show the full picture". > Would it be possible for some of the bitdefender folks who depend on this > to provide Tested-by adn could they also take the time to review this > patch-set? Sure. Once we rebase the introspection patches on 5.3, we'll replace the previous version this new one in our tree and test it.
Re: [PATCH v5 0/9] Enable Sub-page Write Protection Support
On Tue, Sep 17, 2019 at 04:52:55PM +0800, Yang Weijiang wrote: > EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows > Virtual Machine Monitor(VMM) to specify write-permission for guest > physical memory at a sub-page(128 byte) granularity. When this > capability is enabled, the CPU enforces write-access check for sub-pages > within a 4KB page. > > The feature is targeted to provide fine-grained memory protection for > usages such as device virtualization, memory check-point and VM > introspection etc. > > SPP is active when the "sub-page write protection" (bit 23) is 1 in > Secondary VM-Execution Controls. The feature is backed with a Sub-Page > Permission Table(SPPT), SPPT is referenced via a 64-bit control field > called Sub-Page Permission Table Pointer (SPPTP) which contains a > 4K-aligned physical address. > > To enable SPP for certain physical page, the gfn should be first mapped > to a 4KB entry, then set bit 61 of the corresponding EPT leaf entry. > While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset > physical address to find out the sub-page permissions at the leaf entry. > If the corresponding bit is set, write to sub-page is permitted, > otherwise, SPP induced EPT violation is generated. > > This patch serial passed SPP function test and selftest on Ice-Lake platform. > > Please refer to the SPP introduction document in this patch set and > Intel SDM for details: > > Intel SDM: > https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf > > SPP selftest patch: > https://lkml.org/lkml/2019/6/18/1197 > > Previous patch: > https://lkml.org/lkml/2019/8/14/97 I saw the patches as part of the introspection patch-set. Are you all working together on this? Would it be possible for some of the bitdefender folks who depend on this to provide Tested-by adn could they also take the time to review this patch-set? Thanks.
