Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On Wed, Dec 15, 2021 at 01:00:38PM +0800, Tianyu Lan wrote: > > > On 12/15/2021 6:40 AM, Dave Hansen wrote: > > On 12/14/21 2:23 PM, Tom Lendacky wrote: > > > > I don't really understand how this can be more general any *not* get > > > > utilized by the existing SEV support. > > > > > > The Virtual Top-of-Memory (VTOM) support is an SEV-SNP feature that is > > > meant to be used with a (relatively) un-enlightened guest. The idea is > > > that the C-bit in the guest page tables must be 0 for all accesses. It > > > is only the physical address relative to VTOM that determines if the > > > access is encrypted or not. So setting sme_me_mask will actually cause > > > issues when running with this feature. Since all DMA for an SEV-SNP > > > guest must still be to shared (unencrypted) memory, some enlightenment > > > is needed. In this case, memory mapped above VTOM will provide that via > > > the SWIOTLB update. For SEV-SNP guests running with VTOM, they are > > > likely to also be running with the Reflect #VC feature, allowing a > > > "paravisor" to handle any #VCs generated by the guest. > > > > > > See sections 15.36.8 "Virtual Top-of-Memory" and 15.36.9 "Reflect #VC" > > > in volume 2 of the AMD APM [1]. > > > > Thanks, Tom, that's pretty much what I was looking for. > > > > The C-bit normally comes from the page tables. But, the hardware also > > provides an alternative way to effectively get C-bit behavior without > > actually setting the bit in the page tables: Virtual Top-of-Memory > > (VTOM). Right? > > > > It sounds like Hyper-V has chosen to use VTOM instead of requiring the > > guest to do the C-bit in its page tables. > > > > But, the thing that confuses me is when you said: "it (VTOM) is meant to > > be used with a (relatively) un-enlightened guest". We don't have an > > unenlightened guest here. We have Linux, which is quite enlightened. > > > > > Is VTOM being used because there's something that completely rules out > > > using the C-bit in the page tables? What's that "something"? > > > For "un-enlightened" guest, there is an another system running insider > the VM to emulate some functions(tsc, timer, interrupt and so on) and > this helps not to modify OS(Linux/Windows) a lot. In Hyper-V Isolation > VM, we called the new system as HCL/paravisor. HCL runs in the VMPL0 and > Linux runs in VMPL2. This is similar with nested virtualization. HCL > plays similar role as L1 hypervisor to emulate some general functions > (e.g, rdmsr/wrmsr accessing and interrupt injection) which needs to be > enlightened in the enlightened guest. Linux kernel needs to handle > #vc/#ve exception directly in the enlightened guest. HCL handles such > exception in un-enlightened guest and emulate interrupt injection which > helps not to modify OS core part code. Using vTOM also is same purpose. > Hyper-V uses vTOM avoid changing page table related code in OS(include > Windows and Linux)and just needs to map memory into decrypted address > space above vTOM in the driver code. > > Linux has generic swiotlb bounce buffer implementation and so introduce > swiotlb_unencrypted_base here to set shared memory boundary or vTOM. > Hyper-V Isolation VM is un-enlightened guest. Hyper-V doesn't expose sev/sme > capability to guest and so SEV code actually doesn't work. > So we also can't interact current existing SEV code and these code is > for enlightened guest support without HCL/paravisor. If other platforms > or SEV want to use similar vTOM feature, swiotlb_unencrypted_base can > be reused. So swiotlb_unencrypted_base is a general solution for all > platforms besides SEV and Hyper-V. > Thanks for the detailed explanation. Dave, are you happy with this? The code looks pretty solid to my untrained eyes. And the series has collected necessary acks from stakeholders. If I don't hear objection by EOD Friday I will apply this series to hyperv-next. Wei. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/15/2021 6:40 AM, Dave Hansen wrote: On 12/14/21 2:23 PM, Tom Lendacky wrote: I don't really understand how this can be more general any *not* get utilized by the existing SEV support. The Virtual Top-of-Memory (VTOM) support is an SEV-SNP feature that is meant to be used with a (relatively) un-enlightened guest. The idea is that the C-bit in the guest page tables must be 0 for all accesses. It is only the physical address relative to VTOM that determines if the access is encrypted or not. So setting sme_me_mask will actually cause issues when running with this feature. Since all DMA for an SEV-SNP guest must still be to shared (unencrypted) memory, some enlightenment is needed. In this case, memory mapped above VTOM will provide that via the SWIOTLB update. For SEV-SNP guests running with VTOM, they are likely to also be running with the Reflect #VC feature, allowing a "paravisor" to handle any #VCs generated by the guest. See sections 15.36.8 "Virtual Top-of-Memory" and 15.36.9 "Reflect #VC" in volume 2 of the AMD APM [1]. Thanks, Tom, that's pretty much what I was looking for. The C-bit normally comes from the page tables. But, the hardware also provides an alternative way to effectively get C-bit behavior without actually setting the bit in the page tables: Virtual Top-of-Memory (VTOM). Right? It sounds like Hyper-V has chosen to use VTOM instead of requiring the guest to do the C-bit in its page tables. But, the thing that confuses me is when you said: "it (VTOM) is meant to be used with a (relatively) un-enlightened guest". We don't have an unenlightened guest here. We have Linux, which is quite enlightened. Is VTOM being used because there's something that completely rules out using the C-bit in the page tables? What's that "something"? For "un-enlightened" guest, there is an another system running insider the VM to emulate some functions(tsc, timer, interrupt and so on) and this helps not to modify OS(Linux/Windows) a lot. In Hyper-V Isolation VM, we called the new system as HCL/paravisor. HCL runs in the VMPL0 and Linux runs in VMPL2. This is similar with nested virtualization. HCL plays similar role as L1 hypervisor to emulate some general functions (e.g, rdmsr/wrmsr accessing and interrupt injection) which needs to be enlightened in the enlightened guest. Linux kernel needs to handle #vc/#ve exception directly in the enlightened guest. HCL handles such exception in un-enlightened guest and emulate interrupt injection which helps not to modify OS core part code. Using vTOM also is same purpose. Hyper-V uses vTOM avoid changing page table related code in OS(include Windows and Linux)and just needs to map memory into decrypted address space above vTOM in the driver code. Linux has generic swiotlb bounce buffer implementation and so introduce swiotlb_unencrypted_base here to set shared memory boundary or vTOM. Hyper-V Isolation VM is un-enlightened guest. Hyper-V doesn't expose sev/sme capability to guest and so SEV code actually doesn't work. So we also can't interact current existing SEV code and these code is for enlightened guest support without HCL/paravisor. If other platforms or SEV want to use similar vTOM feature, swiotlb_unencrypted_base can be reused. So swiotlb_unencrypted_base is a general solution for all platforms besides SEV and Hyper-V. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/14/21 2:23 PM, Tom Lendacky wrote: >> I don't really understand how this can be more general any *not* get >> utilized by the existing SEV support. > > The Virtual Top-of-Memory (VTOM) support is an SEV-SNP feature that is > meant to be used with a (relatively) un-enlightened guest. The idea is > that the C-bit in the guest page tables must be 0 for all accesses. It > is only the physical address relative to VTOM that determines if the > access is encrypted or not. So setting sme_me_mask will actually cause > issues when running with this feature. Since all DMA for an SEV-SNP > guest must still be to shared (unencrypted) memory, some enlightenment > is needed. In this case, memory mapped above VTOM will provide that via > the SWIOTLB update. For SEV-SNP guests running with VTOM, they are > likely to also be running with the Reflect #VC feature, allowing a > "paravisor" to handle any #VCs generated by the guest. > > See sections 15.36.8 "Virtual Top-of-Memory" and 15.36.9 "Reflect #VC" > in volume 2 of the AMD APM [1]. Thanks, Tom, that's pretty much what I was looking for. The C-bit normally comes from the page tables. But, the hardware also provides an alternative way to effectively get C-bit behavior without actually setting the bit in the page tables: Virtual Top-of-Memory (VTOM). Right? It sounds like Hyper-V has chosen to use VTOM instead of requiring the guest to do the C-bit in its page tables. But, the thing that confuses me is when you said: "it (VTOM) is meant to be used with a (relatively) un-enlightened guest". We don't have an unenlightened guest here. We have Linux, which is quite enlightened. Is VTOM being used because there's something that completely rules out using the C-bit in the page tables? What's that "something"? ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/14/21 12:40 PM, Dave Hansen wrote: On 12/13/21 8:36 PM, Tianyu Lan wrote: On 12/14/2021 12:45 AM, Dave Hansen wrote: On 12/12/21 11:14 PM, Tianyu Lan wrote: In Isolation VM with AMD SEV, bounce buffer needs to be accessed via extra address space which is above shared_gpa_boundary (E.G 39 bit address line) reported by Hyper-V CPUID ISOLATION_CONFIG. The access physical address will be original physical address + shared_gpa_boundary. The shared_gpa_boundary in the AMD SEV SNP spec is called virtual top of memory(vTOM). Memory addresses below vTOM are automatically treated as private while memory above vTOM is treated as shared. This seems to be independently reintroducing some of the SEV infrastructure. Is it really OK that this doesn't interact at all with any existing SEV code? For instance, do we need a new 'swiotlb_unencrypted_base', or should this just be using sme_me_mask somehow? Thanks for your review. Hyper-V provides a para-virtualized confidential computing solution based on the AMD SEV function and not expose sev capabilities to guest. So sme_me_mask is unset in the Hyper-V Isolation VM. swiotlb_unencrypted_base is more general solution to handle such case of different address space for encrypted and decrypted memory and other platform also may reuse it. I don't really understand how this can be more general any *not* get utilized by the existing SEV support. The Virtual Top-of-Memory (VTOM) support is an SEV-SNP feature that is meant to be used with a (relatively) un-enlightened guest. The idea is that the C-bit in the guest page tables must be 0 for all accesses. It is only the physical address relative to VTOM that determines if the access is encrypted or not. So setting sme_me_mask will actually cause issues when running with this feature. Since all DMA for an SEV-SNP guest must still be to shared (unencrypted) memory, some enlightenment is needed. In this case, memory mapped above VTOM will provide that via the SWIOTLB update. For SEV-SNP guests running with VTOM, they are likely to also be running with the Reflect #VC feature, allowing a "paravisor" to handle any #VCs generated by the guest. See sections 15.36.8 "Virtual Top-of-Memory" and 15.36.9 "Reflect #VC" in volume 2 of the AMD APM [1]. I'm not sure if that will answer your question or generate more :) Thanks, Tom [1] https://www.amd.com/system/files/TechDocs/24593.pdf ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/13/21 8:36 PM, Tianyu Lan wrote: > On 12/14/2021 12:45 AM, Dave Hansen wrote: >> On 12/12/21 11:14 PM, Tianyu Lan wrote: >>> In Isolation VM with AMD SEV, bounce buffer needs to be accessed via >>> extra address space which is above shared_gpa_boundary (E.G 39 bit >>> address line) reported by Hyper-V CPUID ISOLATION_CONFIG. The access >>> physical address will be original physical address + >>> shared_gpa_boundary. >>> The shared_gpa_boundary in the AMD SEV SNP spec is called virtual top of >>> memory(vTOM). Memory addresses below vTOM are automatically treated as >>> private while memory above vTOM is treated as shared. >> >> This seems to be independently reintroducing some of the SEV >> infrastructure. Is it really OK that this doesn't interact at all with >> any existing SEV code? >> >> For instance, do we need a new 'swiotlb_unencrypted_base', or should >> this just be using sme_me_mask somehow? > > Thanks for your review. Hyper-V provides a para-virtualized > confidential computing solution based on the AMD SEV function and not > expose sev capabilities to guest. So sme_me_mask is unset in the > Hyper-V Isolation VM. swiotlb_unencrypted_base is more general solution > to handle such case of different address space for encrypted and > decrypted memory and other platform also may reuse it. I don't really understand how this can be more general any *not* get utilized by the existing SEV support. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/14/2021 12:45 AM, Dave Hansen wrote: On 12/12/21 11:14 PM, Tianyu Lan wrote: In Isolation VM with AMD SEV, bounce buffer needs to be accessed via extra address space which is above shared_gpa_boundary (E.G 39 bit address line) reported by Hyper-V CPUID ISOLATION_CONFIG. The access physical address will be original physical address + shared_gpa_boundary. The shared_gpa_boundary in the AMD SEV SNP spec is called virtual top of memory(vTOM). Memory addresses below vTOM are automatically treated as private while memory above vTOM is treated as shared. This seems to be independently reintroducing some of the SEV infrastructure. Is it really OK that this doesn't interact at all with any existing SEV code? For instance, do we need a new 'swiotlb_unencrypted_base', or should this just be using sme_me_mask somehow? Hi Dave: Thanks for your review. Hyper-V provides a para-virtualized confidential computing solution based on the AMD SEV function and not expose sev capabilities to guest. So sme_me_mask is unset in the Hyper-V Isolation VM. swiotlb_unencrypted_base is more general solution to handle such case of different address space for encrypted and decrypted memory and other platform also may reuse it. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V7 1/5] swiotlb: Add swiotlb bounce buffer remap function for HV IVM
On 12/12/21 11:14 PM, Tianyu Lan wrote: > In Isolation VM with AMD SEV, bounce buffer needs to be accessed via > extra address space which is above shared_gpa_boundary (E.G 39 bit > address line) reported by Hyper-V CPUID ISOLATION_CONFIG. The access > physical address will be original physical address + shared_gpa_boundary. > The shared_gpa_boundary in the AMD SEV SNP spec is called virtual top of > memory(vTOM). Memory addresses below vTOM are automatically treated as > private while memory above vTOM is treated as shared. This seems to be independently reintroducing some of the SEV infrastructure. Is it really OK that this doesn't interact at all with any existing SEV code? For instance, do we need a new 'swiotlb_unencrypted_base', or should this just be using sme_me_mask somehow? ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
