Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
Hi Wei, Thanks for your comments. Please see my reply below. On 7/29/2017 1:40 AM, Wei Liu wrote: On Tue, Jul 18, 2017 at 08:22:55PM +1200, Huang, Kai wrote: Hi Wei, Thank you very much for comments. Please see my reply below. On 7/17/2017 9:16 PM, Wei Liu wrote: Hi Kai Thanks for this nice write-up. Some comments and questions below. On Sun, Jul 09, 2017 at 08:03:10PM +1200, Kai Huang wrote: Hi all, [...] 2. SGX Virtualization Design 2.1 High Level Toolstack Changes: 2.1.1 New 'epc' parameter EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin alao should be able to get all domain's virtual EPC size. For this purpose, a new 'epc = ' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted. 2.1.2 New XL commands (?) Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands: # xl sgxinfo Which will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary. # xl sgxlist Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains. Alternatively, we can also extend existing XL commands by adding new option # xl info -sgx Which will print out physical EPC size along with other physinfo. And # xl list -sgx Which will print out domain's virtual EPC size. Comments? Can a guest have multiple EPC? If so, the proposed parameter is not good enough. According to SDM a machine may have multiple EPC, but it may have doesn't mean it must have. EPC is typically reserved by BIOS as Processor Reserved Memory (PRM), and in my understanding, client machine doesn't need to have multiple EPC. Currently, I don't see why we need to expose multiple EPC to guest. Even physical machine reports multiple EPC, exposing one EPC to guest is enough. Currently SGX should not be supported with virtual NUMA simultaneously for a single domain. When you say "is enough", do you mean Intel doesn't recommend users to use more than one? I don't think from reading this doc precludes using more then one technically. No I don't think Intel would make such recommendation. For real hardware yes it's possible there are multiple EPC sections, but for client or single socket server machine, typically there will be only one EPC. In case of VM, I don't see there's any benefit of exposing multiple EPCs to guest, except the vNUMA case. My thinking is although SDM doesn't preclude using more than one EPC but for VM there's no need to use more than one. Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions. See the last section of my design. I saw you've already seen it. :) Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest. Although SDM says typically EPC is allocated by BIOS as PRM, but I think we can just treat EPC as PRM, so I believe yes, physically EPC is backed by normal memory. But EPC is reported as reserved memory in e820 table. Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack. I think we'd better to treat EPC as another type of memory. I am not sure whether it should be modeled as device, as on real machine, EPC is also exposed in ACPI table via "INT0E0C" device under \_SB (however it is not modeled as PCIE device for sure). Finally why do you not allow the users to specify the base address? I don't see any reason why user needs to specify base address. If we do, then specify what address? On real machine, BIOS set the base address, and for VM, I think toolstack/Xen should do this. We can expose an option for user to control that if they want to and at the same time provide the logic to calculate the base address internally. I'm not sure if that's going to be very useful, but I'm not convinced it is entirely useless either. Thinking a bit more we can always extend the syntax and API to support that if need be, so I'm fine with not providing such mechanism at early stage. Yeah I think we can extend if needed in the future. Thanks Wei. Thanks, -Kai ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
On Tue, Jul 18, 2017 at 08:22:55PM +1200, Huang, Kai wrote: > Hi Wei, > > Thank you very much for comments. Please see my reply below. > > On 7/17/2017 9:16 PM, Wei Liu wrote: > > Hi Kai > > > > Thanks for this nice write-up. > > > > Some comments and questions below. > > > > On Sun, Jul 09, 2017 at 08:03:10PM +1200, Kai Huang wrote: > > > Hi all, > > > > > [...] > > > 2. SGX Virtualization Design > > > > > > 2.1 High Level Toolstack Changes: > > > > > > 2.1.1 New 'epc' parameter > > > > > > EPC is limited resource. In order to use EPC efficiently among all > > > domains, > > > when creating guest, administrator should be able to specify domain's > > > virtual > > > EPC size. And admin > > > alao should be able to get all domain's virtual EPC size. > > > > > > For this purpose, a new 'epc = ' parameter is added to XL > > > configuration > > > file. This parameter specifies guest's virtual EPC size. The EPC base > > > address > > > will be calculated by toolstack internally, according to guest's memory > > > size, > > > MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be > > > accepted. > > > > > > 2.1.2 New XL commands (?) > > > > > > Administrator should be able to get physical EPC size, and all domain's > > > virtual > > > EPC size. For this purpose, we can introduce 2 additional commands: > > > > > > # xl sgxinfo > > > > > > Which will print out physical EPC size, and other SGX info (such as SGX1, > > > SGX2, > > > etc) if necessary. > > > > > > # xl sgxlist > > > > > > Which will print out particular domain's virtual EPC size, or list all > > > virtual > > > EPC sizes for all supported domains. > > > > > > Alternatively, we can also extend existing XL commands by adding new > > > option > > > > > > # xl info -sgx > > > > > > Which will print out physical EPC size along with other physinfo. And > > > > > > # xl list -sgx > > > > > > Which will print out domain's virtual EPC size. > > > > > > Comments? > > > > > > > Can a guest have multiple EPC? If so, the proposed parameter is not good > > enough. > > According to SDM a machine may have multiple EPC, but it may have doesn't > mean it must have. EPC is typically reserved by BIOS as Processor Reserved > Memory (PRM), and in my understanding, client machine doesn't need to have > multiple EPC. Currently, I don't see why we need to expose multiple EPC to > guest. Even physical machine reports multiple EPC, exposing one EPC to guest > is enough. Currently SGX should not be supported with virtual NUMA > simultaneously for a single domain. > When you say "is enough", do you mean Intel doesn't recommend users to use more than one? I don't think from reading this doc precludes using more then one technically. > > > > Can a guest with EPC enabled be migrated? The answer to this question > > can lead to multiple other questions. > > See the last section of my design. I saw you've already seen it. :) > > > > > Another question, is EPC going to be backed by normal memory? This is > > related to memory accounting of the guest. > > Although SDM says typically EPC is allocated by BIOS as PRM, but I think we > can just treat EPC as PRM, so I believe yes, physically EPC is backed by > normal memory. But EPC is reported as reserved memory in e820 table. > > > > > Is EPC going to be modeled as a device or another type of memory? This > > is related to how we manage it in the toolstack. > > I think we'd better to treat EPC as another type of memory. I am not sure > whether it should be modeled as device, as on real machine, EPC is also > exposed in ACPI table via "INT0E0C" device under \_SB (however it is not > modeled as PCIE device for sure). > > > > > Finally why do you not allow the users to specify the base address? > > I don't see any reason why user needs to specify base address. If we do, > then specify what address? On real machine, BIOS set the base address, and > for VM, I think toolstack/Xen should do this. We can expose an option for user to control that if they want to and at the same time provide the logic to calculate the base address internally. I'm not sure if that's going to be very useful, but I'm not convinced it is entirely useless either. Thinking a bit more we can always extend the syntax and API to support that if need be, so I'm fine with not providing such mechanism at early stage. ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
On 7/17/2017 6:08 PM, Huang, Kai wrote: Hi Andrew, Thank you very much for comments. Sorry for late reply, and please see my reply below. On 7/12/2017 2:13 AM, Andrew Cooper wrote: On 09/07/17 09:03, Kai Huang wrote: Hi all, This series is RFC Xen SGX virtualization support design and RFC draft patches. Thankyou very much for this design doc. 2. SGX Virtualization Design 2.1 High Level Toolstack Changes: 2.1.1 New 'epc' parameter EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin alao should be able to get all domain's virtual EPC size. For this purpose, a new 'epc = ' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted. How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case. First of all, what are the implications of multi-package SGX? (Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys? The ENCLU[EENTE] will continue to work on multi-package server. Actually I was told all ISA existing behavior documented in SDM won't change for server, as otherwise this would be a bad design :) Unfortunately I was told I cannot talk about MP server SGX a lot now. Basically I can only talk about staff already documented in SDM (sorry :( ). But I guess multiple EPC in CPUID is designed to cover MP server, at lease mainly (we can do reasonable guess). In terms of the design, I think we can follow XL config file parameters for memory. 'epc' parameter will always specify totol EPC size that the domain has. And we can use existing NUMA related parameters, such as setting cpus='...' to physically pin vcpu to specific pCPUs, so that EPC will be mostly allocated from related node. If that node runs out of EPC, we can decide whether to allocate EPC from other node, or fail to create domain. I know Linux supports NUMA policy which can specify whether to allow allocating memory from other nodes, does Xen has such policy? Sorry I haven't checked this. If Xen has such policy, we need to choose whether to use memory policy, or introduce new policy for EPC. If we are going to support vNUAM EPC in the future. We can also use similar way to config vNUMA EPC in XL config. Sorry I mentioned scheduling. I should say *potentially* :). My thinking was as SGX is per-thread, then SGX info reported by different CPU package may be different (ex, whether SGX2 is supported), then we may need scheduler to be aware of SGX. But I think we don't have to consider this now. What's your comments? I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM? The native machine allows running multiple enclaves, even signed by multiple authors. SGX only has limit that before launching any other enclave, Launch Enclave (LE) must be launched. LE is the only enclave that doesn't require EINITTOKEN in EINIT. For LE, its signer (SHA256(sigstruct->modulus)) must be equal to the value in IA32_SGXLEPUBKEYHASHn MSRs. LE will generates EINITTOKEN for other enclaves (EINIT for other enclaves requires EINITTOKEN). For other enclaves, there's no such limitation that enclave's signer must match IA32_SGXLEPUBKEYHASHn so the signer can be anybody. But for other enclaves, before running EINIT, the LE's signer (which is equal to IA32_SGXLEPUBKEYHASHn as explained above) needs to be updated to IA32_SGXLEPUBKEYHASHn (MSRs can be changed, for example, when there's multiple LEs running in OS). This is because EINIT needs to perform EINITTOKEN integrity check (EINITTOKEN contains MAC info that calculated by LE, and EINIT needs LE's IA32_SGXLEPUBKEYHASHn to derive the key to verify MAC). SGX in VM doesn't change those behaviors, so in VM, the enclaves can also be signed by anyone, but Xen needs to emulate IA32_SGXLEPUBKEYHASHn so that when one VM is running, the correct IA32_SGXLEPUBKEYHASHn are already in physical MSRs. 2.1.3 Notify domain's virtual EPC base and size to Xen Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid. I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here:
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
Hi Wei, Thank you very much for comments. Please see my reply below. On 7/17/2017 9:16 PM, Wei Liu wrote: Hi Kai Thanks for this nice write-up. Some comments and questions below. On Sun, Jul 09, 2017 at 08:03:10PM +1200, Kai Huang wrote: Hi all, [...] 2. SGX Virtualization Design 2.1 High Level Toolstack Changes: 2.1.1 New 'epc' parameter EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin alao should be able to get all domain's virtual EPC size. For this purpose, a new 'epc = ' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted. 2.1.2 New XL commands (?) Administrator should be able to get physical EPC size, and all domain's virtual EPC size. For this purpose, we can introduce 2 additional commands: # xl sgxinfo Which will print out physical EPC size, and other SGX info (such as SGX1, SGX2, etc) if necessary. # xl sgxlist Which will print out particular domain's virtual EPC size, or list all virtual EPC sizes for all supported domains. Alternatively, we can also extend existing XL commands by adding new option # xl info -sgx Which will print out physical EPC size along with other physinfo. And # xl list -sgx Which will print out domain's virtual EPC size. Comments? Can a guest have multiple EPC? If so, the proposed parameter is not good enough. According to SDM a machine may have multiple EPC, but it may have doesn't mean it must have. EPC is typically reserved by BIOS as Processor Reserved Memory (PRM), and in my understanding, client machine doesn't need to have multiple EPC. Currently, I don't see why we need to expose multiple EPC to guest. Even physical machine reports multiple EPC, exposing one EPC to guest is enough. Currently SGX should not be supported with virtual NUMA simultaneously for a single domain. Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions. See the last section of my design. I saw you've already seen it. :) Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest. Although SDM says typically EPC is allocated by BIOS as PRM, but I think we can just treat EPC as PRM, so I believe yes, physically EPC is backed by normal memory. But EPC is reported as reserved memory in e820 table. Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack. I think we'd better to treat EPC as another type of memory. I am not sure whether it should be modeled as device, as on real machine, EPC is also exposed in ACPI table via "INT0E0C" device under \_SB (however it is not modeled as PCIE device for sure). Finally why do you not allow the users to specify the base address? I don't see any reason why user needs to specify base address. If we do, then specify what address? On real machine, BIOS set the base address, and for VM, I think toolstack/Xen should do this. In my RFC patches I didn't implement the commands as I don't know which is better. In the github repo I mentioned at the beginning, there's an old branch in which I implemented 'xl sgxinfo' and 'xl sgxlist', but they are implemented via dedicated hypercall for SGX, which I am not sure whether is a good option so I didn't include it in my RFC patches. 2.1.3 Notify domain's virtual EPC base and size to Xen Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid. 2.1.4 Launch Control Support (?) [...] But maybe integrating EPC to MM framework is more reasonable. Comments? 2.2.2 EPC Virtualization (?) This part is how to populate EPC for guests. We have 3 choices: - Static Partitioning - Oversubscription - Ballooning IMHO static partitioning is good enough as a starting point. Ballooning is nice to have but please don't make it mandatory. Not all guests have balloon driver -- imagine a unikernel style secure domain running with EPC. That's good point. Thanks. 2.3 Additional Point: Live Migration, Snapshot Support (?) Oh, here it is. Nice. Actually from hardware's point of view, SGX is not migratable. There are two reasons: - SGX key architecture cannot be virtualized. For example, some keys are bound to CPU. For example, Sealing key, EREPORT key, etc. If VM is migrated to another machine, the same enclave will derive the different keys. Taking Sealing key as an example, Sealing key is typically used by enclave (enclave can get sealing key by EGETKEY) to *seal* its secrets to
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
Hi Kai Thanks for this nice write-up. Some comments and questions below. On Sun, Jul 09, 2017 at 08:03:10PM +1200, Kai Huang wrote: > Hi all, > [...] > 2. SGX Virtualization Design > > 2.1 High Level Toolstack Changes: > > 2.1.1 New 'epc' parameter > > EPC is limited resource. In order to use EPC efficiently among all domains, > when creating guest, administrator should be able to specify domain's virtual > EPC size. And admin > alao should be able to get all domain's virtual EPC size. > > For this purpose, a new 'epc = ' parameter is added to XL configuration > file. This parameter specifies guest's virtual EPC size. The EPC base address > will be calculated by toolstack internally, according to guest's memory size, > MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be > accepted. > > 2.1.2 New XL commands (?) > > Administrator should be able to get physical EPC size, and all domain's > virtual > EPC size. For this purpose, we can introduce 2 additional commands: > > # xl sgxinfo > > Which will print out physical EPC size, and other SGX info (such as SGX1, > SGX2, > etc) if necessary. > > # xl sgxlist > > Which will print out particular domain's virtual EPC size, or list all virtual > EPC sizes for all supported domains. > > Alternatively, we can also extend existing XL commands by adding new option > > # xl info -sgx > > Which will print out physical EPC size along with other physinfo. And > > # xl list -sgx > > Which will print out domain's virtual EPC size. > > Comments? > Can a guest have multiple EPC? If so, the proposed parameter is not good enough. Can a guest with EPC enabled be migrated? The answer to this question can lead to multiple other questions. Another question, is EPC going to be backed by normal memory? This is related to memory accounting of the guest. Is EPC going to be modeled as a device or another type of memory? This is related to how we manage it in the toolstack. Finally why do you not allow the users to specify the base address? > In my RFC patches I didn't implement the commands as I don't know which > is better. In the github repo I mentioned at the beginning, there's an old > branch in which I implemented 'xl sgxinfo' and 'xl sgxlist', but they are > implemented via dedicated hypercall for SGX, which I am not sure whether is a > good option so I didn't include it in my RFC patches. > > 2.1.3 Notify domain's virtual EPC base and size to Xen > > Xen needs to know guest's EPC base and size in order to populate EPC pages for > it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid. > > 2.1.4 Launch Control Support (?) [...] > > But maybe integrating EPC to MM framework is more reasonable. Comments? > > 2.2.2 EPC Virtualization (?) > > This part is how to populate EPC for guests. We have 3 choices: > - Static Partitioning > - Oversubscription > - Ballooning > IMHO static partitioning is good enough as a starting point. Ballooning is nice to have but please don't make it mandatory. Not all guests have balloon driver -- imagine a unikernel style secure domain running with EPC. > > 2.3 Additional Point: Live Migration, Snapshot Support (?) > Oh, here it is. Nice. > Actually from hardware's point of view, SGX is not migratable. There are two > reasons: > > - SGX key architecture cannot be virtualized. > > For example, some keys are bound to CPU. For example, Sealing key, EREPORT > key, etc. If VM is migrated to another machine, the same enclave will > derive > the different keys. Taking Sealing key as an example, Sealing key is > typically used by enclave (enclave can get sealing key by EGETKEY) to > *seal* > its secrets to outside (ex, persistent storage) for further use. If > Sealing > key changes after VM migration, then the enclave can never get the sealed > secrets back by using sealing key, as it has changed, and old sealing key > cannot be got back. > > - There's no ENCLS to evict EPC page to normal memory, but at the meaning > time, still keep content in EPC. Currently once EPC page is evicted, the > EPC > page becomes invalid. So technically, we are unable to implement live > migration (or check pointing, or snapshot) for enclave. > > But, with some workaround, and some facts of existing SGX driver, technically > we are able to support Live migration (or even check pointing, snapshot). This > is because: > > - Changing key (which is bound to CPU) is not a problem in reality > > Take Sealing key as an example. Losing sealed data is not a problem, > because > sealing key is only supposed to encrypt secrets that can be provisioned > again. The typical work model is, enclave gets secrets provisioned from > remote (service provider), and use sealing key to store it for further > use. > When enclave tries to *unseal* use sealing key, if the sealing key is > changed, enclave will
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
Hi Andrew, Thank you very much for comments. Sorry for late reply, and please see my reply below. On 7/12/2017 2:13 AM, Andrew Cooper wrote: On 09/07/17 09:03, Kai Huang wrote: Hi all, This series is RFC Xen SGX virtualization support design and RFC draft patches. Thankyou very much for this design doc. 2. SGX Virtualization Design 2.1 High Level Toolstack Changes: 2.1.1 New 'epc' parameter EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin alao should be able to get all domain's virtual EPC size. For this purpose, a new 'epc = ' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted. How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case. First of all, what are the implications of multi-package SGX? (Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys? The ENCLU[EENTE] will continue to work on multi-package server. Actually I was told all ISA existing behavior documented in SDM won't change for server, as otherwise this would be a bad design :) Unfortunately I was told I cannot talk about MP server SGX a lot now. Basically I can only talk about staff already documented in SDM (sorry :( ). But I guess multiple EPC in CPUID is designed to cover MP server, at lease mainly (we can do reasonable guess). In terms of the design, I think we can follow XL config file parameters for memory. 'epc' parameter will always specify totol EPC size that the domain has. And we can use existing NUMA related parameters, such as setting cpus='...' to physically pin vcpu to specific pCPUs, so that EPC will be mostly allocated from related node. If that node runs out of EPC, we can decide whether to allocate EPC from other node, or fail to create domain. I know Linux supports NUMA policy which can specify whether to allow allocating memory from other nodes, does Xen has such policy? Sorry I haven't checked this. If Xen has such policy, we need to choose whether to use memory policy, or introduce new policy for EPC. If we are going to support vNUAM EPC in the future. We can also use similar way to config vNUMA EPC in XL config. Sorry I mentioned scheduling. I should say *potentially* :). My thinking was as SGX is per-thread, then SGX info reported by different CPU package may be different (ex, whether SGX2 is supported), then we may need scheduler to be aware of SGX. But I think we don't have to consider this now. What's your comments? I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM? The native machine allows running multiple enclaves, even signed by multiple authors. SGX only has limit that before launching any other enclave, Launch Enclave (LE) must be launched. LE is the only enclave that doesn't require EINITTOKEN in EINIT. For LE, its signer (SHA256(sigstruct->modulus)) must be equal to the value in IA32_SGXLEPUBKEYHASHn MSRs. LE will generates EINITTOKEN for other enclaves (EINIT for other enclaves requires EINITTOKEN). For other enclaves, there's no such limitation that enclave's signer must match IA32_SGXLEPUBKEYHASHn so the signer can be anybody. But for other enclaves, before running EINIT, the LE's signer (which is equal to IA32_SGXLEPUBKEYHASHn as explained above) needs to be updated to IA32_SGXLEPUBKEYHASHn (MSRs can be changed, for example, when there's multiple LEs running in OS). This is because EINIT needs to perform EINITTOKEN integrity check (EINITTOKEN contains MAC info that calculated by LE, and EINIT needs LE's IA32_SGXLEPUBKEYHASHn to derive the key to verify MAC). SGX in VM doesn't change those behaviors, so in VM, the enclaves can also be signed by anyone, but Xen needs to emulate IA32_SGXLEPUBKEYHASHn so that when one VM is running, the correct IA32_SGXLEPUBKEYHASHn are already in physical MSRs. 2.1.3 Notify domain's virtual EPC base and size to Xen Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid. I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here:
Re: [Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
On 09/07/17 09:03, Kai Huang wrote: Hi all, This series is RFC Xen SGX virtualization support design and RFC draft patches. Thankyou very much for this design doc. 2. SGX Virtualization Design 2.1 High Level Toolstack Changes: 2.1.1 New 'epc' parameter EPC is limited resource. In order to use EPC efficiently among all domains, when creating guest, administrator should be able to specify domain's virtual EPC size. And admin alao should be able to get all domain's virtual EPC size. For this purpose, a new 'epc = ' parameter is added to XL configuration file. This parameter specifies guest's virtual EPC size. The EPC base address will be calculated by toolstack internally, according to guest's memory size, MMIO size, etc. 'epc' is MB in unit and any 1MB aligned value will be accepted. How will this interact with multi-package servers? Even though its fine to implement the single-package support first, the design should be extensible to the multi-package case. First of all, what are the implications of multi-package SGX? (Somewhere) you mention changes to scheduling. I presume this is because a guest with EPC mappings in EPT must be scheduled on the same package, or ENCLU[EENTER] will fail. I presume also that each package will have separate, unrelated private keys? I presume there is no sensible way (even on native) for a single logical process to use multiple different enclaves? By extension, does it make sense to try and offer parts of multiple enclaves to a single VM? 2.1.3 Notify domain's virtual EPC base and size to Xen Xen needs to know guest's EPC base and size in order to populate EPC pages for it. Toolstack notifies EPC base and size to Xen via XEN_DOMCTL_set_cpuid. I am currently in the process of reworking the Xen/Toolstack interface when it comes to CPUID handling. The latest design is available here: https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00378.html but the end result will be the toolstack expressing its CPUID policy in terms of the architectural layout. Therefore, I would expect that, however the setting is represented in the configuration file, xl/libxl would configure it with the hypervisor by setting CPUID.0x12[2] with the appropriate base and size. 2.1.4 Launch Control Support (?) Xen Launch Control Support is about to support running multiple domains with each running its own LE signed by different owners (if HW allows, explained below). As explained in 1.4 SGX Launch Control, EINIT for LE (Launch Enclave) only succeeds when SHA256(SIGSTRUCT.modulus) matches IA32_SGXLEPUBKEYHASHn, and EINIT for other enclaves will derive EINITTOKEN key according to IA32_SGXLEPUBKEYHASHn. Therefore, to support this, guest's virtual IA32_SGXLEPUBKEYHASHn must be updated to phyiscal MSRs before EINIT (which also means the physical IA32_SGXLEPUBKEYHASHn need to be *unlocked* in BIOS before booting to OS). For physical machine, it is BIOS's writer's decision that whether BIOS would provide interface for user to specify customerized IA32_SGXLEPUBKEYHASHn (it is default to digest of Intel's signing key after reset). In reality, OS's SGX driver may require BIOS to make MSRs *unlocked* and actively write the hash value to MSRs in order to run EINIT successfully, as in this case, the driver will not depend on BIOS's capability (whether it allows user to customerize IA32_SGXLEPUBKEYHASHn value). The problem is for Xen, do we need a new parameter, such as 'lehash=' to specify the default value of guset's virtual IA32_SGXLEPUBKEYHASHn? And do we need a new parameter, such as 'lewr' to specify whether guest's virtual MSRs are locked or not before handling to guest's OS? I tends to not introduce 'lehash', as it seems SGX driver would actively update the MSRs. And new parameter would add additional changes for upper layer software (such as openstack). And 'lewr' is not needed either as Xen can always *unlock* the MSRs to guest. Please give comments? Currently in my RFC patches above two parameters are not implemented. Xen hypervisor will always *unlock* the MSRs. Whether there is 'lehash' parameter or not doesn't impact Xen hypervisor's emulation of IA32_SGXLEPUBKEYHASHn. See below Xen hypervisor changes for details. Reading around, am I correct with the following? 1) Some processors have no launch control. There is no restriction on which enclaves can boot. 2) Some Skylake client processors claim to have launch control, but the MSRs are unavailable (is this an erratum?). These are limited to booting enclaves matching the Intel public key. 3) Launch control may be locked by the BIOS. There may be a custom hash, or it might be the Intel default. Xen can't adjust it at all, but can support running any number of VMs with matching enclaves. 4) Launch control may be unlocked by the BIOS. In this case, Xen can context switch a hash per domain, and run all enclaves. The eventual plans for CPUID and MSR levelling should allow all
[Xen-devel] [RFC PATCH 00/15] RFC: SGX virtualization design and draft patches
Hi all, This series is RFC Xen SGX virtualization support design and RFC draft patches. Intel SGX (Software Guard Extensions) is a new set of instructions and memory access mechanisms targetting for application developers seeking to protect select code and data from disclosure or modification. The SGX specification can be found in latest Intel SDM as Volume 3D: https://software.intel.com/sites/default/files/managed/7c/f1/332831-sdm-vol-3d.pdf SGX is relatively more complicated on specification (entire Volume D) so it is unrealistic to list all hardware details here. First part of the design is the brief SGX introduction, which I think is mandatory for introducing SGX virtualization support. Part 2 is design itself. And I put some reference at last. In first part I only introduced the info related virtualization support, although this is definitely not the most important part of SGX. Other parts of SGX (most related to cryptography), ie, enclave measurement, SGX key architecture, Sealing & Attestion (which is critical feature actually) are ommited. Please refer to SGX specification for detailed info. In the design there are some particualr points that I don't know which implementation is better. For those I added a question mark (?) at the right of the menu. Your comments on those parts (and other comments as well, of course) are highly appreciated. Because SGX has lots of details, so I think the design itself can only be high level, and I also included the RFC patches which contains lots of details. Your comments on the patches are also highly appreciated. The code can also be found at below github repo for your access: # git clone https://github.com/01org/xen-sgx -b rfc-v1 And there is another branch named 4.6-sgx is another implementation based on Xen 4.6, it is old but it has some different implementation with this rfc-v1 patches in terms of design choice (ex, it adds a dedicated hypercall). Please help to review and give comments. Thanks in advance. == 1. SGX Introduction 1.1 Overview 1.1.1 Enclave 1.1.2 EPC (Enclave Paage Cache) 1.1.3 ENCLS and ENCLU 1.2 Discovering SGX Capability 1.2.1 Enumerate SGX via CPUID 1.2.2 Intel SGX Opt-in Configuration 1.3 Enclave Life Cycle 1.3.1 Constructing & Destroying Enclave 1.3.2 Enclave Entry and Exit 1.3.2.1 Synchonous Entry and Exit 1.3.2.2 Asynchounous Enclave Exit 1.3.3 EPC Eviction and Reload 1.4 SGX Launch Control 1.5 SGX Interaction with IA32 and IA64 Architecture 2. SGX Virtualization Design 2.1 High Level Toolstack Changes 2.1.1 New 'epc' parameter 2.1.2 New XL commands (?) 2.1.3 Notify domain's virtual EPC base and size to Xen 2.1.4 Launch Control Support (?) 2.2 High Level Hypervisor Changes 2.2.1 EPC Management (?) 2.2.2 EPC Virtualization (?) 2.2.3 Populate EPC for Guest 2.2.4 New Dedicated Hypercall (?) 2.2.5 Launch Control Support 2.2.6 CPUID Emulation 2.2.7 MSR Emulation 2.2.8 EPT Violation & ENCLS Trapping Handling 2.2.9 Guest Suspend & Resume 2.2.10 Destroying Domain 2.3 Additional Point: Live Migration, Snapshot Support (?) 3. Reference 1. SGX Introduction 1.1 Overview 1.1.1 Enclave Intel Software Guard Extensions (SGX) is a set of instructions and mechanisms for memory accesses in order to provide security accesses for sensitive applications and data. SGX allows an application to use it's pariticular address space as an *enclave*, which is a protected area provides confidentiality and integrity even in the presence of privileged malware. Accesses to the enclave memory area from any software not resident in the enclave are prevented, including those from privileged software. Below diagram illustrates the presence of Enclave in application. |---| | | | |---| | | | OS kernel | | |---| | |---| | | | | | | | | |---| | | |---| | | | Entry table | | | | Enclave |---|-> | |---| | | |---| | | | Enclave stack | | | | App code| | | |---| | | |---| | | | Enclave heap | | | | Enclave | | | |---| | | |---| | | | Enclave code | | | | App code| | | |---| | | |---| | | | | | | |---| |---| SGX
