Re: [Xen-devel] [PATCH 09/15] xen: vmx: handle SGX related MSRs
On 7/21/2017 9:42 PM, Huang, Kai wrote: On 7/20/2017 5:27 AM, Andrew Cooper wrote: On 09/07/17 09:09, Kai Huang wrote: This patch handles IA32_FEATURE_CONTROL and IA32_SGXLEPUBKEYHASHn MSRs. For IA32_FEATURE_CONTROL, if SGX is exposed to domain, then SGX_ENABLE bit is always set. If SGX launch control is also exposed to domain, and physical IA32_SGXLEPUBKEYHASHn are writable, then SGX_LAUNCH_CONTROL_ENABLE bit is also always set. Write to IA32_FEATURE_CONTROL is ignored. For IA32_SGXLEPUBKEYHASHn, a new 'struct sgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtual IA32_SGXLEPUBKEYHASHn are readable and writable. During vcpu is initialized, virtual ia32_sgxlepubkeyhash are also initialized. If physical IA32_SGXLEPUBKEYHASHn are writable, then ia32_sgxlepubkeyhash are set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing is rdmsr_safe is used, as although SDM says if SGX is present, IA32_SGXLEPUBKEYHASHn are available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we use rdmsr_safe and set readable to false if it returns error code. For IA32_SGXLEPUBKEYHASHn MSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned. For IA32_SGXLEPUBKEYHASHn MSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected. To make EINIT run successfully in guest, vcpu's virtual IA32_SGXLEPUBKEYHASHn will be update to physical MSRs when vcpu is scheduled in. Signed-off-by: Kai Huang--- xen/arch/x86/hvm/vmx/sgx.c | 194 + xen/arch/x86/hvm/vmx/vmx.c | 24 + xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 + xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h| 6 ++ 6 files changed, 251 insertions(+) diff --git a/xen/arch/x86/hvm/vmx/sgx.c b/xen/arch/x86/hvm/vmx/sgx.c index 14379151e8..4944e57aef 100644 --- a/xen/arch/x86/hvm/vmx/sgx.c +++ b/xen/arch/x86/hvm/vmx/sgx.c @@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d) hvm_reset_epc(d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable(void) +{ +uint64_t sgx_lc_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | + IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | + IA32_FEATURE_CONTROL_LOCK; +uint64_t val; + +rdmsrl(MSR_IA32_FEATURE_CONTROL, val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx(struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated(d); +} + +bool_t domain_has_sgx_launch_control(struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid; + +if ( !domain_has_sgx(d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +} Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers. The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this. Thanks for comments. Will change to simple check against d->arch.cpuid->feat.{sgx,sgx_lc}. + +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init(struct vcpu *v) +{ +struct sgx_vcpu *sgxv = to_sgx_vcpu(v); + +memset(sgxv, 0, sizeof (*sgxv)); + +if ( sgx_ia32_sgxlepubkeyhash_writable() ) +{ +/* + * If physical MSRs are writable, set vcpu's default value to Intel's + * default value. For real machine, after reset, MSRs contain Intel's + * default value. + */ +sgxv->ia32_sgxlepubkeyhash[0] = SGX_INTEL_DEFAULT_LEPUBKEYHASH0; +sgxv->ia32_sgxlepubkeyhash[1] = SGX_INTEL_DEFAULT_LEPUBKEYHASH1; +sgxv->ia32_sgxlepubkeyhash[2] =
Re: [Xen-devel] [PATCH 09/15] xen: vmx: handle SGX related MSRs
On 7/20/2017 5:27 AM, Andrew Cooper wrote: On 09/07/17 09:09, Kai Huang wrote: This patch handles IA32_FEATURE_CONTROL and IA32_SGXLEPUBKEYHASHn MSRs. For IA32_FEATURE_CONTROL, if SGX is exposed to domain, then SGX_ENABLE bit is always set. If SGX launch control is also exposed to domain, and physical IA32_SGXLEPUBKEYHASHn are writable, then SGX_LAUNCH_CONTROL_ENABLE bit is also always set. Write to IA32_FEATURE_CONTROL is ignored. For IA32_SGXLEPUBKEYHASHn, a new 'struct sgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtual IA32_SGXLEPUBKEYHASHn are readable and writable. During vcpu is initialized, virtual ia32_sgxlepubkeyhash are also initialized. If physical IA32_SGXLEPUBKEYHASHn are writable, then ia32_sgxlepubkeyhash are set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing is rdmsr_safe is used, as although SDM says if SGX is present, IA32_SGXLEPUBKEYHASHn are available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we use rdmsr_safe and set readable to false if it returns error code. For IA32_SGXLEPUBKEYHASHn MSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned. For IA32_SGXLEPUBKEYHASHn MSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected. To make EINIT run successfully in guest, vcpu's virtual IA32_SGXLEPUBKEYHASHn will be update to physical MSRs when vcpu is scheduled in. Signed-off-by: Kai Huang--- xen/arch/x86/hvm/vmx/sgx.c | 194 + xen/arch/x86/hvm/vmx/vmx.c | 24 + xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 + xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h| 6 ++ 6 files changed, 251 insertions(+) diff --git a/xen/arch/x86/hvm/vmx/sgx.c b/xen/arch/x86/hvm/vmx/sgx.c index 14379151e8..4944e57aef 100644 --- a/xen/arch/x86/hvm/vmx/sgx.c +++ b/xen/arch/x86/hvm/vmx/sgx.c @@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d) hvm_reset_epc(d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable(void) +{ +uint64_t sgx_lc_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | + IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | + IA32_FEATURE_CONTROL_LOCK; +uint64_t val; + +rdmsrl(MSR_IA32_FEATURE_CONTROL, val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx(struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated(d); +} + +bool_t domain_has_sgx_launch_control(struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid; + +if ( !domain_has_sgx(d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +} Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers. The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this. Thanks for comments. Will change to simple check against d->arch.cpuid->feat.{sgx,sgx_lc}. + +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init(struct vcpu *v) +{ +struct sgx_vcpu *sgxv = to_sgx_vcpu(v); + +memset(sgxv, 0, sizeof (*sgxv)); + +if ( sgx_ia32_sgxlepubkeyhash_writable() ) +{ +/* + * If physical MSRs are writable, set vcpu's default value to Intel's + * default value. For real machine, after reset, MSRs contain Intel's + * default value. + */ +sgxv->ia32_sgxlepubkeyhash[0] = SGX_INTEL_DEFAULT_LEPUBKEYHASH0; +sgxv->ia32_sgxlepubkeyhash[1] = SGX_INTEL_DEFAULT_LEPUBKEYHASH1; +sgxv->ia32_sgxlepubkeyhash[2] = SGX_INTEL_DEFAULT_LEPUBKEYHASH2; +sgxv->ia32_sgxlepubkeyhash[3] =
Re: [Xen-devel] [PATCH 09/15] xen: vmx: handle SGX related MSRs
On 09/07/17 09:09, Kai Huang wrote: > This patch handles IA32_FEATURE_CONTROL and IA32_SGXLEPUBKEYHASHn MSRs. > > For IA32_FEATURE_CONTROL, if SGX is exposed to domain, then SGX_ENABLE bit > is always set. If SGX launch control is also exposed to domain, and physical > IA32_SGXLEPUBKEYHASHn are writable, then SGX_LAUNCH_CONTROL_ENABLE bit is > also always set. Write to IA32_FEATURE_CONTROL is ignored. > > For IA32_SGXLEPUBKEYHASHn, a new 'struct sgx_vcpu' is added for per-vcpu SGX > staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two > boolean 'readable' and 'writable' are also added to indicate whether virtual > IA32_SGXLEPUBKEYHASHn are readable and writable. > > During vcpu is initialized, virtual ia32_sgxlepubkeyhash are also initialized. > If physical IA32_SGXLEPUBKEYHASHn are writable, then ia32_sgxlepubkeyhash are > set to Intel's default value, as for physical machine, those MSRs will have > Intel's default value. If physical MSRs are not writable (it is *locked* by > BIOS before handling to Xen), then we try to read those MSRs and use physical > values as defult value for virtual MSRs. One thing is rdmsr_safe is used, as > although SDM says if SGX is present, IA32_SGXLEPUBKEYHASHn are available for > read, but in reality, skylake client (at least some, depending on BIOS) > doesn't > have those MSRs available, so we use rdmsr_safe and set readable to false if > it > returns error code. > > For IA32_SGXLEPUBKEYHASHn MSR read from guest, if physical MSRs are not > readable, guest is not allowed to read either, otherwise vcpu's virtual MSR > value is returned. > > For IA32_SGXLEPUBKEYHASHn MSR write from guest, we allow guest to write if > both > physical MSRs are writable and SGX launch control is exposed to domain, > otherwise error is injected. > > To make EINIT run successfully in guest, vcpu's virtual IA32_SGXLEPUBKEYHASHn > will be update to physical MSRs when vcpu is scheduled in. > > Signed-off-by: Kai Huang> --- > xen/arch/x86/hvm/vmx/sgx.c | 194 > + > xen/arch/x86/hvm/vmx/vmx.c | 24 + > xen/include/asm-x86/cpufeature.h | 3 + > xen/include/asm-x86/hvm/vmx/sgx.h | 22 + > xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + > xen/include/asm-x86/msr-index.h| 6 ++ > 6 files changed, 251 insertions(+) > > diff --git a/xen/arch/x86/hvm/vmx/sgx.c b/xen/arch/x86/hvm/vmx/sgx.c > index 14379151e8..4944e57aef 100644 > --- a/xen/arch/x86/hvm/vmx/sgx.c > +++ b/xen/arch/x86/hvm/vmx/sgx.c > @@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d) > hvm_reset_epc(d, true); > } > > +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ > +bool_t sgx_ia32_sgxlepubkeyhash_writable(void) > +{ > +uint64_t sgx_lc_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | > + IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE > | > + IA32_FEATURE_CONTROL_LOCK; > +uint64_t val; > + > +rdmsrl(MSR_IA32_FEATURE_CONTROL, val); > + > +return (val & sgx_lc_enabled) == sgx_lc_enabled; > +} > + > +bool_t domain_has_sgx(struct domain *d) > +{ > +/* hvm_epc_populated(d) implies CPUID has SGX */ > +return hvm_epc_populated(d); > +} > + > +bool_t domain_has_sgx_launch_control(struct domain *d) > +{ > +struct cpuid_policy *p = d->arch.cpuid; > + > +if ( !domain_has_sgx(d) ) > +return false; > + > +/* Unnecessary but check anyway */ > +if ( !cpu_has_sgx_launch_control ) > +return false; > + > +return !!p->feat.sgx_launch_control; > +} Both of these should be d->arch.cpuid->feat.{sgx,sgx_lc} only, and not from having individual helpers. The CPUID setup during host boot and domain construction should take care of setting everything up properly, or hiding the features from the guest. The point of the work I've been doing is to prevent situations where the guest can see SGX but something doesn't work because of Xen using nested checks like this. > + > +/* Digest of Intel signing key. MSR's default value after reset. */ > +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac > +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d > +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d > +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b > + > +void sgx_vcpu_init(struct vcpu *v) > +{ > +struct sgx_vcpu *sgxv = to_sgx_vcpu(v); > + > +memset(sgxv, 0, sizeof (*sgxv)); > + > +if ( sgx_ia32_sgxlepubkeyhash_writable() ) > +{ > +/* > + * If physical MSRs are writable, set vcpu's default value to Intel's > + * default value. For real machine, after reset, MSRs contain Intel's > + * default value. > + */ > +sgxv->ia32_sgxlepubkeyhash[0] = SGX_INTEL_DEFAULT_LEPUBKEYHASH0; > +sgxv->ia32_sgxlepubkeyhash[1] = SGX_INTEL_DEFAULT_LEPUBKEYHASH1; > +sgxv->ia32_sgxlepubkeyhash[2] =
[Xen-devel] [PATCH 09/15] xen: vmx: handle SGX related MSRs
This patch handles IA32_FEATURE_CONTROL and IA32_SGXLEPUBKEYHASHn MSRs. For IA32_FEATURE_CONTROL, if SGX is exposed to domain, then SGX_ENABLE bit is always set. If SGX launch control is also exposed to domain, and physical IA32_SGXLEPUBKEYHASHn are writable, then SGX_LAUNCH_CONTROL_ENABLE bit is also always set. Write to IA32_FEATURE_CONTROL is ignored. For IA32_SGXLEPUBKEYHASHn, a new 'struct sgx_vcpu' is added for per-vcpu SGX staff, and currently it has vcpu's virtual ia32_sgxlepubkeyhash[0-3]. Two boolean 'readable' and 'writable' are also added to indicate whether virtual IA32_SGXLEPUBKEYHASHn are readable and writable. During vcpu is initialized, virtual ia32_sgxlepubkeyhash are also initialized. If physical IA32_SGXLEPUBKEYHASHn are writable, then ia32_sgxlepubkeyhash are set to Intel's default value, as for physical machine, those MSRs will have Intel's default value. If physical MSRs are not writable (it is *locked* by BIOS before handling to Xen), then we try to read those MSRs and use physical values as defult value for virtual MSRs. One thing is rdmsr_safe is used, as although SDM says if SGX is present, IA32_SGXLEPUBKEYHASHn are available for read, but in reality, skylake client (at least some, depending on BIOS) doesn't have those MSRs available, so we use rdmsr_safe and set readable to false if it returns error code. For IA32_SGXLEPUBKEYHASHn MSR read from guest, if physical MSRs are not readable, guest is not allowed to read either, otherwise vcpu's virtual MSR value is returned. For IA32_SGXLEPUBKEYHASHn MSR write from guest, we allow guest to write if both physical MSRs are writable and SGX launch control is exposed to domain, otherwise error is injected. To make EINIT run successfully in guest, vcpu's virtual IA32_SGXLEPUBKEYHASHn will be update to physical MSRs when vcpu is scheduled in. Signed-off-by: Kai Huang--- xen/arch/x86/hvm/vmx/sgx.c | 194 + xen/arch/x86/hvm/vmx/vmx.c | 24 + xen/include/asm-x86/cpufeature.h | 3 + xen/include/asm-x86/hvm/vmx/sgx.h | 22 + xen/include/asm-x86/hvm/vmx/vmcs.h | 2 + xen/include/asm-x86/msr-index.h| 6 ++ 6 files changed, 251 insertions(+) diff --git a/xen/arch/x86/hvm/vmx/sgx.c b/xen/arch/x86/hvm/vmx/sgx.c index 14379151e8..4944e57aef 100644 --- a/xen/arch/x86/hvm/vmx/sgx.c +++ b/xen/arch/x86/hvm/vmx/sgx.c @@ -405,6 +405,200 @@ void hvm_destroy_epc(struct domain *d) hvm_reset_epc(d, true); } +/* Whether IA32_SGXLEPUBKEYHASHn are physically *unlocked* by BIOS */ +bool_t sgx_ia32_sgxlepubkeyhash_writable(void) +{ +uint64_t sgx_lc_enabled = IA32_FEATURE_CONTROL_SGX_ENABLE | + IA32_FEATURE_CONTROL_SGX_LAUNCH_CONTROL_ENABLE | + IA32_FEATURE_CONTROL_LOCK; +uint64_t val; + +rdmsrl(MSR_IA32_FEATURE_CONTROL, val); + +return (val & sgx_lc_enabled) == sgx_lc_enabled; +} + +bool_t domain_has_sgx(struct domain *d) +{ +/* hvm_epc_populated(d) implies CPUID has SGX */ +return hvm_epc_populated(d); +} + +bool_t domain_has_sgx_launch_control(struct domain *d) +{ +struct cpuid_policy *p = d->arch.cpuid; + +if ( !domain_has_sgx(d) ) +return false; + +/* Unnecessary but check anyway */ +if ( !cpu_has_sgx_launch_control ) +return false; + +return !!p->feat.sgx_launch_control; +} + +/* Digest of Intel signing key. MSR's default value after reset. */ +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH0 0xa6053e051270b7ac +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH1 0x6cfbe8ba8b3b413d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH2 0xc4916d99f2b3735d +#define SGX_INTEL_DEFAULT_LEPUBKEYHASH3 0xd4f8c05909f9bb3b + +void sgx_vcpu_init(struct vcpu *v) +{ +struct sgx_vcpu *sgxv = to_sgx_vcpu(v); + +memset(sgxv, 0, sizeof (*sgxv)); + +if ( sgx_ia32_sgxlepubkeyhash_writable() ) +{ +/* + * If physical MSRs are writable, set vcpu's default value to Intel's + * default value. For real machine, after reset, MSRs contain Intel's + * default value. + */ +sgxv->ia32_sgxlepubkeyhash[0] = SGX_INTEL_DEFAULT_LEPUBKEYHASH0; +sgxv->ia32_sgxlepubkeyhash[1] = SGX_INTEL_DEFAULT_LEPUBKEYHASH1; +sgxv->ia32_sgxlepubkeyhash[2] = SGX_INTEL_DEFAULT_LEPUBKEYHASH2; +sgxv->ia32_sgxlepubkeyhash[3] = SGX_INTEL_DEFAULT_LEPUBKEYHASH3; + +sgxv->readable = 1; +sgxv->writable = domain_has_sgx_launch_control(v->domain); +} +else +{ +uint64_t v; +/* + * Although SDM says if SGX is present, then IA32_SGXLEPUBKEYHASHn are + * available for read, but in reality for SKYLAKE client machines, + * those MSRs are not available if SGX is present, so we cannot rely on + * cpu_has_sgx to determine whether to we are able to read MSRs, + * instead, we always use rdmsr_safe. + */ +sgxv->readable =