Re: [PATCH v5 1/7] Documentation/powerpc: Ultravisor API
On 8/9/19 9:45 AM, Michael Ellerman wrote: > Claudio Carvalho writes: >> From: Sukadev Bhattiprolu >> >> POWER9 processor includes support for Protected Execution Facility (PEF). >> Which POWER9? Please be more precise. >> >> It's public knowledge that some versions of Power9 don't have PEF (or >> have it broken / fused off). >> >> People are going to try and test this on various chip revisions that are >> out in the wild, we need to make it clear where it's expected to work >> and where it's not. Updated this part of the commit message to: Protected Execution Facility (PEF) is an architectural change for POWER 9 that enables Secure Virtual Machines (SVMs). When enabled, PEF adds a new higher privileged mode, called Ultravisor mode, to POWER architecture. Along with the new mode there is new firmware called the Protected Execution Ultravisor (or Ultravisor for short). POWER 9 DD2.3 chips (PVR=0x004e1203) or greater will be PEF-capable. Attached documentation ... > >> Attached documentation provides an overview of PEF and defines the API >> for various interfaces that must be implemented in the Ultravisor >> firmware as well as in the KVM Hypervisor. >> >> Based on input from Mike Anderson, Thiago Bauermann, Claudio Carvalho, >> Ben Herrenschmidt, Guerney Hunt, Paul Mackerras. >> >> Signed-off-by: Sukadev Bhattiprolu >> Signed-off-by: Ram Pai >> Signed-off-by: Guerney Hunt >> Reviewed-by: Claudio Carvalho >> Reviewed-by: Michael Anderson >> Reviewed-by: Thiago Bauermann >> Signed-off-by: Claudio Carvalho >> --- >> Documentation/powerpc/ultravisor.rst | 1055 ++ >> 1 file changed, 1055 insertions(+) >> create mode 100644 Documentation/powerpc/ultravisor.rst >> >> diff --git a/Documentation/powerpc/ultravisor.rst >> b/Documentation/powerpc/ultravisor.rst >> new file mode 100644 >> index ..8d5246585b66 >> --- /dev/null >> +++ b/Documentation/powerpc/ultravisor.rst >> @@ -0,0 +1,1055 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> +.. _ultravisor: >> + >> + >> +Protected Execution Facility >> + >> + >> +.. contents:: >> +:depth: 3 >> + >> +.. sectnum:: >> +:depth: 3 >> + >> +Protected Execution Facility >> + >> + >> +Protected Execution Facility (PEF) is an architectural change for >> +POWER 9 that enables Secure Virtual Machines (SVMs). When enabled, > Ditto here. > > Also you don't mention which ISA version PEF is (will be) documented in. > Do we know? Or can we at least reference the RFC number so folks can > find it. > > Otherwise this looks really good. I'll try and find time to proof read > it thoroughly. > > cheers > >> +PEF adds a new higher privileged mode, called Ultravisor mode, to >> +POWER architecture. Along with the new mode there is new firmware >> +called the Protected Execution Ultravisor (or Ultravisor for short). >> +Ultravisor mode is the highest privileged mode in POWER architecture. Updated this part to: Protected Execution Facility (PEF) is an architectural change for POWER 9 that enables Secure Virtual Machines (SVMs). DD2.3 chips (PVR=0x004e1203) or greater will be PEF-capable. A new ISA release will include the PEF RFC02487 changes. When enabled, PEF adds a new higher privileged mode, called Ultravisor mode, to POWER architecture. Along with the new mode there is new firmware called the Protected Execution Ultravisor (or Ultravisor for short). Ultravisor mode is the highest privileged mode in POWER architecture. Followed by the original table below. Thanks Michael Claudio. >> + >> ++--+ >> +| Privilege States | >> ++==+ >> +| Problem | >> ++--+ >> +| Supervisor | >> ++--+ >> +| Hypervisor | >> ++--+ >> +| Ultravisor | >> ++--+ >> + >> +PEF protects SVMs from the hypervisor, privileged users, and other >> +VMs in the system. SVMs are protected while at rest and can only be >> +executed by an authorized machine. All virtual machines utilize >> +hypervisor services. The Ultravisor filters calls between the SVMs >> +and the hypervisor to assure that information does not accidentally >> +leak. All hypercalls except H_RANDOM are reflected to the hypervisor. >> +H_RANDOM is not reflected to prevent the hypervisor from influencing >> +random values in the SVM. >> + >> +To support this there is a refactoring of the ownership of resources >> +in the CPU. Some of the resources which were previously hypervisor >> +privileged are now ultravisor privileged. >> + >> +Hardware >> + >> + >> +The hardware changes include the following: >> + >> +* There is a new bit in the MSR that determines whether the current >> + process
Re: [PATCH v5 1/7] Documentation/powerpc: Ultravisor API
On 8/12/19 12:58 PM, Fabiano Rosas wrote: > Claudio Carvalho writes: > > Some small suggestions below: > >> + >> +* The privilege of a process is now determined by three MSR bits, >> + MSR(S, HV, PR). In each of the tables below the modes are listed >> + from least privilege to highest privilege. The higher privilege >> + modes can access all the resources of the lower privilege modes. >> + >> + **Secure Mode MSR Settings** >> + >> + +---+---+---+---+ >> + | S | HV| PR|Privilege | >> + +===+===+===+===+ >> + | 1 | 0 | 1 | Problem | >> + +---+---+---+---+ >> + | 1 | 0 | 0 | Privileged(OS)| >> + +---+---+---+---+ >> + | 1 | 1 | 0 | Ultravisor| >> + +---+---+---+---+ >> + | 1 | 1 | 1 | Reserved | >> + +---+---+---+---+ >> + >> + **Normal Mode MSR Settings** >> + >> + +---+---+---+---+ >> + | S | HV| PR|Privilege | >> + +===+===+===+===+ >> + | 0 | 0 | 1 | Problem | >> + +---+---+---+---+ >> + | 0 | 0 | 0 | Privileged(OS)| >> + +---+---+---+---+ >> + | 0 | 1 | 0 | Hypervisor| >> + +---+---+---+---+ >> + | 0 | 1 | 1 | Problem (HV) | >> + +---+---+---+---+ > I find the use of '(HV)' in this last line a bit ambiguous. Since we are > already using 'HV' to refer to MSR(HV). I'd suggest using '(Host)' or > simply leaving it out. > >> + >> +* Memory is partitioned into secure and normal memory. Only processes >> + that are running in secure mode can access secure memory. >> + >> +* The hardware does not allow anything that is not running secure to >> + access secure memory. This means that the Hypervisor cannot access >> + the memory of the SVM without using an ultracall (asking the >> + Ultravisor). The Ultravisor will only allow the hypervisor to see >> + the SVM memory encrypted. >> + >> +* I/O systems are not allowed to directly address secure memory. This >> + limits the SVMs to virtual I/O only. >> + >> +* The architecture allows the SVM to share pages of memory with the >> + hypervisor that are not protected with encryption. However, this >> + sharing must be initiated by the SVM. >> + >> +* When a process is running in secure mode all hypercalls >> + (syscall lev=1) are reflected to the Ultravisor. > Here 'reflected' refers to the Ultravisor. Later on, it is used as > meaning the Ultravisor reflects hypercalls/interrupts to the > Hypervisor. I suggest we use this term to mean the latter only. > >> + >> +* When a process is in secure mode all interrupts go to the >> + Ultravisor. >> + >> +* The following resources have become Ultravisor privileged and >> + require an Ultravisor interface to manipulate: >> + >> + * Processor configurations registers (SCOMs). >> + >> + * Stop state information. >> + >> + * The debug registers CIABR, DAWR, and DAWRX become Ultravisor >> +resources when SMFCTRL(D) is set. If SMFCTRL(D) is not set they do > It looks like you could go without "become Ultravisor resources" since > it is already mentioned in the parent item above (The following...). > >> +not work in secure mode. When set, reading and writing requires >> +an Ultravisor call, otherwise that will cause a Hypervisor Emulation >> +Assistance interrupt. >> + >> + * PTCR and partition table entries (partition table is in secure >> +memory). An attempt to write to PTCR will cause a Hypervisor >> +Emulation Assitance interrupt. >> + >> + * LDBAR (LD Base Address Register) and IMC (In-Memory Collection) >> +non-architected registers. An attempt to write to them will cause a >> +Hypervisor Emulation Assistance interrupt. >> + >> + * Paging for an SVM, sharing of memory with Hypervisor for an SVM. >> +(Including Virtual Processor Area (VPA) and virtual I/O). >> + >> + >> +Software/Microcode >> +== >> + >> +The software changes include: >> + >> +* SVMs are created from normal VM using (open source) tooling supplied >> + by IBM. >> + >> +* All SVMs start as normal VMs and utilize an ultracall, UV_ESM >> + (Enter Secure Mode), to make the transition. >> + >> +* When the UV_ESM ultracall is made the Ultravisor copies the VM into >> + secure memory, decrypts the verification information, and checks the >> + integrity of the SVM. If the integrity check passes the Ultravisor >> + passes control in secure mode. >> + >> +* For external interrupts the Ultravisor saves the state of the SVM, >> + and reflects the interrupt to the hypervisor for processing. >> + For hypercalls, the Ultravisor inserts neutral state into all >> + registers not needed for the hypercall
Re: [PATCH v5 1/7] Documentation/powerpc: Ultravisor API
Claudio Carvalho writes: Some small suggestions below: > + > +* The privilege of a process is now determined by three MSR bits, > + MSR(S, HV, PR). In each of the tables below the modes are listed > + from least privilege to highest privilege. The higher privilege > + modes can access all the resources of the lower privilege modes. > + > + **Secure Mode MSR Settings** > + > + +---+---+---+---+ > + | S | HV| PR|Privilege | > + +===+===+===+===+ > + | 1 | 0 | 1 | Problem | > + +---+---+---+---+ > + | 1 | 0 | 0 | Privileged(OS)| > + +---+---+---+---+ > + | 1 | 1 | 0 | Ultravisor| > + +---+---+---+---+ > + | 1 | 1 | 1 | Reserved | > + +---+---+---+---+ > + > + **Normal Mode MSR Settings** > + > + +---+---+---+---+ > + | S | HV| PR|Privilege | > + +===+===+===+===+ > + | 0 | 0 | 1 | Problem | > + +---+---+---+---+ > + | 0 | 0 | 0 | Privileged(OS)| > + +---+---+---+---+ > + | 0 | 1 | 0 | Hypervisor| > + +---+---+---+---+ > + | 0 | 1 | 1 | Problem (HV) | > + +---+---+---+---+ I find the use of '(HV)' in this last line a bit ambiguous. Since we are already using 'HV' to refer to MSR(HV). I'd suggest using '(Host)' or simply leaving it out. > + > +* Memory is partitioned into secure and normal memory. Only processes > + that are running in secure mode can access secure memory. > + > +* The hardware does not allow anything that is not running secure to > + access secure memory. This means that the Hypervisor cannot access > + the memory of the SVM without using an ultracall (asking the > + Ultravisor). The Ultravisor will only allow the hypervisor to see > + the SVM memory encrypted. > + > +* I/O systems are not allowed to directly address secure memory. This > + limits the SVMs to virtual I/O only. > + > +* The architecture allows the SVM to share pages of memory with the > + hypervisor that are not protected with encryption. However, this > + sharing must be initiated by the SVM. > + > +* When a process is running in secure mode all hypercalls > + (syscall lev=1) are reflected to the Ultravisor. Here 'reflected' refers to the Ultravisor. Later on, it is used as meaning the Ultravisor reflects hypercalls/interrupts to the Hypervisor. I suggest we use this term to mean the latter only. > + > +* When a process is in secure mode all interrupts go to the > + Ultravisor. > + > +* The following resources have become Ultravisor privileged and > + require an Ultravisor interface to manipulate: > + > + * Processor configurations registers (SCOMs). > + > + * Stop state information. > + > + * The debug registers CIABR, DAWR, and DAWRX become Ultravisor > +resources when SMFCTRL(D) is set. If SMFCTRL(D) is not set they do It looks like you could go without "become Ultravisor resources" since it is already mentioned in the parent item above (The following...). > +not work in secure mode. When set, reading and writing requires > +an Ultravisor call, otherwise that will cause a Hypervisor Emulation > +Assistance interrupt. > + > + * PTCR and partition table entries (partition table is in secure > +memory). An attempt to write to PTCR will cause a Hypervisor > +Emulation Assitance interrupt. > + > + * LDBAR (LD Base Address Register) and IMC (In-Memory Collection) > +non-architected registers. An attempt to write to them will cause a > +Hypervisor Emulation Assistance interrupt. > + > + * Paging for an SVM, sharing of memory with Hypervisor for an SVM. > +(Including Virtual Processor Area (VPA) and virtual I/O). > + > + > +Software/Microcode > +== > + > +The software changes include: > + > +* SVMs are created from normal VM using (open source) tooling supplied > + by IBM. > + > +* All SVMs start as normal VMs and utilize an ultracall, UV_ESM > + (Enter Secure Mode), to make the transition. > + > +* When the UV_ESM ultracall is made the Ultravisor copies the VM into > + secure memory, decrypts the verification information, and checks the > + integrity of the SVM. If the integrity check passes the Ultravisor > + passes control in secure mode. > + > +* For external interrupts the Ultravisor saves the state of the SVM, > + and reflects the interrupt to the hypervisor for processing. > + For hypercalls, the Ultravisor inserts neutral state into all > + registers not needed for the hypercall then reflects the call to > + the hypervisor for processing. The H_RANDOM hypercall is performed > + by the Ultravisor and not reflected. > + > +* The
Re: [PATCH v5 1/7] Documentation/powerpc: Ultravisor API
Claudio Carvalho writes: > From: Sukadev Bhattiprolu > > POWER9 processor includes support for Protected Execution Facility (PEF). Which POWER9? Please be more precise. It's public knowledge that some versions of Power9 don't have PEF (or have it broken / fused off). People are going to try and test this on various chip revisions that are out in the wild, we need to make it clear where it's expected to work and where it's not. > Attached documentation provides an overview of PEF and defines the API > for various interfaces that must be implemented in the Ultravisor > firmware as well as in the KVM Hypervisor. > > Based on input from Mike Anderson, Thiago Bauermann, Claudio Carvalho, > Ben Herrenschmidt, Guerney Hunt, Paul Mackerras. > > Signed-off-by: Sukadev Bhattiprolu > Signed-off-by: Ram Pai > Signed-off-by: Guerney Hunt > Reviewed-by: Claudio Carvalho > Reviewed-by: Michael Anderson > Reviewed-by: Thiago Bauermann > Signed-off-by: Claudio Carvalho > --- > Documentation/powerpc/ultravisor.rst | 1055 ++ > 1 file changed, 1055 insertions(+) > create mode 100644 Documentation/powerpc/ultravisor.rst > > diff --git a/Documentation/powerpc/ultravisor.rst > b/Documentation/powerpc/ultravisor.rst > new file mode 100644 > index ..8d5246585b66 > --- /dev/null > +++ b/Documentation/powerpc/ultravisor.rst > @@ -0,0 +1,1055 @@ > +.. SPDX-License-Identifier: GPL-2.0 > +.. _ultravisor: > + > + > +Protected Execution Facility > + > + > +.. contents:: > +:depth: 3 > + > +.. sectnum:: > +:depth: 3 > + > +Protected Execution Facility > + > + > +Protected Execution Facility (PEF) is an architectural change for > +POWER 9 that enables Secure Virtual Machines (SVMs). When enabled, Ditto here. Also you don't mention which ISA version PEF is (will be) documented in. Do we know? Or can we at least reference the RFC number so folks can find it. Otherwise this looks really good. I'll try and find time to proof read it thoroughly. cheers > +PEF adds a new higher privileged mode, called Ultravisor mode, to > +POWER architecture. Along with the new mode there is new firmware > +called the Protected Execution Ultravisor (or Ultravisor for short). > +Ultravisor mode is the highest privileged mode in POWER architecture. > + > + +--+ > + | Privilege States | > + +==+ > + | Problem | > + +--+ > + | Supervisor | > + +--+ > + | Hypervisor | > + +--+ > + | Ultravisor | > + +--+ > + > +PEF protects SVMs from the hypervisor, privileged users, and other > +VMs in the system. SVMs are protected while at rest and can only be > +executed by an authorized machine. All virtual machines utilize > +hypervisor services. The Ultravisor filters calls between the SVMs > +and the hypervisor to assure that information does not accidentally > +leak. All hypercalls except H_RANDOM are reflected to the hypervisor. > +H_RANDOM is not reflected to prevent the hypervisor from influencing > +random values in the SVM. > + > +To support this there is a refactoring of the ownership of resources > +in the CPU. Some of the resources which were previously hypervisor > +privileged are now ultravisor privileged. > + > +Hardware > + > + > +The hardware changes include the following: > + > +* There is a new bit in the MSR that determines whether the current > + process is running in secure mode, MSR(S) bit 41. MSR(S)=1, process > + is in secure mode, MSR(s)=0 process is in normal mode. > + > +* The MSR(S) bit can only be set by the Ultravisor. > + > +* HRFID cannot be used to set the MSR(S) bit. If the hypervisor needs > + to return to a SVM it must use an ultracall. It can determine if > + the VM it is returning to is secure. > + > +* There is a new Ultravisor privileged register, SMFCTRL, which has an > + enable/disable bit SMFCTRL(E). > + > +* The privilege of a process is now determined by three MSR bits, > + MSR(S, HV, PR). In each of the tables below the modes are listed > + from least privilege to highest privilege. The higher privilege > + modes can access all the resources of the lower privilege modes. > + > + **Secure Mode MSR Settings** > + > + +---+---+---+---+ > + | S | HV| PR|Privilege | > + +===+===+===+===+ > + | 1 | 0 | 1 | Problem | > + +---+---+---+---+ > + | 1 | 0 | 0 | Privileged(OS)| > + +---+---+---+---+ > + | 1 | 1 | 0 | Ultravisor| > + +---+---+---+---+ > + | 1 | 1 | 1 | Reserved | > + +---+---+---+---+ > + > + **Normal Mode MSR Settings** > +
[PATCH v5 1/7] Documentation/powerpc: Ultravisor API
From: Sukadev Bhattiprolu POWER9 processor includes support for Protected Execution Facility (PEF). Attached documentation provides an overview of PEF and defines the API for various interfaces that must be implemented in the Ultravisor firmware as well as in the KVM Hypervisor. Based on input from Mike Anderson, Thiago Bauermann, Claudio Carvalho, Ben Herrenschmidt, Guerney Hunt, Paul Mackerras. Signed-off-by: Sukadev Bhattiprolu Signed-off-by: Ram Pai Signed-off-by: Guerney Hunt Reviewed-by: Claudio Carvalho Reviewed-by: Michael Anderson Reviewed-by: Thiago Bauermann Signed-off-by: Claudio Carvalho --- Documentation/powerpc/ultravisor.rst | 1055 ++ 1 file changed, 1055 insertions(+) create mode 100644 Documentation/powerpc/ultravisor.rst diff --git a/Documentation/powerpc/ultravisor.rst b/Documentation/powerpc/ultravisor.rst new file mode 100644 index ..8d5246585b66 --- /dev/null +++ b/Documentation/powerpc/ultravisor.rst @@ -0,0 +1,1055 @@ +.. SPDX-License-Identifier: GPL-2.0 +.. _ultravisor: + + +Protected Execution Facility + + +.. contents:: +:depth: 3 + +.. sectnum:: +:depth: 3 + +Protected Execution Facility + + +Protected Execution Facility (PEF) is an architectural change for +POWER 9 that enables Secure Virtual Machines (SVMs). When enabled, +PEF adds a new higher privileged mode, called Ultravisor mode, to +POWER architecture. Along with the new mode there is new firmware +called the Protected Execution Ultravisor (or Ultravisor for short). +Ultravisor mode is the highest privileged mode in POWER architecture. + + +--+ + | Privilege States | + +==+ + | Problem | + +--+ + | Supervisor | + +--+ + | Hypervisor | + +--+ + | Ultravisor | + +--+ + +PEF protects SVMs from the hypervisor, privileged users, and other +VMs in the system. SVMs are protected while at rest and can only be +executed by an authorized machine. All virtual machines utilize +hypervisor services. The Ultravisor filters calls between the SVMs +and the hypervisor to assure that information does not accidentally +leak. All hypercalls except H_RANDOM are reflected to the hypervisor. +H_RANDOM is not reflected to prevent the hypervisor from influencing +random values in the SVM. + +To support this there is a refactoring of the ownership of resources +in the CPU. Some of the resources which were previously hypervisor +privileged are now ultravisor privileged. + +Hardware + + +The hardware changes include the following: + +* There is a new bit in the MSR that determines whether the current + process is running in secure mode, MSR(S) bit 41. MSR(S)=1, process + is in secure mode, MSR(s)=0 process is in normal mode. + +* The MSR(S) bit can only be set by the Ultravisor. + +* HRFID cannot be used to set the MSR(S) bit. If the hypervisor needs + to return to a SVM it must use an ultracall. It can determine if + the VM it is returning to is secure. + +* There is a new Ultravisor privileged register, SMFCTRL, which has an + enable/disable bit SMFCTRL(E). + +* The privilege of a process is now determined by three MSR bits, + MSR(S, HV, PR). In each of the tables below the modes are listed + from least privilege to highest privilege. The higher privilege + modes can access all the resources of the lower privilege modes. + + **Secure Mode MSR Settings** + + +---+---+---+---+ + | S | HV| PR|Privilege | + +===+===+===+===+ + | 1 | 0 | 1 | Problem | + +---+---+---+---+ + | 1 | 0 | 0 | Privileged(OS)| + +---+---+---+---+ + | 1 | 1 | 0 | Ultravisor| + +---+---+---+---+ + | 1 | 1 | 1 | Reserved | + +---+---+---+---+ + + **Normal Mode MSR Settings** + + +---+---+---+---+ + | S | HV| PR|Privilege | + +===+===+===+===+ + | 0 | 0 | 1 | Problem | + +---+---+---+---+ + | 0 | 0 | 0 | Privileged(OS)| + +---+---+---+---+ + | 0 | 1 | 0 | Hypervisor| + +---+---+---+---+ + | 0 | 1 | 1 | Problem (HV) | + +---+---+---+---+ + +* Memory is partitioned into secure and normal memory. Only processes + that are running in secure mode can access secure memory. + +* The hardware does not allow anything that is not running secure to + access secure memory. This means that the Hypervisor cannot access + the memory of the SVM without using an ultracall (asking the + Ultravisor). The
