Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/22/2018 12:13 PM, Harald Freudenberger wrote: ... about control domains Talked with the s390 firmware guys. The convention that the control domain mask is a superset of the usage domain mask is only true for 1st level guests. It is absolutely valid to run a kvm guest with restricted control domain mask bitmap in the CRYCB. It is valid to have an empty control domain mask and the guest should be able to run crypto CPRBs on the usage domain(s) without any problems. However, nobody has tried this. I did try this ;). regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/22/2018 12:13 PM, Harald Freudenberger wrote: ... about control domains Talked with the s390 firmware guys. The convention that the control domain mask is a superset of the usage domain mask is only true for 1st level guests. It is absolutely valid to run a kvm guest with restricted control domain mask bitmap in the CRYCB. It is valid to have an empty control domain mask and the guest should be able to run crypto CPRBs on the usage domain(s) without any problems. However, nobody has tried this. I did try this ;). regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
... about control domains Talked with the s390 firmware guys. The convention that the control domain mask is a superset of the usage domain mask is only true for 1st level guests. It is absolutely valid to run a kvm guest with restricted control domain mask bitmap in the CRYCB. It is valid to have an empty control domain mask and the guest should be able to run crypto CPRBs on the usage domain(s) without any problems. However, nobody has tried this. regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
... about control domains Talked with the s390 firmware guys. The convention that the control domain mask is a superset of the usage domain mask is only true for 1st level guests. It is absolutely valid to run a kvm guest with restricted control domain mask bitmap in the CRYCB. It is valid to have an empty control domain mask and the guest should be able to run crypto CPRBs on the usage domain(s) without any problems. However, nobody has tried this. regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Wed, 22 Aug 2018 09:04:13 +0200 Harald Freudenberger wrote: > Well, sooner or later this has to work. Yesterday we tested the control > domain thing with trying to pull some simple data from a 'controlled' domain > to the TKE - doesn't work with a Linux LPAR. I will investigate the details > in the > next weeks. However, long-term it should be possible to run scenarios > like having one KVM guest control all the domains used by other KVM guests. > With respect to the KVM vfio driver, currently there should be just the > rule that for a guest the control domain mask should be equal or a superset > of the usage domain mask. This is by convention as the architecture is > not so clear here, but this is enforced on every place which deals with > usage and control domains (SE, TKE). Thanks for the update; this makes me think we really should fiddle with the masks in the kernel (as opposed to doing it higher up in the stack).
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Wed, 22 Aug 2018 09:04:13 +0200 Harald Freudenberger wrote: > Well, sooner or later this has to work. Yesterday we tested the control > domain thing with trying to pull some simple data from a 'controlled' domain > to the TKE - doesn't work with a Linux LPAR. I will investigate the details > in the > next weeks. However, long-term it should be possible to run scenarios > like having one KVM guest control all the domains used by other KVM guests. > With respect to the KVM vfio driver, currently there should be just the > rule that for a guest the control domain mask should be equal or a superset > of the usage domain mask. This is by convention as the architecture is > not so clear here, but this is enforced on every place which deals with > usage and control domains (SE, TKE). Thanks for the update; this makes me think we really should fiddle with the masks in the kernel (as opposed to doing it higher up in the stack).
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Tue, 21 Aug 2018 20:54:49 +0200 Halil Pasic wrote: > On 08/20/2018 10:16 PM, Tony Krowiak wrote: > >> Does the SIE complain if you specify a control > >> domain that the host does not have access to (I'd guess so)? > > > > The SIE does not complain if you specify a domain to which the host - or a > > lower level guest - does not have access. The firmware performs a logical > > AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and > > ADMs > > Rather a bit-wise AND, I guess (of the same type masks corresponding to Guest > 1 and > Guest 2). The result of a logical AND is a logical value (true or false) as > far as I remember. > > > to create effective masks EAPM, EAQM and EADM. Only devices corresponding to > > the bits set in the EAPM, EAQM and EADM will be accessible by the guest. > > I'm not sure what is the intended meaning of 'the SIE complains'. If it means > getting out of (SIE when interpreting lets say an NQAP under the discussed > circumstances) with some sort of error code, I think Tony's answer, ' SIE > does not complain' > makes a lot of sense. It's the guest that's is trying to stretch further than > the blanket reaches, and it's the guest that needs to be educated on this > fact. Yep, that's what I meant. If the hypervisor can call the SIE with that config, but the guest gets an error if it tries to use something that it cannot use, that's fine.
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Tue, 21 Aug 2018 20:54:49 +0200 Halil Pasic wrote: > On 08/20/2018 10:16 PM, Tony Krowiak wrote: > >> Does the SIE complain if you specify a control > >> domain that the host does not have access to (I'd guess so)? > > > > The SIE does not complain if you specify a domain to which the host - or a > > lower level guest - does not have access. The firmware performs a logical > > AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and > > ADMs > > Rather a bit-wise AND, I guess (of the same type masks corresponding to Guest > 1 and > Guest 2). The result of a logical AND is a logical value (true or false) as > far as I remember. > > > to create effective masks EAPM, EAQM and EADM. Only devices corresponding to > > the bits set in the EAPM, EAQM and EADM will be accessible by the guest. > > I'm not sure what is the intended meaning of 'the SIE complains'. If it means > getting out of (SIE when interpreting lets say an NQAP under the discussed > circumstances) with some sort of error code, I think Tony's answer, ' SIE > does not complain' > makes a lot of sense. It's the guest that's is trying to stretch further than > the blanket reaches, and it's the guest that needs to be educated on this > fact. Yep, that's what I meant. If the hypervisor can call the SIE with that config, but the guest gets an error if it tries to use something that it cannot use, that's fine.
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 21.08.2018 17:53, Cornelia Huck wrote: > On Tue, 21 Aug 2018 11:00:00 +0200 > Harald Freudenberger wrote: > >> On 20.08.2018 18:03, Cornelia Huck wrote: >>> On Mon, 13 Aug 2018 17:48:19 -0400 >>> Tony Krowiak wrote: +* AP Instructions: + + There are three AP instructions: + + * NQAP: to enqueue an AP command-request message to a queue + * DQAP: to dequeue an AP command-reply message from a queue + * PQAP: to administer the queues >>> So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or >>> is it that all of them need usage domains, but PQAP can target a control >>> domain as well? >>> >>> [I don't want to dive deeply into the AP architecture here, just far >>> enough to really understand the design implications.] >> Well, to be honest, nobody ever tried this under Linux. Theoretically >> one should be able to send a CPRB to a usage domain where inside >> the CPRB another domain (the control domain) is addressed. However, >> as of now I am only aware of applications controlling the same usage >> domain. I don't know any application which is able to address another >> control domain and I am not sure if the zcrypt device driver would >> handle such a CPRB correctly. NQAP, DQAP and PQAP always address >> a usage domain. But the CPRB send down the pipe via NQAP may >> address some control thing on another domain. I am not sure which >> code and where do the sorting out here. There are two candidates: >> the firmware layer in the CEC and the crypto card code. > OK, so it's possible as by the architecture, but at least Linux does > not (currently) do it? > > Perhaps we should simply not overthink that whole control domain > thingy :) It's mostly yet another knob, and as long as the design does > not go against the general architecture, it's probably fine, I guess. Well, sooner or later this has to work. Yesterday we tested the control domain thing with trying to pull some simple data from a 'controlled' domain to the TKE - doesn't work with a Linux LPAR. I will investigate the details in the next weeks. However, long-term it should be possible to run scenarios like having one KVM guest control all the domains used by other KVM guests. With respect to the KVM vfio driver, currently there should be just the rule that for a guest the control domain mask should be equal or a superset of the usage domain mask. This is by convention as the architecture is not so clear here, but this is enforced on every place which deals with usage and control domains (SE, TKE). regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 21.08.2018 17:53, Cornelia Huck wrote: > On Tue, 21 Aug 2018 11:00:00 +0200 > Harald Freudenberger wrote: > >> On 20.08.2018 18:03, Cornelia Huck wrote: >>> On Mon, 13 Aug 2018 17:48:19 -0400 >>> Tony Krowiak wrote: +* AP Instructions: + + There are three AP instructions: + + * NQAP: to enqueue an AP command-request message to a queue + * DQAP: to dequeue an AP command-reply message from a queue + * PQAP: to administer the queues >>> So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or >>> is it that all of them need usage domains, but PQAP can target a control >>> domain as well? >>> >>> [I don't want to dive deeply into the AP architecture here, just far >>> enough to really understand the design implications.] >> Well, to be honest, nobody ever tried this under Linux. Theoretically >> one should be able to send a CPRB to a usage domain where inside >> the CPRB another domain (the control domain) is addressed. However, >> as of now I am only aware of applications controlling the same usage >> domain. I don't know any application which is able to address another >> control domain and I am not sure if the zcrypt device driver would >> handle such a CPRB correctly. NQAP, DQAP and PQAP always address >> a usage domain. But the CPRB send down the pipe via NQAP may >> address some control thing on another domain. I am not sure which >> code and where do the sorting out here. There are two candidates: >> the firmware layer in the CEC and the crypto card code. > OK, so it's possible as by the architecture, but at least Linux does > not (currently) do it? > > Perhaps we should simply not overthink that whole control domain > thingy :) It's mostly yet another knob, and as long as the design does > not go against the general architecture, it's probably fine, I guess. Well, sooner or later this has to work. Yesterday we tested the control domain thing with trying to pull some simple data from a 'controlled' domain to the TKE - doesn't work with a Linux LPAR. I will investigate the details in the next weeks. However, long-term it should be possible to run scenarios like having one KVM guest control all the domains used by other KVM guests. With respect to the KVM vfio driver, currently there should be just the rule that for a guest the control domain mask should be equal or a superset of the usage domain mask. This is by convention as the architecture is not so clear here, but this is enforced on every place which deals with usage and control domains (SE, TKE). regards Harald Freudenberger
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/21/2018 12:13 PM, Cornelia Huck wrote: On Mon, 20 Aug 2018 16:16:15 -0400 Tony Krowiak wrote: On 08/20/2018 12:03 PM, Cornelia Huck wrote: On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: +AP Architectural Overview: += +To facilitate the comprehension of the design, let's start with some +definitions: + +* AP adapter + + An AP adapter is an IBM Z adapter card that can perform cryptographic + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters + assigned to the LPAR in which a linux host is running will be available to + the linux host. Each adapter is identified by a number from 0 to 255. When + installed, an AP adapter is accessed by AP instructions executed by any CPU. + + The AP adapter cards are assigned to a given LPAR via the system's Activation + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... I'm not quite sure how one can describe s390-specific devices that can be installed only on an s390 system without using s390 jargon. I would think that one who is administering a linux host or guest running on an s390 system would have some basic knowledge of s390. If you have any suggestions, I'd be happy to entertain them. I fear the jargon is mostly unavoidable :( +* AP Instructions: + + There are three AP instructions: + + * NQAP: to enqueue an AP command-request message to a queue + * DQAP: to dequeue an AP command-reply message from a queue + * PQAP: to administer the queues So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or is it that all of them need usage domains, but PQAP can target a control domain as well? All AP instructions - the lone exception being the PQAP(QCI) subfunction - identify the usage domain that is the target of the instruction. I think using the term 'control domain' is the source of much confusion. It makes it seem as if there are two types of domains that serve different purposes. That is simply not true. A domain is a partition within an AP adapter that can process AP command request messages. All AP commands are sent to a domain. Configuring a domain as a usage domain means it can be used to process AP commands; in other words, it can be the target of an AP instruction. Configuring a domain as a control domain means it can be changed by an AP command. AP commands that change a domain are sent to a usage domain, but the domain to be changed is specified in the payload of the AP command message. The domain thus specified must be identified via the AP configuration as a control domain, or the AP command will be rejected. Yes, the 'control domain' term is a source of much confusion :( [I don't want to dive deeply into the AP architecture here, just far enough to really understand the design implications.] Are you suggesting some of the above should be removed? If so, what? Not removed. What about an explanation like the following somewhere: "AP instructions identify the domain that is targeted to process the command: This must be one of the usage domains. They may modify a domain that is not one of the usage domains, but the modified domain must be one of the control domains." I hope that is both correct and understandable ;) Yes, it is both correct and understandable. Does the SIE complain if you specify a control domain that the host does not have access to (I'd guess so)? The SIE does not complain if you specify a domain to which the host - or a lower level guest - does not have access. The firmware performs a logical AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and ADMs to create effective masks EAPM, EAQM and EADM. Only devices corresponding to the bits set in the EAPM, EAQM and EADM will be accessible by the guest. OK, so the guest effectively won't see the domain. That makes sense. It is one of the positive aspects of the architecture. + +The APQNs can provide secure key functionality - i.e., a private key is stored +on the adapter card for each of its domains - so each APQN must be assigned to +at most one guest or to the linux host. + + Example 1: Valid configuration: + -- + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1,2 domain 7 + + This is valid because both guests have a unique set of APQNs: Guest1 has + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7). + + Example 2: Invalid configuration: + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1domains 6,7 + + This is an invalid configuration because both guests have access to + APQN (1,6). So, the adapters or the domains can overlap , but the cross product mustn't? If I had Guest1: adapters 1,2 domains 5,6 Guest2: adapters 3,4 domains 5,6 would that be fine? Yes, that would be fine because Guest1 would have access to
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/21/2018 12:13 PM, Cornelia Huck wrote: On Mon, 20 Aug 2018 16:16:15 -0400 Tony Krowiak wrote: On 08/20/2018 12:03 PM, Cornelia Huck wrote: On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: +AP Architectural Overview: += +To facilitate the comprehension of the design, let's start with some +definitions: + +* AP adapter + + An AP adapter is an IBM Z adapter card that can perform cryptographic + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters + assigned to the LPAR in which a linux host is running will be available to + the linux host. Each adapter is identified by a number from 0 to 255. When + installed, an AP adapter is accessed by AP instructions executed by any CPU. + + The AP adapter cards are assigned to a given LPAR via the system's Activation + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... I'm not quite sure how one can describe s390-specific devices that can be installed only on an s390 system without using s390 jargon. I would think that one who is administering a linux host or guest running on an s390 system would have some basic knowledge of s390. If you have any suggestions, I'd be happy to entertain them. I fear the jargon is mostly unavoidable :( +* AP Instructions: + + There are three AP instructions: + + * NQAP: to enqueue an AP command-request message to a queue + * DQAP: to dequeue an AP command-reply message from a queue + * PQAP: to administer the queues So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or is it that all of them need usage domains, but PQAP can target a control domain as well? All AP instructions - the lone exception being the PQAP(QCI) subfunction - identify the usage domain that is the target of the instruction. I think using the term 'control domain' is the source of much confusion. It makes it seem as if there are two types of domains that serve different purposes. That is simply not true. A domain is a partition within an AP adapter that can process AP command request messages. All AP commands are sent to a domain. Configuring a domain as a usage domain means it can be used to process AP commands; in other words, it can be the target of an AP instruction. Configuring a domain as a control domain means it can be changed by an AP command. AP commands that change a domain are sent to a usage domain, but the domain to be changed is specified in the payload of the AP command message. The domain thus specified must be identified via the AP configuration as a control domain, or the AP command will be rejected. Yes, the 'control domain' term is a source of much confusion :( [I don't want to dive deeply into the AP architecture here, just far enough to really understand the design implications.] Are you suggesting some of the above should be removed? If so, what? Not removed. What about an explanation like the following somewhere: "AP instructions identify the domain that is targeted to process the command: This must be one of the usage domains. They may modify a domain that is not one of the usage domains, but the modified domain must be one of the control domains." I hope that is both correct and understandable ;) Yes, it is both correct and understandable. Does the SIE complain if you specify a control domain that the host does not have access to (I'd guess so)? The SIE does not complain if you specify a domain to which the host - or a lower level guest - does not have access. The firmware performs a logical AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and ADMs to create effective masks EAPM, EAQM and EADM. Only devices corresponding to the bits set in the EAPM, EAQM and EADM will be accessible by the guest. OK, so the guest effectively won't see the domain. That makes sense. It is one of the positive aspects of the architecture. + +The APQNs can provide secure key functionality - i.e., a private key is stored +on the adapter card for each of its domains - so each APQN must be assigned to +at most one guest or to the linux host. + + Example 1: Valid configuration: + -- + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1,2 domain 7 + + This is valid because both guests have a unique set of APQNs: Guest1 has + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7). + + Example 2: Invalid configuration: + Guest1: adapters 1,2 domains 5,6 + Guest2: adapter 1domains 6,7 + + This is an invalid configuration because both guests have access to + APQN (1,6). So, the adapters or the domains can overlap , but the cross product mustn't? If I had Guest1: adapters 1,2 domains 5,6 Guest2: adapters 3,4 domains 5,6 would that be fine? Yes, that would be fine because Guest1 would have access to
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/20/2018 10:16 PM, Tony Krowiak wrote: Does the SIE complain if you specify a control domain that the host does not have access to (I'd guess so)? The SIE does not complain if you specify a domain to which the host - or a lower level guest - does not have access. The firmware performs a logical AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and ADMs Rather a bit-wise AND, I guess (of the same type masks corresponding to Guest 1 and Guest 2). The result of a logical AND is a logical value (true or false) as far as I remember. to create effective masks EAPM, EAQM and EADM. Only devices corresponding to the bits set in the EAPM, EAQM and EADM will be accessible by the guest. I'm not sure what is the intended meaning of 'the SIE complains'. If it means getting out of (SIE when interpreting lets say an NQAP under the discussed circumstances) with some sort of error code, I think Tony's answer, ' SIE does not complain' makes a lot of sense. It's the guest that's is trying to stretch further than the blanket reaches, and it's the guest that needs to be educated on this fact. AFAIR SIE does the right thing (whatever the right thing is) and we don't have to worry about it. As a matter of fact I can't recall exactly what is supposed to happen when a guest tries to modify a domain such that the guest does not have privileges to modify (in terms of EADM, either because the guest or because the host does not have the corresponding bit set). I'm sure I did not try it out. Tony did you test this scenario? (BTW my best guess at the moment is, that the situation is handled via the command-reply.) Regards, Halil
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/20/2018 10:16 PM, Tony Krowiak wrote: Does the SIE complain if you specify a control domain that the host does not have access to (I'd guess so)? The SIE does not complain if you specify a domain to which the host - or a lower level guest - does not have access. The firmware performs a logical AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and ADMs Rather a bit-wise AND, I guess (of the same type masks corresponding to Guest 1 and Guest 2). The result of a logical AND is a logical value (true or false) as far as I remember. to create effective masks EAPM, EAQM and EADM. Only devices corresponding to the bits set in the EAPM, EAQM and EADM will be accessible by the guest. I'm not sure what is the intended meaning of 'the SIE complains'. If it means getting out of (SIE when interpreting lets say an NQAP under the discussed circumstances) with some sort of error code, I think Tony's answer, ' SIE does not complain' makes a lot of sense. It's the guest that's is trying to stretch further than the blanket reaches, and it's the guest that needs to be educated on this fact. AFAIR SIE does the right thing (whatever the right thing is) and we don't have to worry about it. As a matter of fact I can't recall exactly what is supposed to happen when a guest tries to modify a domain such that the guest does not have privileges to modify (in terms of EADM, either because the guest or because the host does not have the corresponding bit set). I'm sure I did not try it out. Tony did you test this scenario? (BTW my best guess at the moment is, that the situation is handled via the command-reply.) Regards, Halil
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Mon, 20 Aug 2018 16:16:15 -0400 Tony Krowiak wrote: > On 08/20/2018 12:03 PM, Cornelia Huck wrote: > > On Mon, 13 Aug 2018 17:48:19 -0400 > > Tony Krowiak wrote: > >> +AP Architectural Overview: > >> += > >> +To facilitate the comprehension of the design, let's start with some > >> +definitions: > >> + > >> +* AP adapter > >> + > >> + An AP adapter is an IBM Z adapter card that can perform cryptographic > >> + functions. There can be from 0 to 256 adapters assigned to an LPAR. > >> Adapters > >> + assigned to the LPAR in which a linux host is running will be available > >> to > >> + the linux host. Each adapter is identified by a number from 0 to 255. > >> When > >> + installed, an AP adapter is accessed by AP instructions executed by any > >> CPU. > >> + > >> + The AP adapter cards are assigned to a given LPAR via the system's > >> Activation > >> + Profile which can be edited via the HMC. When the system is IPL'd, the > >> AP bus > > There's lots of s390 jargon in here... but one hopes that someone > > trying to understand AP is already familiar with the basics... > > I'm not quite sure how one can describe s390-specific devices that can > be installed > only on an s390 system without using s390 jargon. I would think that one > who is > administering a linux host or guest running on an s390 system would have > some > basic knowledge of s390. If you have any suggestions, I'd be happy to > entertain them. I fear the jargon is mostly unavoidable :( > >> +* AP Instructions: > >> + > >> + There are three AP instructions: > >> + > >> + * NQAP: to enqueue an AP command-request message to a queue > >> + * DQAP: to dequeue an AP command-reply message from a queue > >> + * PQAP: to administer the queues > > So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or > > is it that all of them need usage domains, but PQAP can target a control > > domain as well? > > All AP instructions - the lone exception being the PQAP(QCI) subfunction - > identify the usage domain that is the target of the instruction. I think > using the term 'control domain' is the source of much confusion. It makes > it seem as if there are two types of domains that serve different purposes. > That is simply not true. A domain is a partition within an AP adapter that > can process AP command request messages. All AP commands are sent to a > domain. Configuring a domain as a usage domain means it can be used to > process AP commands; in other words, it can be the target of an AP > instruction. Configuring a domain as a control domain means it can be > changed by an AP command. AP commands that change a domain are sent to > a usage domain, but the domain to be changed is specified in the payload > of the AP command message. The domain thus specified must be > identified via the AP configuration as a control domain, or the AP command > will be rejected. Yes, the 'control domain' term is a source of much confusion :( > > > > > [I don't want to dive deeply into the AP architecture here, just far > > enough to really understand the design implications.] > > Are you suggesting some of the above should be removed? If so, what? Not removed. What about an explanation like the following somewhere: "AP instructions identify the domain that is targeted to process the command: This must be one of the usage domains. They may modify a domain that is not one of the usage domains, but the modified domain must be one of the control domains." I hope that is both correct and understandable ;) > > Does the SIE complain if you specify a control > > domain that the host does not have access to (I'd guess so)? > > The SIE does not complain if you specify a domain to which the host - or a > lower level guest - does not have access. The firmware performs a logical > AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and > ADMs > to create effective masks EAPM, EAQM and EADM. Only devices corresponding to > the bits set in the EAPM, EAQM and EADM will be accessible by the guest. OK, so the guest effectively won't see the domain. That makes sense. > > > > >> + > >> +The APQNs can provide secure key functionality - i.e., a private key is > >> stored > >> +on the adapter card for each of its domains - so each APQN must be > >> assigned to > >> +at most one guest or to the linux host. > >> + > >> + Example 1: Valid configuration: > >> + -- > >> + Guest1: adapters 1,2 domains 5,6 > >> + Guest2: adapter 1,2 domain 7 > >> + > >> + This is valid because both guests have a unique set of APQNs: Guest1 > >> has > >> + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7). > >> + > >> + Example 2: Invalid configuration: > >> + Guest1: adapters 1,2 domains 5,6 > >> + Guest2: adapter 1domains 6,7 > >> + > >> + This is an invalid configuration because both guests have access to > >> + APQN (1,6).
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Mon, 20 Aug 2018 16:16:15 -0400 Tony Krowiak wrote: > On 08/20/2018 12:03 PM, Cornelia Huck wrote: > > On Mon, 13 Aug 2018 17:48:19 -0400 > > Tony Krowiak wrote: > >> +AP Architectural Overview: > >> += > >> +To facilitate the comprehension of the design, let's start with some > >> +definitions: > >> + > >> +* AP adapter > >> + > >> + An AP adapter is an IBM Z adapter card that can perform cryptographic > >> + functions. There can be from 0 to 256 adapters assigned to an LPAR. > >> Adapters > >> + assigned to the LPAR in which a linux host is running will be available > >> to > >> + the linux host. Each adapter is identified by a number from 0 to 255. > >> When > >> + installed, an AP adapter is accessed by AP instructions executed by any > >> CPU. > >> + > >> + The AP adapter cards are assigned to a given LPAR via the system's > >> Activation > >> + Profile which can be edited via the HMC. When the system is IPL'd, the > >> AP bus > > There's lots of s390 jargon in here... but one hopes that someone > > trying to understand AP is already familiar with the basics... > > I'm not quite sure how one can describe s390-specific devices that can > be installed > only on an s390 system without using s390 jargon. I would think that one > who is > administering a linux host or guest running on an s390 system would have > some > basic knowledge of s390. If you have any suggestions, I'd be happy to > entertain them. I fear the jargon is mostly unavoidable :( > >> +* AP Instructions: > >> + > >> + There are three AP instructions: > >> + > >> + * NQAP: to enqueue an AP command-request message to a queue > >> + * DQAP: to dequeue an AP command-reply message from a queue > >> + * PQAP: to administer the queues > > So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or > > is it that all of them need usage domains, but PQAP can target a control > > domain as well? > > All AP instructions - the lone exception being the PQAP(QCI) subfunction - > identify the usage domain that is the target of the instruction. I think > using the term 'control domain' is the source of much confusion. It makes > it seem as if there are two types of domains that serve different purposes. > That is simply not true. A domain is a partition within an AP adapter that > can process AP command request messages. All AP commands are sent to a > domain. Configuring a domain as a usage domain means it can be used to > process AP commands; in other words, it can be the target of an AP > instruction. Configuring a domain as a control domain means it can be > changed by an AP command. AP commands that change a domain are sent to > a usage domain, but the domain to be changed is specified in the payload > of the AP command message. The domain thus specified must be > identified via the AP configuration as a control domain, or the AP command > will be rejected. Yes, the 'control domain' term is a source of much confusion :( > > > > > [I don't want to dive deeply into the AP architecture here, just far > > enough to really understand the design implications.] > > Are you suggesting some of the above should be removed? If so, what? Not removed. What about an explanation like the following somewhere: "AP instructions identify the domain that is targeted to process the command: This must be one of the usage domains. They may modify a domain that is not one of the usage domains, but the modified domain must be one of the control domains." I hope that is both correct and understandable ;) > > Does the SIE complain if you specify a control > > domain that the host does not have access to (I'd guess so)? > > The SIE does not complain if you specify a domain to which the host - or a > lower level guest - does not have access. The firmware performs a logical > AND of the guest's and hosts's - or lower level guest's - APMs, AQMs and > ADMs > to create effective masks EAPM, EAQM and EADM. Only devices corresponding to > the bits set in the EAPM, EAQM and EADM will be accessible by the guest. OK, so the guest effectively won't see the domain. That makes sense. > > > > >> + > >> +The APQNs can provide secure key functionality - i.e., a private key is > >> stored > >> +on the adapter card for each of its domains - so each APQN must be > >> assigned to > >> +at most one guest or to the linux host. > >> + > >> + Example 1: Valid configuration: > >> + -- > >> + Guest1: adapters 1,2 domains 5,6 > >> + Guest2: adapter 1,2 domain 7 > >> + > >> + This is valid because both guests have a unique set of APQNs: Guest1 > >> has > >> + APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQNs (1,7) and (2,7). > >> + > >> + Example 2: Invalid configuration: > >> + Guest1: adapters 1,2 domains 5,6 > >> + Guest2: adapter 1domains 6,7 > >> + > >> + This is an invalid configuration because both guests have access to > >> + APQN (1,6).
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Tue, 21 Aug 2018 11:00:00 +0200 Harald Freudenberger wrote: > On 20.08.2018 18:03, Cornelia Huck wrote: > > On Mon, 13 Aug 2018 17:48:19 -0400 > > Tony Krowiak wrote: > >> +* AP Instructions: > >> + > >> + There are three AP instructions: > >> + > >> + * NQAP: to enqueue an AP command-request message to a queue > >> + * DQAP: to dequeue an AP command-reply message from a queue > >> + * PQAP: to administer the queues > > So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or > > is it that all of them need usage domains, but PQAP can target a control > > domain as well? > > > > [I don't want to dive deeply into the AP architecture here, just far > > enough to really understand the design implications.] > Well, to be honest, nobody ever tried this under Linux. Theoretically > one should be able to send a CPRB to a usage domain where inside > the CPRB another domain (the control domain) is addressed. However, > as of now I am only aware of applications controlling the same usage > domain. I don't know any application which is able to address another > control domain and I am not sure if the zcrypt device driver would > handle such a CPRB correctly. NQAP, DQAP and PQAP always address > a usage domain. But the CPRB send down the pipe via NQAP may > address some control thing on another domain. I am not sure which > code and where do the sorting out here. There are two candidates: > the firmware layer in the CEC and the crypto card code. OK, so it's possible as by the architecture, but at least Linux does not (currently) do it? Perhaps we should simply not overthink that whole control domain thingy :) It's mostly yet another knob, and as long as the design does not go against the general architecture, it's probably fine, I guess.
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Tue, 21 Aug 2018 11:00:00 +0200 Harald Freudenberger wrote: > On 20.08.2018 18:03, Cornelia Huck wrote: > > On Mon, 13 Aug 2018 17:48:19 -0400 > > Tony Krowiak wrote: > >> +* AP Instructions: > >> + > >> + There are three AP instructions: > >> + > >> + * NQAP: to enqueue an AP command-request message to a queue > >> + * DQAP: to dequeue an AP command-reply message from a queue > >> + * PQAP: to administer the queues > > So, NQAP/DQAP need usage domains, while PQAP needs a control domain? Or > > is it that all of them need usage domains, but PQAP can target a control > > domain as well? > > > > [I don't want to dive deeply into the AP architecture here, just far > > enough to really understand the design implications.] > Well, to be honest, nobody ever tried this under Linux. Theoretically > one should be able to send a CPRB to a usage domain where inside > the CPRB another domain (the control domain) is addressed. However, > as of now I am only aware of applications controlling the same usage > domain. I don't know any application which is able to address another > control domain and I am not sure if the zcrypt device driver would > handle such a CPRB correctly. NQAP, DQAP and PQAP always address > a usage domain. But the CPRB send down the pipe via NQAP may > address some control thing on another domain. I am not sure which > code and where do the sorting out here. There are two candidates: > the firmware layer in the CEC and the crypto card code. OK, so it's possible as by the architecture, but at least Linux does not (currently) do it? Perhaps we should simply not overthink that whole control domain thingy :) It's mostly yet another knob, and as long as the design does not go against the general architecture, it's probably fine, I guess.
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 20.08.2018 18:03, Cornelia Huck wrote: > On Mon, 13 Aug 2018 17:48:19 -0400 > Tony Krowiak wrote: > >> From: Tony Krowiak >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak >> Reviewed-by: Halil Pasic >> Signed-off-by: Christian Borntraeger >> --- >> Documentation/s390/vfio-ap.txt | 615 >> >> MAINTAINERS|1 + >> 2 files changed, 616 insertions(+), 0 deletions(-) >> create mode 100644 Documentation/s390/vfio-ap.txt >> >> +AP Architectural Overview: >> += >> +To facilitate the comprehension of the design, let's start with some >> +definitions: >> + >> +* AP adapter >> + >> + An AP adapter is an IBM Z adapter card that can perform cryptographic >> + functions. There can be from 0 to 256 adapters assigned to an LPAR. >> Adapters >> + assigned to the LPAR in which a linux host is running will be available to >> + the linux host. Each adapter is identified by a number from 0 to 255. When >> + installed, an AP adapter is accessed by AP instructions executed by any >> CPU. >> + >> + The AP adapter cards are assigned to a given LPAR via the system's >> Activation >> + Profile which can be edited via the HMC. When the system is IPL'd, the AP >> bus > There's lots of s390 jargon in here... but one hopes that someone > trying to understand AP is already familiar with the basics... > >> + module is loaded and detects the AP adapter cards assigned to the LPAR. >> The AP >> + bus creates a sysfs device for each adapter as they are detected. For >> example, >> + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will >> + create the following sysfs entries: >> + >> +/sys/devices/ap/card04 >> +/sys/devices/ap/card0a >> + >> + Symbolic links to these devices will also be created in the AP bus devices >> + sub-directory: >> + >> +/sys/bus/ap/devices/[card04] >> +/sys/bus/ap/devices/[card04] >> + >> +* AP domain >> + >> + An adapter is partitioned into domains. Each domain can be thought of as >> + a set of hardware registers for processing AP instructions. An adapter can >> + hold up to 256 domains. Each domain is identified by a number from 0 to >> 255. >> + Domains can be further classified into two types: >> + >> +* Usage domains are domains that can be accessed directly to process AP >> + commands. >> + >> +* Control domains are domains that are accessed indirectly by AP >> + commands sent to a usage domain to control or change the domain; for >> + example, to set a secure private key for the domain. >> + >> + The AP usage and control domains are assigned to a given LPAR via the >> system's >> + Activation Profile which can be edited via the HMC. When the system is >> IPL'd, >> + the AP bus module is loaded and detects the AP usage and control domains >> + assigned to the LPAR. The domain number of each usage domain will be >> coupled >> + with the adapter number of each AP adapter assigned to the LPAR to >> identify >> + the AP queues (see AP Queue section below). The domain number of each >> control >> + domain will be represented in a bitmask and stored in a sysfs file >> + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the >> mask, >> + from most to least significant bit, correspond to domains 0-255. >> + >> + A domain may be assigned to a system as both a usage and control domain, >> or >> + as a control domain only. Consequently, all domains assigned as both a >> usage >> + and control domain can both process AP commands as well as be changed by >> an AP >> + command sent to any usage domain assigned to the same system. Domains >> assigned >> + only as control domains can not process AP commands but can be changed by >> AP >> + commands sent to any usage domain assigned to the system. > I'm struggling a bit with this paragraph. Does that mean that you can > use control domains as the target of an instruction changing > configuration on the system? (Or on the VM, if they are listed in the > relevant control block?) Yes. You can send an CPRB to a (usage) domain which includes a command for controlling another (control) domain. > >> + >> +* AP Queue >> + >> + An AP queue is the means by which an AP command-request message is sent >> to a >> + usage domain inside a specific adapter. An AP queue is identified by a >> tuple >> + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The >> + APQI corresponds to a given usage domain number within the adapter. This >> tuple >> + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP >> + instructions include a field containing the APQN to identify the AP queue >> to >> + which the AP command-request
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 20.08.2018 18:03, Cornelia Huck wrote: > On Mon, 13 Aug 2018 17:48:19 -0400 > Tony Krowiak wrote: > >> From: Tony Krowiak >> >> This patch provides documentation describing the AP architecture and >> design concepts behind the virtualization of AP devices. It also >> includes an example of how to configure AP devices for exclusive >> use of KVM guests. >> >> Signed-off-by: Tony Krowiak >> Reviewed-by: Halil Pasic >> Signed-off-by: Christian Borntraeger >> --- >> Documentation/s390/vfio-ap.txt | 615 >> >> MAINTAINERS|1 + >> 2 files changed, 616 insertions(+), 0 deletions(-) >> create mode 100644 Documentation/s390/vfio-ap.txt >> >> +AP Architectural Overview: >> += >> +To facilitate the comprehension of the design, let's start with some >> +definitions: >> + >> +* AP adapter >> + >> + An AP adapter is an IBM Z adapter card that can perform cryptographic >> + functions. There can be from 0 to 256 adapters assigned to an LPAR. >> Adapters >> + assigned to the LPAR in which a linux host is running will be available to >> + the linux host. Each adapter is identified by a number from 0 to 255. When >> + installed, an AP adapter is accessed by AP instructions executed by any >> CPU. >> + >> + The AP adapter cards are assigned to a given LPAR via the system's >> Activation >> + Profile which can be edited via the HMC. When the system is IPL'd, the AP >> bus > There's lots of s390 jargon in here... but one hopes that someone > trying to understand AP is already familiar with the basics... > >> + module is loaded and detects the AP adapter cards assigned to the LPAR. >> The AP >> + bus creates a sysfs device for each adapter as they are detected. For >> example, >> + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will >> + create the following sysfs entries: >> + >> +/sys/devices/ap/card04 >> +/sys/devices/ap/card0a >> + >> + Symbolic links to these devices will also be created in the AP bus devices >> + sub-directory: >> + >> +/sys/bus/ap/devices/[card04] >> +/sys/bus/ap/devices/[card04] >> + >> +* AP domain >> + >> + An adapter is partitioned into domains. Each domain can be thought of as >> + a set of hardware registers for processing AP instructions. An adapter can >> + hold up to 256 domains. Each domain is identified by a number from 0 to >> 255. >> + Domains can be further classified into two types: >> + >> +* Usage domains are domains that can be accessed directly to process AP >> + commands. >> + >> +* Control domains are domains that are accessed indirectly by AP >> + commands sent to a usage domain to control or change the domain; for >> + example, to set a secure private key for the domain. >> + >> + The AP usage and control domains are assigned to a given LPAR via the >> system's >> + Activation Profile which can be edited via the HMC. When the system is >> IPL'd, >> + the AP bus module is loaded and detects the AP usage and control domains >> + assigned to the LPAR. The domain number of each usage domain will be >> coupled >> + with the adapter number of each AP adapter assigned to the LPAR to >> identify >> + the AP queues (see AP Queue section below). The domain number of each >> control >> + domain will be represented in a bitmask and stored in a sysfs file >> + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the >> mask, >> + from most to least significant bit, correspond to domains 0-255. >> + >> + A domain may be assigned to a system as both a usage and control domain, >> or >> + as a control domain only. Consequently, all domains assigned as both a >> usage >> + and control domain can both process AP commands as well as be changed by >> an AP >> + command sent to any usage domain assigned to the same system. Domains >> assigned >> + only as control domains can not process AP commands but can be changed by >> AP >> + commands sent to any usage domain assigned to the system. > I'm struggling a bit with this paragraph. Does that mean that you can > use control domains as the target of an instruction changing > configuration on the system? (Or on the VM, if they are listed in the > relevant control block?) Yes. You can send an CPRB to a (usage) domain which includes a command for controlling another (control) domain. > >> + >> +* AP Queue >> + >> + An AP queue is the means by which an AP command-request message is sent >> to a >> + usage domain inside a specific adapter. An AP queue is identified by a >> tuple >> + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The >> + APQI corresponds to a given usage domain number within the adapter. This >> tuple >> + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP >> + instructions include a field containing the APQN to identify the AP queue >> to >> + which the AP command-request
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/20/2018 12:03 PM, Cornelia Huck wrote: On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic Signed-off-by: Christian Borntraeger --- Documentation/s390/vfio-ap.txt | 615 MAINTAINERS|1 + 2 files changed, 616 insertions(+), 0 deletions(-) create mode 100644 Documentation/s390/vfio-ap.txt +AP Architectural Overview: += +To facilitate the comprehension of the design, let's start with some +definitions: + +* AP adapter + + An AP adapter is an IBM Z adapter card that can perform cryptographic + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters + assigned to the LPAR in which a linux host is running will be available to + the linux host. Each adapter is identified by a number from 0 to 255. When + installed, an AP adapter is accessed by AP instructions executed by any CPU. + + The AP adapter cards are assigned to a given LPAR via the system's Activation + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... I'm not quite sure how one can describe s390-specific devices that can be installed only on an s390 system without using s390 jargon. I would think that one who is administering a linux host or guest running on an s390 system would have some basic knowledge of s390. If you have any suggestions, I'd be happy to entertain them. + module is loaded and detects the AP adapter cards assigned to the LPAR. The AP + bus creates a sysfs device for each adapter as they are detected. For example, + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will + create the following sysfs entries: + +/sys/devices/ap/card04 +/sys/devices/ap/card0a + + Symbolic links to these devices will also be created in the AP bus devices + sub-directory: + +/sys/bus/ap/devices/[card04] +/sys/bus/ap/devices/[card04] + +* AP domain + + An adapter is partitioned into domains. Each domain can be thought of as + a set of hardware registers for processing AP instructions. An adapter can + hold up to 256 domains. Each domain is identified by a number from 0 to 255. + Domains can be further classified into two types: + +* Usage domains are domains that can be accessed directly to process AP + commands. + +* Control domains are domains that are accessed indirectly by AP + commands sent to a usage domain to control or change the domain; for + example, to set a secure private key for the domain. + + The AP usage and control domains are assigned to a given LPAR via the system's + Activation Profile which can be edited via the HMC. When the system is IPL'd, + the AP bus module is loaded and detects the AP usage and control domains + assigned to the LPAR. The domain number of each usage domain will be coupled + with the adapter number of each AP adapter assigned to the LPAR to identify + the AP queues (see AP Queue section below). The domain number of each control + domain will be represented in a bitmask and stored in a sysfs file + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the mask, + from most to least significant bit, correspond to domains 0-255. + + A domain may be assigned to a system as both a usage and control domain, or + as a control domain only. Consequently, all domains assigned as both a usage + and control domain can both process AP commands as well as be changed by an AP + command sent to any usage domain assigned to the same system. Domains assigned + only as control domains can not process AP commands but can be changed by AP + commands sent to any usage domain assigned to the system. I'm struggling a bit with this paragraph. Does that mean that you can use control domains as the target of an instruction changing configuration on the system? (Or on the VM, if they are listed in the relevant control block?) Only usage domains can be the target of an AP command request message. If an AP message sent to a usage domain is a request to change a domain, the number of the domain to be changed will be contained in the command request message. That domain number must be configured as a control domain or the AP command will fail. The fact you are struggling with understanding the last paragraph leads me to believe it should probably be rewritten, or eliminated. Allow me to reconsider this section. + +* AP Queue + + An AP queue is the means by which an AP command-request message is sent to a + usage domain inside a specific
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On 08/20/2018 12:03 PM, Cornelia Huck wrote: On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: From: Tony Krowiak This patch provides documentation describing the AP architecture and design concepts behind the virtualization of AP devices. It also includes an example of how to configure AP devices for exclusive use of KVM guests. Signed-off-by: Tony Krowiak Reviewed-by: Halil Pasic Signed-off-by: Christian Borntraeger --- Documentation/s390/vfio-ap.txt | 615 MAINTAINERS|1 + 2 files changed, 616 insertions(+), 0 deletions(-) create mode 100644 Documentation/s390/vfio-ap.txt +AP Architectural Overview: += +To facilitate the comprehension of the design, let's start with some +definitions: + +* AP adapter + + An AP adapter is an IBM Z adapter card that can perform cryptographic + functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters + assigned to the LPAR in which a linux host is running will be available to + the linux host. Each adapter is identified by a number from 0 to 255. When + installed, an AP adapter is accessed by AP instructions executed by any CPU. + + The AP adapter cards are assigned to a given LPAR via the system's Activation + Profile which can be edited via the HMC. When the system is IPL'd, the AP bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... I'm not quite sure how one can describe s390-specific devices that can be installed only on an s390 system without using s390 jargon. I would think that one who is administering a linux host or guest running on an s390 system would have some basic knowledge of s390. If you have any suggestions, I'd be happy to entertain them. + module is loaded and detects the AP adapter cards assigned to the LPAR. The AP + bus creates a sysfs device for each adapter as they are detected. For example, + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will + create the following sysfs entries: + +/sys/devices/ap/card04 +/sys/devices/ap/card0a + + Symbolic links to these devices will also be created in the AP bus devices + sub-directory: + +/sys/bus/ap/devices/[card04] +/sys/bus/ap/devices/[card04] + +* AP domain + + An adapter is partitioned into domains. Each domain can be thought of as + a set of hardware registers for processing AP instructions. An adapter can + hold up to 256 domains. Each domain is identified by a number from 0 to 255. + Domains can be further classified into two types: + +* Usage domains are domains that can be accessed directly to process AP + commands. + +* Control domains are domains that are accessed indirectly by AP + commands sent to a usage domain to control or change the domain; for + example, to set a secure private key for the domain. + + The AP usage and control domains are assigned to a given LPAR via the system's + Activation Profile which can be edited via the HMC. When the system is IPL'd, + the AP bus module is loaded and detects the AP usage and control domains + assigned to the LPAR. The domain number of each usage domain will be coupled + with the adapter number of each AP adapter assigned to the LPAR to identify + the AP queues (see AP Queue section below). The domain number of each control + domain will be represented in a bitmask and stored in a sysfs file + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the mask, + from most to least significant bit, correspond to domains 0-255. + + A domain may be assigned to a system as both a usage and control domain, or + as a control domain only. Consequently, all domains assigned as both a usage + and control domain can both process AP commands as well as be changed by an AP + command sent to any usage domain assigned to the same system. Domains assigned + only as control domains can not process AP commands but can be changed by AP + commands sent to any usage domain assigned to the system. I'm struggling a bit with this paragraph. Does that mean that you can use control domains as the target of an instruction changing configuration on the system? (Or on the VM, if they are listed in the relevant control block?) Only usage domains can be the target of an AP command request message. If an AP message sent to a usage domain is a request to change a domain, the number of the domain to be changed will be contained in the command request message. That domain number must be configured as a control domain or the AP command will fail. The fact you are struggling with understanding the last paragraph leads me to believe it should probably be rewritten, or eliminated. Allow me to reconsider this section. + +* AP Queue + + An AP queue is the means by which an AP command-request message is sent to a + usage domain inside a specific
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: > From: Tony Krowiak > > This patch provides documentation describing the AP architecture and > design concepts behind the virtualization of AP devices. It also > includes an example of how to configure AP devices for exclusive > use of KVM guests. > > Signed-off-by: Tony Krowiak > Reviewed-by: Halil Pasic > Signed-off-by: Christian Borntraeger > --- > Documentation/s390/vfio-ap.txt | 615 > > MAINTAINERS|1 + > 2 files changed, 616 insertions(+), 0 deletions(-) > create mode 100644 Documentation/s390/vfio-ap.txt > > +AP Architectural Overview: > += > +To facilitate the comprehension of the design, let's start with some > +definitions: > + > +* AP adapter > + > + An AP adapter is an IBM Z adapter card that can perform cryptographic > + functions. There can be from 0 to 256 adapters assigned to an LPAR. > Adapters > + assigned to the LPAR in which a linux host is running will be available to > + the linux host. Each adapter is identified by a number from 0 to 255. When > + installed, an AP adapter is accessed by AP instructions executed by any > CPU. > + > + The AP adapter cards are assigned to a given LPAR via the system's > Activation > + Profile which can be edited via the HMC. When the system is IPL'd, the AP > bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... > + module is loaded and detects the AP adapter cards assigned to the LPAR. > The AP > + bus creates a sysfs device for each adapter as they are detected. For > example, > + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will > + create the following sysfs entries: > + > +/sys/devices/ap/card04 > +/sys/devices/ap/card0a > + > + Symbolic links to these devices will also be created in the AP bus devices > + sub-directory: > + > +/sys/bus/ap/devices/[card04] > +/sys/bus/ap/devices/[card04] > + > +* AP domain > + > + An adapter is partitioned into domains. Each domain can be thought of as > + a set of hardware registers for processing AP instructions. An adapter can > + hold up to 256 domains. Each domain is identified by a number from 0 to > 255. > + Domains can be further classified into two types: > + > +* Usage domains are domains that can be accessed directly to process AP > + commands. > + > +* Control domains are domains that are accessed indirectly by AP > + commands sent to a usage domain to control or change the domain; for > + example, to set a secure private key for the domain. > + > + The AP usage and control domains are assigned to a given LPAR via the > system's > + Activation Profile which can be edited via the HMC. When the system is > IPL'd, > + the AP bus module is loaded and detects the AP usage and control domains > + assigned to the LPAR. The domain number of each usage domain will be > coupled > + with the adapter number of each AP adapter assigned to the LPAR to identify > + the AP queues (see AP Queue section below). The domain number of each > control > + domain will be represented in a bitmask and stored in a sysfs file > + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the > mask, > + from most to least significant bit, correspond to domains 0-255. > + > + A domain may be assigned to a system as both a usage and control domain, or > + as a control domain only. Consequently, all domains assigned as both a > usage > + and control domain can both process AP commands as well as be changed by > an AP > + command sent to any usage domain assigned to the same system. Domains > assigned > + only as control domains can not process AP commands but can be changed by > AP > + commands sent to any usage domain assigned to the system. I'm struggling a bit with this paragraph. Does that mean that you can use control domains as the target of an instruction changing configuration on the system? (Or on the VM, if they are listed in the relevant control block?) > + > +* AP Queue > + > + An AP queue is the means by which an AP command-request message is sent to > a > + usage domain inside a specific adapter. An AP queue is identified by a > tuple > + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The > + APQI corresponds to a given usage domain number within the adapter. This > tuple > + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP > + instructions include a field containing the APQN to identify the AP queue > to > + which the AP command-request message is to be sent for processing. > + > + The AP bus will create a sysfs device for each APQN that can be derived > from > + the cross product of the AP adapter and usage domain numbers detected when > the > + AP bus module is loaded. For example, if adapters 4 and 10
Re: [PATCH v9 22/22] s390: doc: detailed specifications for AP virtualization
On Mon, 13 Aug 2018 17:48:19 -0400 Tony Krowiak wrote: > From: Tony Krowiak > > This patch provides documentation describing the AP architecture and > design concepts behind the virtualization of AP devices. It also > includes an example of how to configure AP devices for exclusive > use of KVM guests. > > Signed-off-by: Tony Krowiak > Reviewed-by: Halil Pasic > Signed-off-by: Christian Borntraeger > --- > Documentation/s390/vfio-ap.txt | 615 > > MAINTAINERS|1 + > 2 files changed, 616 insertions(+), 0 deletions(-) > create mode 100644 Documentation/s390/vfio-ap.txt > > +AP Architectural Overview: > += > +To facilitate the comprehension of the design, let's start with some > +definitions: > + > +* AP adapter > + > + An AP adapter is an IBM Z adapter card that can perform cryptographic > + functions. There can be from 0 to 256 adapters assigned to an LPAR. > Adapters > + assigned to the LPAR in which a linux host is running will be available to > + the linux host. Each adapter is identified by a number from 0 to 255. When > + installed, an AP adapter is accessed by AP instructions executed by any > CPU. > + > + The AP adapter cards are assigned to a given LPAR via the system's > Activation > + Profile which can be edited via the HMC. When the system is IPL'd, the AP > bus There's lots of s390 jargon in here... but one hopes that someone trying to understand AP is already familiar with the basics... > + module is loaded and detects the AP adapter cards assigned to the LPAR. > The AP > + bus creates a sysfs device for each adapter as they are detected. For > example, > + if AP adapters 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will > + create the following sysfs entries: > + > +/sys/devices/ap/card04 > +/sys/devices/ap/card0a > + > + Symbolic links to these devices will also be created in the AP bus devices > + sub-directory: > + > +/sys/bus/ap/devices/[card04] > +/sys/bus/ap/devices/[card04] > + > +* AP domain > + > + An adapter is partitioned into domains. Each domain can be thought of as > + a set of hardware registers for processing AP instructions. An adapter can > + hold up to 256 domains. Each domain is identified by a number from 0 to > 255. > + Domains can be further classified into two types: > + > +* Usage domains are domains that can be accessed directly to process AP > + commands. > + > +* Control domains are domains that are accessed indirectly by AP > + commands sent to a usage domain to control or change the domain; for > + example, to set a secure private key for the domain. > + > + The AP usage and control domains are assigned to a given LPAR via the > system's > + Activation Profile which can be edited via the HMC. When the system is > IPL'd, > + the AP bus module is loaded and detects the AP usage and control domains > + assigned to the LPAR. The domain number of each usage domain will be > coupled > + with the adapter number of each AP adapter assigned to the LPAR to identify > + the AP queues (see AP Queue section below). The domain number of each > control > + domain will be represented in a bitmask and stored in a sysfs file > + /sys/bus/ap/ap_control_domain_mask created by the bus. The bits in the > mask, > + from most to least significant bit, correspond to domains 0-255. > + > + A domain may be assigned to a system as both a usage and control domain, or > + as a control domain only. Consequently, all domains assigned as both a > usage > + and control domain can both process AP commands as well as be changed by > an AP > + command sent to any usage domain assigned to the same system. Domains > assigned > + only as control domains can not process AP commands but can be changed by > AP > + commands sent to any usage domain assigned to the system. I'm struggling a bit with this paragraph. Does that mean that you can use control domains as the target of an instruction changing configuration on the system? (Or on the VM, if they are listed in the relevant control block?) > + > +* AP Queue > + > + An AP queue is the means by which an AP command-request message is sent to > a > + usage domain inside a specific adapter. An AP queue is identified by a > tuple > + comprised of an AP adapter ID (APID) and an AP queue index (APQI). The > + APQI corresponds to a given usage domain number within the adapter. This > tuple > + forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP > + instructions include a field containing the APQN to identify the AP queue > to > + which the AP command-request message is to be sent for processing. > + > + The AP bus will create a sysfs device for each APQN that can be derived > from > + the cross product of the AP adapter and usage domain numbers detected when > the > + AP bus module is loaded. For example, if adapters 4 and 10
