Re: [Xen-devel] [DOC v7] PV Calls protocol design
On Fri, 20 Jan 2017, Konrad Rzeszutek Wilk wrote: > On Tue, Jan 10, 2017 at 04:13:26PM -0800, Stefano Stabellini wrote: > > Upon request from Konrad, I am attaching the output of pahole on the C > > structs defined by PVCalls. As you can see, alignments and sizes of all > > Thank you! > > fields are the same, except for the padding at the end of many request > > structs (they need to be multiple of 8 bytes on 64bit). However, struct > > Would it make sense to add > > #ifdef CONFIG_X86_32 > uint8_t[X] _pad; > #endif > > so it will nicely fill it out? I added them for each struct in the union too. > > xen_pvcalls_dummy dummy ensures that the overall size of struct > > xen_pvcalls_request is the same on both 32 and 64. > > Yes indeed. Thank you! > > > > # PV Calls Protocol version 1 > > > > > > ## Glossary > > > > > > The following is a list of terms and definitions used in the Xen > > > community. If you are a Xen contributor you can skip this section. > > > > > > * PV > > > > > > Short for paravirtualized. > > > > > > * Dom0 > > > > > > First virtual machine that boots. In most configurations Dom0 is > > > privileged and has control over hardware devices, such as network > > > cards, graphic cards, etc. > > > > > > * DomU > > > > > > Regular unprivileged Xen virtual machine. > > > > > > * Domain > > > > > > A Xen virtual machine. Dom0 and all DomUs are all separate Xen > > > domains. > > > > > > * Guest > > > > > > Same as domain: a Xen virtual machine. > > > > > > * Frontend > > > > > > Each DomU has one or more paravirtualized frontend drivers to access > > > disks, network, console, graphics, etc. The presence of PV devices is > > > advertized on XenStore, a cross domain key-value database. Frontends > > > are similar in intent to the virtio drivers in Linux. > > > > > > * Backend > > > > > > A Xen paravirtualized backend typically runs in Dom0 and it is used to > > > export disks, network, console, graphics, etcs, to DomUs. Backends can > > > live both in kernel space and in userspace. For example xen-blkback > > > lives under drivers/block in the Linux kernel and xen_disk lives under > > > hw/block in QEMU. Paravirtualized backends are similar in intent to > > > virtio device emulators. > > > > > > * VMX and SVM > > > > > > On Intel processors, VMX is the CPU flag for VT-x, hardware > > > virtualization support. It corresponds to SVM on AMD processors. > > > > > > > > > > > > ## Rationale > > > > > > PV Calls is a paravirtualized protocol that allows the implementation of > > > a set of POSIX functions in a different domain. The PV Calls frontend > > > sends POSIX function calls to the backend, which implements them and > > > returns a value to the frontend. > > "returns a value to the frontend and acts on the function call." OK > > > > > > This version of the document covers networking function calls, such as > > > connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but > > > the protocol is meant to be easily extended to cover different sets of > > > calls. Unimplemented commands return ENOTSUP. > > > > > > PV Calls provide the following benefits: > > > * full visibility of the guest behavior on the backend domain, allowing > > > for inexpensive filtering and manipulation of any guest calls > > > * excellent performance > > > > > > Specifically, PV Calls for networking offer these advantages: > > > * guest networking works out of the box with VPNs, wireless networks and > > > any other complex configurations on the host > > > * guest services listen on ports bound directly to the backend domain IP > > > addresses > > > * localhost becomes a secure namespace for inter-VMs communications > > > > > [Not sure I understand that but it probably is explained in detail later on?] > > Does it make sense to define what 'namespace' is ? I mean I know what it is > in context of C++ but I presume with cgroups and such it is different? I reworded it to: * localhost becomes a secure host wide network for inter-VMs communications > > > > > > ## Design > > > > > > ### Why Xen? > > > > > > PV Calls are part of an effort to create a secure runtime environment > > > for containers (OCI images to be precise). PV Calls are based on Xen, > > OCI = Open Containers Initiative? That's right > Perhaps mention that in glossary or just spell it out here, like: > > s/OCI images/Open Containers Initiative, aka OCI/ OK > > > although porting them to other hypervisors is possible. Xen was chosen > > > because of its security and isolation properties and because it supports > > > PV guests, a type of virtual machines that does not require hardware > > > virtualization extensions (VMX on Intel processors and SVM on AMD > > > processors). This is important because PV Calls is meant for containers > > > and containers are often run on top of public cloud instances, which do > > > not support nested VMX (or SVM) as
Re: [Xen-devel] [DOC v7] PV Calls protocol design
On Tue, Jan 10, 2017 at 04:13:26PM -0800, Stefano Stabellini wrote: > Upon request from Konrad, I am attaching the output of pahole on the C > structs defined by PVCalls. As you can see, alignments and sizes of all Thank you! > fields are the same, except for the padding at the end of many request > structs (they need to be multiple of 8 bytes on 64bit). However, struct Would it make sense to add #ifdef CONFIG_X86_32 uint8_t[X] _pad; #endif so it will nicely fill it out? > xen_pvcalls_dummy dummy ensures that the overall size of struct > xen_pvcalls_request is the same on both 32 and 64. Yes indeed. Thank you! > > # PV Calls Protocol version 1 > > > > ## Glossary > > > > The following is a list of terms and definitions used in the Xen > > community. If you are a Xen contributor you can skip this section. > > > > * PV > > > > Short for paravirtualized. > > > > * Dom0 > > > > First virtual machine that boots. In most configurations Dom0 is > > privileged and has control over hardware devices, such as network > > cards, graphic cards, etc. > > > > * DomU > > > > Regular unprivileged Xen virtual machine. > > > > * Domain > > > > A Xen virtual machine. Dom0 and all DomUs are all separate Xen > > domains. > > > > * Guest > > > > Same as domain: a Xen virtual machine. > > > > * Frontend > > > > Each DomU has one or more paravirtualized frontend drivers to access > > disks, network, console, graphics, etc. The presence of PV devices is > > advertized on XenStore, a cross domain key-value database. Frontends > > are similar in intent to the virtio drivers in Linux. > > > > * Backend > > > > A Xen paravirtualized backend typically runs in Dom0 and it is used to > > export disks, network, console, graphics, etcs, to DomUs. Backends can > > live both in kernel space and in userspace. For example xen-blkback > > lives under drivers/block in the Linux kernel and xen_disk lives under > > hw/block in QEMU. Paravirtualized backends are similar in intent to > > virtio device emulators. > > > > * VMX and SVM > > > > On Intel processors, VMX is the CPU flag for VT-x, hardware > > virtualization support. It corresponds to SVM on AMD processors. > > > > > > > > ## Rationale > > > > PV Calls is a paravirtualized protocol that allows the implementation of > > a set of POSIX functions in a different domain. The PV Calls frontend > > sends POSIX function calls to the backend, which implements them and > > returns a value to the frontend. "returns a value to the frontend and acts on the function call." > > > > This version of the document covers networking function calls, such as > > connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but > > the protocol is meant to be easily extended to cover different sets of > > calls. Unimplemented commands return ENOTSUP. > > > > PV Calls provide the following benefits: > > * full visibility of the guest behavior on the backend domain, allowing > > for inexpensive filtering and manipulation of any guest calls > > * excellent performance > > > > Specifically, PV Calls for networking offer these advantages: > > * guest networking works out of the box with VPNs, wireless networks and > > any other complex configurations on the host > > * guest services listen on ports bound directly to the backend domain IP > > addresses > > * localhost becomes a secure namespace for inter-VMs communications > > [Not sure I understand that but it probably is explained in detail later on?] Does it make sense to define what 'namespace' is ? I mean I know what it is in context of C++ but I presume with cgroups and such it is different? > > > > ## Design > > > > ### Why Xen? > > > > PV Calls are part of an effort to create a secure runtime environment > > for containers (OCI images to be precise). PV Calls are based on Xen, OCI = Open Containers Initiative? Perhaps mention that in glossary or just spell it out here, like: s/OCI images/Open Containers Initiative, aka OCI/ > > although porting them to other hypervisors is possible. Xen was chosen > > because of its security and isolation properties and because it supports > > PV guests, a type of virtual machines that does not require hardware > > virtualization extensions (VMX on Intel processors and SVM on AMD > > processors). This is important because PV Calls is meant for containers > > and containers are often run on top of public cloud instances, which do > > not support nested VMX (or SVM) as of today (late 2016). Xen PV guests s/late 2016/early 2017/ > > are lightweight, minimalist, and do not require machine emulation: all > > properties that make them a good fit for the project. s/the project/this project/ > > > > ### Xenstore > > > > The frontend and the backend connect via [xenstore] to > > exchange information. The toolstack creates front and back nodes with > > state [XenbusStateInitialising]. The protocol node name s/state/state of/ > > is
Re: [Xen-devel] [DOC v7] PV Calls protocol design
Upon request from Konrad, I am attaching the output of pahole on the C structs defined by PVCalls. As you can see, alignments and sizes of all fields are the same, except for the padding at the end of many request structs (they need to be multiple of 8 bytes on 64bit). However, struct xen_pvcalls_dummy dummy ensures that the overall size of struct xen_pvcalls_request is the same on both 32 and 64. On Thu, 13 Oct 2016, Stefano Stabellini wrote: > Hi all, > > This is the design document of the PV Calls protocol. You can find > prototypes of the Linux frontend and backend drivers here: > > git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-7 > > To use them, make sure to enable CONFIG_XEN_PVCALLS in your kernel > config and add "pvcalls=1" to the command line of your DomU Linux > kernel. You also need the toolstack to create the initial xenstore nodes > for the protocol. To do that, please apply the attached patch to libxl > (the patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU > config file. > > Please review! > > Cheers, > > Stefano > > > Changes in v7: > - add a glossary of Xen terms > - add a paragraph on why Xen was chosen > - wording improvements > - add links to xenstore documents and headers > - specify that the current version is "1" > - rename max-dataring-page-order to max-page-order > - rename networking-calls to function-calls > - add links to [Data ring] throughout the document > - explain the difference between req_id and id > - mention that future commands larger than 56 bytes will require a > version increase > - mention that the list of commands is in calling order > - clarify that reuse data rings are found by ref > - rename ENOTSUPP to ENOTSUP > - add padding in struct pvcalls_data_intf for cachelining > - rename pvcalls_ring_queued to pvcalls_ring_unconsumed > > > Changes in v6: > - add reuse field in release command > - add "networking-calls" backend node on xenstore > - fixed tab/whitespace indentation > > Changes in v5: > - clarify text > - rename id to req_id > - rename sockid to id > - move id to request and response specific fields > - add version node to xenstore > > Changes in v4: > - rename xensock to pvcalls > > Changes in v3: > - add a dummy element to struct xen_xensock_request to make sure the > size of the struct is the same on both x86_32 and x86_64 > > Changes in v2: > - add max-dataring-page-order > - add "Publish backend features and transport parameters" to backend > xenbus workflow > - update new cmd values > - update xen_xensock_request > - add backlog parameter to listen and binary layout > - add description of new data ring format (interface+data) > - modify connect and accept to reflect new data ring format > - add link to POSIX docs > - add error numbers > - add address format section and relevant numeric definitions > - add explicit mention of unimplemented commands > - add protocol node name > - add xenbus shutdown diagram > - add socket operation > > --- > > > # PV Calls Protocol version 1 > > ## Glossary > > The following is a list of terms and definitions used in the Xen > community. If you are a Xen contributor you can skip this section. > > * PV > > Short for paravirtualized. > > * Dom0 > > First virtual machine that boots. In most configurations Dom0 is > privileged and has control over hardware devices, such as network > cards, graphic cards, etc. > > * DomU > > Regular unprivileged Xen virtual machine. > > * Domain > > A Xen virtual machine. Dom0 and all DomUs are all separate Xen > domains. > > * Guest > > Same as domain: a Xen virtual machine. > > * Frontend > > Each DomU has one or more paravirtualized frontend drivers to access > disks, network, console, graphics, etc. The presence of PV devices is > advertized on XenStore, a cross domain key-value database. Frontends > are similar in intent to the virtio drivers in Linux. > > * Backend > > A Xen paravirtualized backend typically runs in Dom0 and it is used to > export disks, network, console, graphics, etcs, to DomUs. Backends can > live both in kernel space and in userspace. For example xen-blkback > lives under drivers/block in the Linux kernel and xen_disk lives under > hw/block in QEMU. Paravirtualized backends are similar in intent to > virtio device emulators. > > * VMX and SVM > > On Intel processors, VMX is the CPU flag for VT-x, hardware > virtualization support. It corresponds to SVM on AMD processors. > > > > ## Rationale > > PV Calls is a paravirtualized protocol that allows the implementation of > a set of POSIX functions in a different domain. The PV Calls frontend > sends POSIX function calls to the backend, which implements them and > returns a value to the frontend. > > This version of the document covers networking function calls, such as > connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but > the protocol is meant to be easily
Re: [Xen-devel] [DOC v7] PV Calls protocol design
ping? On Thu, 13 Oct 2016, Stefano Stabellini wrote: > Hi all, > > This is the design document of the PV Calls protocol. You can find > prototypes of the Linux frontend and backend drivers here: > > git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-7 > > To use them, make sure to enable CONFIG_XEN_PVCALLS in your kernel > config and add "pvcalls=1" to the command line of your DomU Linux > kernel. You also need the toolstack to create the initial xenstore nodes > for the protocol. To do that, please apply the attached patch to libxl > (the patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU > config file. > > Please review! > > Cheers, > > Stefano > > > Changes in v7: > - add a glossary of Xen terms > - add a paragraph on why Xen was chosen > - wording improvements > - add links to xenstore documents and headers > - specify that the current version is "1" > - rename max-dataring-page-order to max-page-order > - rename networking-calls to function-calls > - add links to [Data ring] throughout the document > - explain the difference between req_id and id > - mention that future commands larger than 56 bytes will require a > version increase > - mention that the list of commands is in calling order > - clarify that reuse data rings are found by ref > - rename ENOTSUPP to ENOTSUP > - add padding in struct pvcalls_data_intf for cachelining > - rename pvcalls_ring_queued to pvcalls_ring_unconsumed > > > Changes in v6: > - add reuse field in release command > - add "networking-calls" backend node on xenstore > - fixed tab/whitespace indentation > > Changes in v5: > - clarify text > - rename id to req_id > - rename sockid to id > - move id to request and response specific fields > - add version node to xenstore > > Changes in v4: > - rename xensock to pvcalls > > Changes in v3: > - add a dummy element to struct xen_xensock_request to make sure the > size of the struct is the same on both x86_32 and x86_64 > > Changes in v2: > - add max-dataring-page-order > - add "Publish backend features and transport parameters" to backend > xenbus workflow > - update new cmd values > - update xen_xensock_request > - add backlog parameter to listen and binary layout > - add description of new data ring format (interface+data) > - modify connect and accept to reflect new data ring format > - add link to POSIX docs > - add error numbers > - add address format section and relevant numeric definitions > - add explicit mention of unimplemented commands > - add protocol node name > - add xenbus shutdown diagram > - add socket operation > > --- > > > # PV Calls Protocol version 1 > > ## Glossary > > The following is a list of terms and definitions used in the Xen > community. If you are a Xen contributor you can skip this section. > > * PV > > Short for paravirtualized. > > * Dom0 > > First virtual machine that boots. In most configurations Dom0 is > privileged and has control over hardware devices, such as network > cards, graphic cards, etc. > > * DomU > > Regular unprivileged Xen virtual machine. > > * Domain > > A Xen virtual machine. Dom0 and all DomUs are all separate Xen > domains. > > * Guest > > Same as domain: a Xen virtual machine. > > * Frontend > > Each DomU has one or more paravirtualized frontend drivers to access > disks, network, console, graphics, etc. The presence of PV devices is > advertized on XenStore, a cross domain key-value database. Frontends > are similar in intent to the virtio drivers in Linux. > > * Backend > > A Xen paravirtualized backend typically runs in Dom0 and it is used to > export disks, network, console, graphics, etcs, to DomUs. Backends can > live both in kernel space and in userspace. For example xen-blkback > lives under drivers/block in the Linux kernel and xen_disk lives under > hw/block in QEMU. Paravirtualized backends are similar in intent to > virtio device emulators. > > * VMX and SVM > > On Intel processors, VMX is the CPU flag for VT-x, hardware > virtualization support. It corresponds to SVM on AMD processors. > > > > ## Rationale > > PV Calls is a paravirtualized protocol that allows the implementation of > a set of POSIX functions in a different domain. The PV Calls frontend > sends POSIX function calls to the backend, which implements them and > returns a value to the frontend. > > This version of the document covers networking function calls, such as > connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but > the protocol is meant to be easily extended to cover different sets of > calls. Unimplemented commands return ENOTSUP. > > PV Calls provide the following benefits: > * full visibility of the guest behavior on the backend domain, allowing > for inexpensive filtering and manipulation of any guest calls > * excellent performance > > Specifically, PV Calls for networking offer these advantages: > * guest networking works out of the
Re: [Xen-devel] [DOC v7] PV Calls protocol design
On Thu, 13 Oct 2016, Stefano Stabellini wrote: > Hi all, > > This is the design document of the PV Calls protocol. You can find > prototypes of the Linux frontend and backend drivers here: > > git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-7 > > To use them, make sure to enable CONFIG_XEN_PVCALLS in your kernel > config and add "pvcalls=1" to the command line of your DomU Linux > kernel. You also need the toolstack to create the initial xenstore nodes > for the protocol. To do that, please apply the attached patch to libxl > (the patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU > config file. There is a small mistake in the patch attached to my previous email, which breaks cases where pvcalls is unspecified. This patch should work as expected.diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5 index a4cc1b3..fd67429 100644 --- a/docs/man/xl.cfg.pod.5 +++ b/docs/man/xl.cfg.pod.5 @@ -488,6 +488,11 @@ Specifies the networking provision (both emulated network adapters, and Xen virtual interfaces) to provided to the guest. See F. +=item
[Xen-devel] [DOC v7] PV Calls protocol design
Hi all, This is the design document of the PV Calls protocol. You can find prototypes of the Linux frontend and backend drivers here: git://git.kernel.org/pub/scm/linux/kernel/git/sstabellini/xen.git pvcalls-7 To use them, make sure to enable CONFIG_XEN_PVCALLS in your kernel config and add "pvcalls=1" to the command line of your DomU Linux kernel. You also need the toolstack to create the initial xenstore nodes for the protocol. To do that, please apply the attached patch to libxl (the patch is based on Xen 4.7.0-rc3) and add "pvcalls=1" to your DomU config file. Please review! Cheers, Stefano Changes in v7: - add a glossary of Xen terms - add a paragraph on why Xen was chosen - wording improvements - add links to xenstore documents and headers - specify that the current version is "1" - rename max-dataring-page-order to max-page-order - rename networking-calls to function-calls - add links to [Data ring] throughout the document - explain the difference between req_id and id - mention that future commands larger than 56 bytes will require a version increase - mention that the list of commands is in calling order - clarify that reuse data rings are found by ref - rename ENOTSUPP to ENOTSUP - add padding in struct pvcalls_data_intf for cachelining - rename pvcalls_ring_queued to pvcalls_ring_unconsumed Changes in v6: - add reuse field in release command - add "networking-calls" backend node on xenstore - fixed tab/whitespace indentation Changes in v5: - clarify text - rename id to req_id - rename sockid to id - move id to request and response specific fields - add version node to xenstore Changes in v4: - rename xensock to pvcalls Changes in v3: - add a dummy element to struct xen_xensock_request to make sure the size of the struct is the same on both x86_32 and x86_64 Changes in v2: - add max-dataring-page-order - add "Publish backend features and transport parameters" to backend xenbus workflow - update new cmd values - update xen_xensock_request - add backlog parameter to listen and binary layout - add description of new data ring format (interface+data) - modify connect and accept to reflect new data ring format - add link to POSIX docs - add error numbers - add address format section and relevant numeric definitions - add explicit mention of unimplemented commands - add protocol node name - add xenbus shutdown diagram - add socket operation --- # PV Calls Protocol version 1 ## Glossary The following is a list of terms and definitions used in the Xen community. If you are a Xen contributor you can skip this section. * PV Short for paravirtualized. * Dom0 First virtual machine that boots. In most configurations Dom0 is privileged and has control over hardware devices, such as network cards, graphic cards, etc. * DomU Regular unprivileged Xen virtual machine. * Domain A Xen virtual machine. Dom0 and all DomUs are all separate Xen domains. * Guest Same as domain: a Xen virtual machine. * Frontend Each DomU has one or more paravirtualized frontend drivers to access disks, network, console, graphics, etc. The presence of PV devices is advertized on XenStore, a cross domain key-value database. Frontends are similar in intent to the virtio drivers in Linux. * Backend A Xen paravirtualized backend typically runs in Dom0 and it is used to export disks, network, console, graphics, etcs, to DomUs. Backends can live both in kernel space and in userspace. For example xen-blkback lives under drivers/block in the Linux kernel and xen_disk lives under hw/block in QEMU. Paravirtualized backends are similar in intent to virtio device emulators. * VMX and SVM On Intel processors, VMX is the CPU flag for VT-x, hardware virtualization support. It corresponds to SVM on AMD processors. ## Rationale PV Calls is a paravirtualized protocol that allows the implementation of a set of POSIX functions in a different domain. The PV Calls frontend sends POSIX function calls to the backend, which implements them and returns a value to the frontend. This version of the document covers networking function calls, such as connect, accept, bind, release, listen, poll, recvmsg and sendmsg; but the protocol is meant to be easily extended to cover different sets of calls. Unimplemented commands return ENOTSUP. PV Calls provide the following benefits: * full visibility of the guest behavior on the backend domain, allowing for inexpensive filtering and manipulation of any guest calls * excellent performance Specifically, PV Calls for networking offer these advantages: * guest networking works out of the box with VPNs, wireless networks and any other complex configurations on the host * guest services listen on ports bound directly to the backend domain IP addresses * localhost becomes a secure namespace for inter-VMs communications ## Design ### Why Xen? PV Calls are part of an effort to create a secure runtime environment for containers (OCI
