From: Jan Kiszka <jan.kis...@siemens.com> This is a rewrite, pointing to the spec document for details on the device, describing the new configuration format. The demo section is left blank until the related ivshmem-demos are updated.
Signed-off-by: Jan Kiszka <jan.kis...@siemens.com> --- Documentation/inter-cell-communication.md | 79 ++++++++++++++++++++++++++++++ Documentation/inter-cell-communication.txt | 76 ---------------------------- 2 files changed, 79 insertions(+), 76 deletions(-) create mode 100644 Documentation/inter-cell-communication.md delete mode 100644 Documentation/inter-cell-communication.txt diff --git a/Documentation/inter-cell-communication.md b/Documentation/inter-cell-communication.md new file mode 100644 index 00000000..02136de2 --- /dev/null +++ b/Documentation/inter-cell-communication.md @@ -0,0 +1,79 @@ +Inter-Cell communication of the Jailhouse Hypervisor +==================================================== + +The hypervisor isolates cells but sometimes there is a need to exchange data +between cells. For that purpose Jailhouse provides shared memory and signaling +between cells via virtual ivshmem PCI devices. This device type is specified +in [1]. + + +Adding Inter-cell communication +------------------------------- + +In order to set up a communication channel between two or more cells you first +have to add memory regions to both cells. Each cell needs the follow regions: + + - read-only region to hold state table, generally one page large + - one region that is read/writable for all peers + - one output region per peer that is only read-writeable for one of them + +With 2 peers connected, this means a consecutive series of 4 regions needs to +be created. When connecting 3 peers, 5 regions are needed, and so on. All +regions must also be consecutive in guest-physical memory because only the +address for the first region is communicated to the guest. + +The second, common read-write region is optional and can be zero-sized. Also +the output regions are optional. All output regions must have the same size. +Write permission to the first output region must only be granted to the cell +that has the ivshmem device ID 0, write permission to the second region must be +granted to ID 1, and so forth. + +Non-root cells sharing memory with the root cell need the memory flag +`JAILHOUSE_MEM_ROOTSHARED` on the region. + +To define the memory regions of an ivshmem-net device, the macro +`JAILHOUSE_SHMEM_NET_REGIONS(base_address, id)` is provided. It uses 1 MB of +memory at the specified base address and assigns access according to the +specified ID. Shared memory based network devices only connect 2 peers, thus +4 memory regions will be added. + +After creating the memory regions, also a PCI device needs to be added to each +connected cell. Set the device type to `JAILHOUSE_PCI_TYPE_IVSHMEM`. The `bdf` +field must specify an unused bus-device-function slot on a physical or virtual +PCI controller. All connected peers must use the same `bdf` value in order to +establish the link. They may use different `domain` values, though. + +Set the bar_mask to either `JAILHOUSE_IVSHMEM_BAR_MASK_MSIX` or +`JAILHOUSE_IVSHMEM_BAR_MASK_INTX`, depending on whether MSI-X is available or +not. When MSI-X is used, num_msix_vectors must be set according to the needs of +the shared memory protocol used on the link. For ivshmem networking, grant 2 +vectors. + +Further fields needed: + - `shmem_regions_start` - index of first shared memory region used by device + - `shmem_dev_id` - ID of the peer (0..`shmem_peers`-1) + - `shmem_peers` - maximum number of connected peers + - `shmem_protocol` - shared memory protocol used over the link + +Set `shmem_protocol` to JAILHOUSE_SHMEM_PROTO_VETH for ivshmem networking, use +`JAILHOUSE_SHMEM_PROTO_UNDEFINED` for custom protocols, or pick an ID from the +custom range defined in [1]. + +You may also need to set the `iommu` field to match the IOMMU unit that the +guest expects based on the `bdf` value. Try 1 if MSI-X interrupts do not make +it when using 0. + +For an example have a look at the cell configuration files `qemu-x86.c`, +`ivshmem-demo.c`, and `linux-x86-demo.c` in `configs/x86`. + + +Demo code +--------- + +TODO + + +References +---------- + +[1] Documentation/ivshmem-v2-specification.md diff --git a/Documentation/inter-cell-communication.txt b/Documentation/inter-cell-communication.txt deleted file mode 100644 index 475ca500..00000000 --- a/Documentation/inter-cell-communication.txt +++ /dev/null @@ -1,76 +0,0 @@ -Inter-Cell communication of the Jailhouse Hypervisor -==================================================== - -The hypervisor isolates cells but sometimes there is a need to exchange data -between cells. For that purpose Jailhouse provides shared memory and signaling -between cells. - -One channel is always between exactly two cells, there is no 1:n or n:m -communication. - -The interface used between the cell and the hypervisor ------------------------------------------------------- - -One end of such a communication channel is modeled as a PCI device that a cell -can discover on it's PCI bus. The device model used closely follows the -"ivshmem" device known from Qemu (see qemu docs/specs/ivshmem_device_spec.txt -and https://gitorious.org/nahanni/). -The device implemented by jailhouse supports MSI-X for signaling. While the -spec would allow for multiple interrupts, Jailhouse supports exactly one per -virtual device. - -The ivshmem device implemented by the jailhouse hypervisor is different to the -mentioned specification in several regards. One is that the location and the -size of the shared memory region itself are not encoded in a PCI BAR. Device -drivers get the relevant information by accessing custom PCI config space -registers. See hypervisor/pci_ivshmem.c IVSHMEM_CFG_SHMEM_* or the ivshmem demo -in the inmates directory. - -To facilitate state synchronisation between cells, two additional MMIO -registers are provided: - - Offset Size Access Reset Function - 16 4 read/write 0 Local state (LSTATE) - 20 4 read-only 0 Remote state (RSTATE) - -Local state: Current value is visible as RSTATE in the connected cell. -Writes trigger the doorbell as by writing to the Doorbell register. - -Remote state: Returns the current value of the LSTATE register in the -connected cell, zero if no peer is connected. - -Moreover, the PCI Class Code field of the Jailhouse ivshmem device differs from -the one used by the original device. The base class code (top byte) is 0xff. -The subclass code (middle byte) is tunable via the cell configuration to encode -the protocol that the connected cells are expected to implement over the shared -memory. The programming interface code (low byte) is tunable as well and is -supposed to encode possible revisions of that protocol. - -Adding Inter-cell communication to cells ----------------------------------------- - -In order to set up a communication channel between two cells you first have to -add a memory region to both cells. Add a read/write region with matching size -and physical address to both cells. Non-root cells sharing memory with the -root cell need the memory flag "JAILHOUSE_MEM_ROOTSHARED" on the region. -To allow cells to discover shared memory and send each other MSIs you also -need to add a virtual PCI device to both cells. The "type" should be set to -"JAILHOUSE_PCI_TYPE_IVSHMEM" and "shmem_region" should be set to the index -of the memory region. "num_msix_vectors" should be set to 1 and for your root -cell config you should make sure that "iommu" is set to the correct value, - try using the same value that works for the other pci devices. -The link between two such virtual PCI devices is established by using the same -"bdf". The size and location of the shared memory can be configured freely but -you have to make sure that the values match on both sides. The "shmem_protocol" -has to match as well. -For an example have a look at the cell configuration files of qemu and the -ivshmem-demo. - -Demo code ---------- - -You can go ahead and connect two non-root cells and run the ivshmem-demo. They -will send each other interrupts. -For the root cell you can find some test code in the following git repository: -https://github.com/henning-schild-work/ivshmem-guest-code -Check out the jailhouse branch and have a look at README.jailhouse. -- 2.16.4 -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to jailhouse-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/8c898c1b3c88f425a1c114a8fcf76d62f3971fd2.1578313099.git.jan.kiszka%40siemens.com.
