Re: [PATCH v6 17/17] powerpc/vas: Document FTW API/usage

[Michael Neuling]

On Tue, 2017-08-08 at 16:07 -0700, Sukadev Bhattiprolu wrote:
> Document the usage of the VAS Fast thread-wakeup API.
> 
> Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
> Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni, Paul Mackerras.
> 
> Cc:Ian Munsie 
> Cc:Paul Mackerras 
> Signed-off-by: Sukadev Bhattiprolu 
> ---
>  Documentation/powerpc/ftw-api.txt | 373
> ++
>  1 file changed, 373 insertions(+)
>  create mode 100644 Documentation/powerpc/ftw-api.txt
> 
> diff --git a/Documentation/powerpc/ftw-api.txt b/Documentation/powerpc/ftw-
> api.txt
> new file mode 100644
> index 000..0b3f16f
> --- /dev/null
> +++ b/Documentation/powerpc/ftw-api.txt
> @@ -0,0 +1,373 @@
> +Virtual Accelerator Switchboard and Fast Thread-Wakeup API
> +
> +Power9 processor supports a hardware subystem known as the Virtual
> +Accelerator Switchboard (VAS) which allows two entities in the Power9
> +system to efficiently exchange messages. Messages must be formatted as
> +Coprocessor Reqeust Blocks (CRB) and be submitted using the COPY/PASTE
> +instructions (new in Power9).
> +
> +Usage of VAS depends on the entities exchanging the messages and
> +currently two usages have been identified.
> +
> +First usage of VAS, referred to as VAS/NX involves a software thread
> +submitting data compression requests to a co-processor (hardware/nest
> +accelerator) aka NX engine. The API for this usage is described in the
> +VAS/NX API document.
> +
> +Alternatively, VAS can be used by two software threads to efficiently
> +exchange messages. Initially, this mechanism is intended to wake up a
> +waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
> +describes the user API for this VAS/FTW mechanism.
> +
> +Application access to the FTW mechanism is provided through the NX-FTW
> +device node (/dev/crypto/nx-ftw) implemented by the VAS/FTW device
> +driver.

crypto?

> +
> +A software thread T1 that intends to wait for an event must first setup
> +a receive window, by opening the NX-FTW device and using the
> +VAS_RX_WIN_OPEN ioctl. Upon successful return from the VAS_RX_WIN_OPEN
> +ioctl, an rx_win_handle is returned.

I realise there is a window here as part of the hardware implementation, but the
users don't care about the window on the receive side. It's hidden from them. 
It's just an rx handle IMHO.

The sender certainly has a window that users care about since they have to mmap
it.

> +
> +A software thread T2 that intends to wake up T1 at some point, must first
> +set up a "send window" using the VAS_TX_WIN_OPEN ioctl and specify the
> +rx_win_handle obtained by T1. After a successful VAS_TX_WIN_OPEN ioctl
> the
> +send window of T2 is considered paired with the receive window of T1. The
> +thread T2 must then use mmap() to obtain a "paste address" for the send
> +window.


> +With this set up, thread T1 can wait for an event using the WAIT
> +instruction.
> +
> +Thread T2 can wake up T1 by using the "COPY/PASTE" instructions and
> +submitting an empty/NULL CRB to the send window's paste address. The
> +wait/wake up process can be repeated as long as the threads have the
> +send/receive windows open.



> +1. NX-FTW Device Node
> +
> +There is one /dev/crypto/nx-ftw node in the system and it provides
> +access to the VAS/FTW functionality.


> +The only valid operations on the NX-FTW node are:
> +
> +- open() the device for read and write.
> +
> +- issue either VAS_RX_WIN_OPEN or VAS_TX_WIN_OPEN ioctls to set up
> +  receive or send (only one of them per open).
> +
> +- if the open is associated with send window (i.e VAS_TX_WIN_OPEN
> +  ioctl was issued) mmap() the send window into the application's
> +  virtual address space. (i.e get a 'paste_address' for the send
> +  window).
> +
> +- close the device node.
> +
> +Other file operations on the NX-FTW node are undefined.
> +
> +Note tHAT the COPY and PASTE operations go directly to the hardware
> +and not go through the NX-FTW device.

I don't understand this statement

> +
> +Although a system may have several instances of the VAS in the system
> +(typically, one per P9 chip) there is just one NX-FTW device node in
> +the system.

> + When the NX-FTW device node is opened, the kernel assigns a suitable
> + instance of VAS to the process. Kernel will make a best-effort
> attempt
> + to assign an optimal instance of VAS for the process. In the initial
> +release, the kernel does not support migrating the VAS instance if the
> +process migrates from a processor on one chip to a processor on another
> +chip.

How is it "optimal"?

> +Applications may chose a 

[PATCH v6 17/17] powerpc/vas: Document FTW API/usage

[Sukadev Bhattiprolu]

Document the usage of the VAS Fast thread-wakeup API.

Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni, Paul Mackerras.

Cc:Ian Munsie 
Cc:Paul Mackerras 
Signed-off-by: Sukadev Bhattiprolu 
---
 Documentation/powerpc/ftw-api.txt | 373 ++
 1 file changed, 373 insertions(+)
 create mode 100644 Documentation/powerpc/ftw-api.txt

diff --git a/Documentation/powerpc/ftw-api.txt 
b/Documentation/powerpc/ftw-api.txt
new file mode 100644
index 000..0b3f16f
--- /dev/null
+++ b/Documentation/powerpc/ftw-api.txt
@@ -0,0 +1,373 @@
+Virtual Accelerator Switchboard and Fast Thread-Wakeup API
+
+Power9 processor supports a hardware subystem known as the Virtual
+Accelerator Switchboard (VAS) which allows two entities in the Power9
+system to efficiently exchange messages. Messages must be formatted as
+Coprocessor Reqeust Blocks (CRB) and be submitted using the COPY/PASTE
+instructions (new in Power9).
+
+Usage of VAS depends on the entities exchanging the messages and
+currently two usages have been identified.
+
+First usage of VAS, referred to as VAS/NX involves a software thread
+submitting data compression requests to a co-processor (hardware/nest
+accelerator) aka NX engine. The API for this usage is described in the
+VAS/NX API document.
+
+Alternatively, VAS can be used by two software threads to efficiently
+exchange messages. Initially, this mechanism is intended to wake up a
+waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
+describes the user API for this VAS/FTW mechanism.
+
+Application access to the FTW mechanism is provided through the NX-FTW
+device node (/dev/crypto/nx-ftw) implemented by the VAS/FTW device
+driver.
+
+A software thread T1 that intends to wait for an event must first setup
+a receive window, by opening the NX-FTW device and using the
+VAS_RX_WIN_OPEN ioctl. Upon successful return from the VAS_RX_WIN_OPEN
+ioctl, an rx_win_handle is returned.
+
+A software thread T2 that intends to wake up T1 at some point, must first
+set up a "send window" using the VAS_TX_WIN_OPEN ioctl and specify the
+rx_win_handle obtained by T1. After a successful VAS_TX_WIN_OPEN ioctl the
+send window of T2 is considered paired with the receive window of T1. The
+thread T2 must then use mmap() to obtain a "paste address" for the send
+window.
+
+With this set up, thread T1 can wait for an event using the WAIT
+instruction.
+
+Thread T2 can wake up T1 by using the "COPY/PASTE" instructions and
+submitting an empty/NULL CRB to the send window's paste address. The
+wait/wake up process can be repeated as long as the threads have the
+send/receive windows open.
+
+1. NX-FTW Device Node
+
+There is one /dev/crypto/nx-ftw node in the system and it provides
+access to the VAS/FTW functionality.
+
+The only valid operations on the NX-FTW node are:
+
+- open() the device for read and write.
+
+- issue either VAS_RX_WIN_OPEN or VAS_TX_WIN_OPEN ioctls to set up
+  receive or send (only one of them per open).
+
+- if the open is associated with send window (i.e VAS_TX_WIN_OPEN
+  ioctl was issued) mmap() the send window into the application's
+  virtual address space. (i.e get a 'paste_address' for the send
+  window).
+
+- close the device node.
+
+Other file operations on the NX-FTW node are undefined.
+
+Note tHAT the COPY and PASTE operations go directly to the hardware
+and not go through the NX-FTW device.
+
+Although a system may have several instances of the VAS in the system
+(typically, one per P9 chip) there is just one NX-FTW device node in
+the system.
+
+   When the NX-FTW device node is opened, the kernel assigns a suitable
+   instance of VAS to the process. Kernel will make a best-effort attempt
+   to assign an optimal instance of VAS for the process. In the initial
+release, the kernel does not support migrating the VAS instance if the
+process migrates from a processor on one chip to a processor on another
+chip.
+
+Applications may chose a specific instance of the VAS using the 'vas_id'
+field in the VAS_TX_WIN_OPEN and VAS_RX_WIN_OPEN ioctls as detailed below.
+
+2. Open NX-FTW node
+
+The device should be opened for read and write. No special privileges
+are needed to open the device. The device may be opened multiple times.
+
+Each open() of the NX-FTW device may be associated with either a send
+window or receive window but not both.
+
+See open(2) system call man pages for other details such as return
+values, error codes and restrictions.
+
+3. Setup Receive window (VAS_RX_WIN_OPEN ioctl)
+
+A