Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Randy Dunlap [rdun...@infradead.org] wrote:

> > +struct ftw_setup_attr ftwattr;
> > +
> > +fd = open("/dev/ftw", O_RDWR);
> > +
> > +memset(, 0, sizeof(rxattr));
> 
> Is that supposed to be ftwattr (2x above)?

Yes. I agree with your other comments as well and will send a new version.

Thanks for the detailed review.

Sukadev

Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Randy Dunlap [rdun...@infradead.org] wrote:

> > +struct ftw_setup_attr ftwattr;
> > +
> > +fd = open("/dev/ftw", O_RDWR);
> > +
> > +memset(, 0, sizeof(rxattr));
> 
> Is that supposed to be ftwattr (2x above)?

Yes. I agree with your other comments as well and will send a new version.

Thanks for the detailed review.

Sukadev

Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Randy Dunlap]

On 01/16/2018 06:50 PM, Sukadev Bhattiprolu wrote:
> Document the usage of the VAS Fast thread-wakeup API and add an entry in
> MAINTAINERS file.
> 
> Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
> Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
> Mackerras.
> 
> Signed-off-by: Sukadev Bhattiprolu 
> ---
> 
> Changelog[v2]
>   - [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
> rather than two ioctls.
>   - [Michael Neuling] Drop "nx" from name "nx-ftw".
> 
> ---
>  Documentation/powerpc/ftw-api.txt | 283 
> ++
>  MAINTAINERS   |   8 ++
>  2 files changed, 291 insertions(+)
>  create mode 100644 Documentation/powerpc/ftw-api.txt
> 
> diff --git a/Documentation/powerpc/ftw-api.txt 
> b/Documentation/powerpc/ftw-api.txt
> new file mode 100644
> index 000..a107628
> --- /dev/null
> +++ b/Documentation/powerpc/ftw-api.txt
> @@ -0,0 +1,283 @@
> +Virtual Accelerator Switchboard and Fast Thread-Wakeup API
> +
...
> +
> +Application access to the FTW mechanism is provided through the FTW
> +device node (/dev/ftw) implemented by the FTW device driver.
> +
> +A multi-threaded software processes that intends to use the FTW

 process

> +mechanism must first setup a channel (consisting of a pair of VAS
> +windows) for the waiting and waking threads to communicate. The
> +channel is set up by opening the FTW device and issuing the FTW_SETUP
> +ioctl. Upon successful return from the ioctl, the waiting side of
> +channel is complete and a thread can issue the "Wait" instruction
> +to wait for an event.
> +
> +After the successful return from the FTW_SETUP ioctl, the waking
> +thread must use mmap() system call on the same file descriptor and
> +obtain a virtual address known as the "paste address".
> +
> +Once the mmap() call succeeds the setup of "waking" side of the channel
> +is complete. To wake up a waiting thread, the waking thread should use
> +the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
> +paste-address.
> +
> +The wait and wake up operations can be repeated as long as the paste
> +address and the FTW file descriptor are valid (i.e until munmap() of
> +the paste address or a close() of the FTW fd).
> +
> +1. FTW Device Node
> +
> +There is one /dev/ftw node in the system and it provides access to the
> +VAS/FTW functionality.
> +
> +The only valid operations (system calls) on the FTW node are:
> +
> +- open() the device for read and write.
> +
> +- issue the FTW_SETUP ioctl to set up a channel.
> +
> +- mmap() the file descriptor
> +
> +- close the device node.
> +
> +Other file operations on the FTW node are undefined.
> +
> +Note that the COPY and PASTE operations go directly to the hardware
> +and do not involve system calls or go through the FTW device.
> +
> +Although a system may have several instances of the VAS in the system
> +(typically, one per P9 chip) there is just one FTW device node in
> +the system.
> +
> +When the FTW device node is opened, the kernel assigns a suitable
> +instance of VAS to the process. Kernel will make a best-effort attempt
> +to assign an optimal instance of VAS for the process - based on the CPU/
> +chip that the process is running on. In the initial release, the kernel
> +does not support migrating the VAS instance if the process migrates from
> +a CPU on one chip to a CPU on another chip.
> +
> +Applications may chose a specific instance of the VAS using the 'vas_id'

choose

> +field in the FTW_SETUP ioctl as detailed below.
> +
> +2. Open FTW node
> +
> +The device should be opened for read and write. No special privileges
> +are needed to open the device. The device may be opened multiple times.
> +
> +Each open() of the FTW device is associated with one channel of
> +communication. There is a system-wide limit (currently 64K windows per
> +chip and since some are reserved for hardware, there are about 32K
> +channels per chip). If no more channels are available, the open() system
> +call will fail.
> +
> +See open(2) system call man pages for other details such as return
> +values, error codes and restrictions.
> +
> +3. Setup a communication channel (FTW_SETUP ioctl)
> +
> +A process that intends to use the Fast Thread-wakeup mechanism must
> +first setup a channel by issuing the FTW_SETUP ioctl.
> +
> +#include 
> +
> +struct ftw_setup_attr ftwattr;
> +
> +rc = ioctl(fd, FTW_SETUP, );
> +
> +The attributes of ftwattr are as follows:
> +
> +struct ftw_setup_attr {
> +int16_t   version;
> +int16_t   vas_id;
> +

Re: [PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Randy Dunlap]

On 01/16/2018 06:50 PM, Sukadev Bhattiprolu wrote:
> Document the usage of the VAS Fast thread-wakeup API and add an entry in
> MAINTAINERS file.
> 
> Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
> Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
> Mackerras.
> 
> Signed-off-by: Sukadev Bhattiprolu 
> ---
> 
> Changelog[v2]
>   - [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
> rather than two ioctls.
>   - [Michael Neuling] Drop "nx" from name "nx-ftw".
> 
> ---
>  Documentation/powerpc/ftw-api.txt | 283 
> ++
>  MAINTAINERS   |   8 ++
>  2 files changed, 291 insertions(+)
>  create mode 100644 Documentation/powerpc/ftw-api.txt
> 
> diff --git a/Documentation/powerpc/ftw-api.txt 
> b/Documentation/powerpc/ftw-api.txt
> new file mode 100644
> index 000..a107628
> --- /dev/null
> +++ b/Documentation/powerpc/ftw-api.txt
> @@ -0,0 +1,283 @@
> +Virtual Accelerator Switchboard and Fast Thread-Wakeup API
> +
...
> +
> +Application access to the FTW mechanism is provided through the FTW
> +device node (/dev/ftw) implemented by the FTW device driver.
> +
> +A multi-threaded software processes that intends to use the FTW

 process

> +mechanism must first setup a channel (consisting of a pair of VAS
> +windows) for the waiting and waking threads to communicate. The
> +channel is set up by opening the FTW device and issuing the FTW_SETUP
> +ioctl. Upon successful return from the ioctl, the waiting side of
> +channel is complete and a thread can issue the "Wait" instruction
> +to wait for an event.
> +
> +After the successful return from the FTW_SETUP ioctl, the waking
> +thread must use mmap() system call on the same file descriptor and
> +obtain a virtual address known as the "paste address".
> +
> +Once the mmap() call succeeds the setup of "waking" side of the channel
> +is complete. To wake up a waiting thread, the waking thread should use
> +the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
> +paste-address.
> +
> +The wait and wake up operations can be repeated as long as the paste
> +address and the FTW file descriptor are valid (i.e until munmap() of
> +the paste address or a close() of the FTW fd).
> +
> +1. FTW Device Node
> +
> +There is one /dev/ftw node in the system and it provides access to the
> +VAS/FTW functionality.
> +
> +The only valid operations (system calls) on the FTW node are:
> +
> +- open() the device for read and write.
> +
> +- issue the FTW_SETUP ioctl to set up a channel.
> +
> +- mmap() the file descriptor
> +
> +- close the device node.
> +
> +Other file operations on the FTW node are undefined.
> +
> +Note that the COPY and PASTE operations go directly to the hardware
> +and do not involve system calls or go through the FTW device.
> +
> +Although a system may have several instances of the VAS in the system
> +(typically, one per P9 chip) there is just one FTW device node in
> +the system.
> +
> +When the FTW device node is opened, the kernel assigns a suitable
> +instance of VAS to the process. Kernel will make a best-effort attempt
> +to assign an optimal instance of VAS for the process - based on the CPU/
> +chip that the process is running on. In the initial release, the kernel
> +does not support migrating the VAS instance if the process migrates from
> +a CPU on one chip to a CPU on another chip.
> +
> +Applications may chose a specific instance of the VAS using the 'vas_id'

choose

> +field in the FTW_SETUP ioctl as detailed below.
> +
> +2. Open FTW node
> +
> +The device should be opened for read and write. No special privileges
> +are needed to open the device. The device may be opened multiple times.
> +
> +Each open() of the FTW device is associated with one channel of
> +communication. There is a system-wide limit (currently 64K windows per
> +chip and since some are reserved for hardware, there are about 32K
> +channels per chip). If no more channels are available, the open() system
> +call will fail.
> +
> +See open(2) system call man pages for other details such as return
> +values, error codes and restrictions.
> +
> +3. Setup a communication channel (FTW_SETUP ioctl)
> +
> +A process that intends to use the Fast Thread-wakeup mechanism must
> +first setup a channel by issuing the FTW_SETUP ioctl.
> +
> +#include 
> +
> +struct ftw_setup_attr ftwattr;
> +
> +rc = ioctl(fd, FTW_SETUP, );
> +
> +The attributes of ftwattr are as follows:
> +
> +struct ftw_setup_attr {
> +int16_t   version;
> +int16_t   vas_id;
> +uint32_t  reserved;
> +
> +  

[PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Document the usage of the VAS Fast thread-wakeup API and add an entry in
MAINTAINERS file.

Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
Mackerras.

Signed-off-by: Sukadev Bhattiprolu 
---

Changelog[v2]
- [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
  rather than two ioctls.
- [Michael Neuling] Drop "nx" from name "nx-ftw".

---
 Documentation/powerpc/ftw-api.txt | 283 ++
 MAINTAINERS   |   8 ++
 2 files changed, 291 insertions(+)
 create mode 100644 Documentation/powerpc/ftw-api.txt

diff --git a/Documentation/powerpc/ftw-api.txt 
b/Documentation/powerpc/ftw-api.txt
new file mode 100644
index 000..a107628
--- /dev/null
+++ b/Documentation/powerpc/ftw-api.txt
@@ -0,0 +1,283 @@
+Virtual Accelerator Switchboard and Fast Thread-Wakeup API
+
+Power9 processor supports a hardware subystem known as the Virtual
+Accelerator Switchboard (VAS) which allows two entities in the Power9
+system to efficiently exchange messages. Messages must be formatted as
+Coprocessor Request Blocks (CRB) and be submitted using the COPY/PASTE
+instructions (new in Power9).
+
+Usage of VAS depends on the entities exchanging the messages and
+currently two usages have been identified.
+
+First usage of VAS, referred to as VAS/NX involves a software thread
+submitting data compression requests to a co-processor (hardware/nest
+accelerator) aka NX engine. This usage is not yet available to user
+applications.
+
+Alternatively, VAS can be used by two software threads to efficiently
+exchange messages. Initially, this mechanism is intended to wake up a
+waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
+describes the user API for this VAS/FTW mechanism.
+
+Application access to the FTW mechanism is provided through the FTW
+device node (/dev/ftw) implemented by the FTW device driver.
+
+A multi-threaded software processes that intends to use the FTW
+mechanism must first setup a channel (consisting of a pair of VAS
+windows) for the waiting and waking threads to communicate. The
+channel is set up by opening the FTW device and issuing the FTW_SETUP
+ioctl. Upon successful return from the ioctl, the waiting side of
+channel is complete and a thread can issue the "Wait" instruction
+to wait for an event.
+
+After the successful return from the FTW_SETUP ioctl, the waking
+thread must use mmap() system call on the same file descriptor and
+obtain a virtual address known as the "paste address".
+
+Once the mmap() call succeeds the setup of "waking" side of the channel
+is complete. To wake up a waiting thread, the waking thread should use
+the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
+paste-address.
+
+The wait and wake up operations can be repeated as long as the paste
+address and the FTW file descriptor are valid (i.e until munmap() of
+the paste address or a close() of the FTW fd).
+
+1. FTW Device Node
+
+There is one /dev/ftw node in the system and it provides access to the
+VAS/FTW functionality.
+
+The only valid operations (system calls) on the FTW node are:
+
+- open() the device for read and write.
+
+- issue the FTW_SETUP ioctl to set up a channel.
+
+- mmap() the file descriptor
+
+- close the device node.
+
+Other file operations on the FTW node are undefined.
+
+Note that the COPY and PASTE operations go directly to the hardware
+and do not involve system calls or go through the FTW device.
+
+Although a system may have several instances of the VAS in the system
+(typically, one per P9 chip) there is just one FTW device node in
+the system.
+
+When the FTW device node is opened, the kernel assigns a suitable
+instance of VAS to the process. Kernel will make a best-effort attempt
+to assign an optimal instance of VAS for the process - based on the CPU/
+chip that the process is running on. In the initial release, the kernel
+does not support migrating the VAS instance if the process migrates from
+a CPU on one chip to a CPU on another chip.
+
+Applications may chose a specific instance of the VAS using the 'vas_id'
+field in the FTW_SETUP ioctl as detailed below.
+
+2. Open FTW node
+
+The device should be opened for read and write. No special privileges
+are needed to open the device. The device may be opened multiple times.
+
+Each open() of the FTW device is associated with one channel of
+communication. There is a system-wide limit (currently 64K windows per
+chip and since some are reserved for hardware, there are about 32K
+channels per chip). If no more channels are available, the open() system

[PATCH 5/5] powerpc/ftw: Document FTW API/usage

[Sukadev Bhattiprolu]

Document the usage of the VAS Fast thread-wakeup API and add an entry in
MAINTAINERS file.

Thanks for input/comments from Benjamin Herrenschmidt, Michael Neuling,
Michael Ellerman, Robert Blackmore, Ian Munsie, Haren Myneni and Paul
Mackerras.

Signed-off-by: Sukadev Bhattiprolu 
---

Changelog[v2]
- [Michael Neuling] Update API to use a single, VAS_FTW_SEUTP ioctl
  rather than two ioctls.
- [Michael Neuling] Drop "nx" from name "nx-ftw".

---
 Documentation/powerpc/ftw-api.txt | 283 ++
 MAINTAINERS   |   8 ++
 2 files changed, 291 insertions(+)
 create mode 100644 Documentation/powerpc/ftw-api.txt

diff --git a/Documentation/powerpc/ftw-api.txt 
b/Documentation/powerpc/ftw-api.txt
new file mode 100644
index 000..a107628
--- /dev/null
+++ b/Documentation/powerpc/ftw-api.txt
@@ -0,0 +1,283 @@
+Virtual Accelerator Switchboard and Fast Thread-Wakeup API
+
+Power9 processor supports a hardware subystem known as the Virtual
+Accelerator Switchboard (VAS) which allows two entities in the Power9
+system to efficiently exchange messages. Messages must be formatted as
+Coprocessor Request Blocks (CRB) and be submitted using the COPY/PASTE
+instructions (new in Power9).
+
+Usage of VAS depends on the entities exchanging the messages and
+currently two usages have been identified.
+
+First usage of VAS, referred to as VAS/NX involves a software thread
+submitting data compression requests to a co-processor (hardware/nest
+accelerator) aka NX engine. This usage is not yet available to user
+applications.
+
+Alternatively, VAS can be used by two software threads to efficiently
+exchange messages. Initially, this mechanism is intended to wake up a
+waiting thread quickly - i.e "fast thread wake-up (FTW)". This document
+describes the user API for this VAS/FTW mechanism.
+
+Application access to the FTW mechanism is provided through the FTW
+device node (/dev/ftw) implemented by the FTW device driver.
+
+A multi-threaded software processes that intends to use the FTW
+mechanism must first setup a channel (consisting of a pair of VAS
+windows) for the waiting and waking threads to communicate. The
+channel is set up by opening the FTW device and issuing the FTW_SETUP
+ioctl. Upon successful return from the ioctl, the waiting side of
+channel is complete and a thread can issue the "Wait" instruction
+to wait for an event.
+
+After the successful return from the FTW_SETUP ioctl, the waking
+thread must use mmap() system call on the same file descriptor and
+obtain a virtual address known as the "paste address".
+
+Once the mmap() call succeeds the setup of "waking" side of the channel
+is complete. To wake up a waiting thread, the waking thread should use
+the "COPY" and "PASTE" instructions to write a zero-filled CRB to the
+paste-address.
+
+The wait and wake up operations can be repeated as long as the paste
+address and the FTW file descriptor are valid (i.e until munmap() of
+the paste address or a close() of the FTW fd).
+
+1. FTW Device Node
+
+There is one /dev/ftw node in the system and it provides access to the
+VAS/FTW functionality.
+
+The only valid operations (system calls) on the FTW node are:
+
+- open() the device for read and write.
+
+- issue the FTW_SETUP ioctl to set up a channel.
+
+- mmap() the file descriptor
+
+- close the device node.
+
+Other file operations on the FTW node are undefined.
+
+Note that the COPY and PASTE operations go directly to the hardware
+and do not involve system calls or go through the FTW device.
+
+Although a system may have several instances of the VAS in the system
+(typically, one per P9 chip) there is just one FTW device node in
+the system.
+
+When the FTW device node is opened, the kernel assigns a suitable
+instance of VAS to the process. Kernel will make a best-effort attempt
+to assign an optimal instance of VAS for the process - based on the CPU/
+chip that the process is running on. In the initial release, the kernel
+does not support migrating the VAS instance if the process migrates from
+a CPU on one chip to a CPU on another chip.
+
+Applications may chose a specific instance of the VAS using the 'vas_id'
+field in the FTW_SETUP ioctl as detailed below.
+
+2. Open FTW node
+
+The device should be opened for read and write. No special privileges
+are needed to open the device. The device may be opened multiple times.
+
+Each open() of the FTW device is associated with one channel of
+communication. There is a system-wide limit (currently 64K windows per
+chip and since some are reserved for hardware, there are about 32K
+channels per chip). If no more channels are available, the open() system
+call will fail.
+
+