Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Sukadev Bhattiprolu [suka...@linux.vnet.ibm.com] wrote:
> Michael Neuling [mi...@neuling.org] wrote:
> > On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > > Define macros for the VAS hardware registers and bit-fields as well
> > > as couple of data structures needed by the VAS driver.
> > > 
> > > > Signed-off-by: Sukadev Bhattiprolu 
> > > ---
> > > Changelog[v3]
> > >   - Rename winctx->pid to winctx->pidr to reflect that its a value
> > >     from the PID register (SPRN_PID), not the linux process id.
> > >   - Make it easier to split header into kernel/user parts
> > >   - To keep user interface simple, use macros rather than enum for
> > >     the threshold-control modes.
> > >   - Add a pid field to struct vas_window - needed for user space
> > >     send windows.
> > > 
> > > Changelog[v2]
> > >   - Add an overview of VAS in vas-internal.h
> > >   - Get window context parameters from device tree and drop
> > >     unnecessary macros.
> > > ---
> > >  MAINTAINERS |   6 +
> > >  arch/powerpc/include/asm/vas.h  |  43 +
> > >  drivers/misc/vas/vas-internal.h | 392 
> > > 
> > 
> > This is going to have to go through gregkh/lkml if it's drivers/misc.  
> > you'll at
> > least need gregkh's ack/ok before mpe will take them (which is what we did 
> > for
> > CAPI).
> > 
> > We might want to keep this in arch/powerpc but I'm not sure.
> > 
> 
> We will have device nodes accessible to user space so put it here and can
> copy Gregkh next time. But let me know if we should move to arch/powerpc.
> 

Thinking about this some more, the VAS module itself does not provide
the device nodes. Rather, the drivers that use VAS, like NX-GZIP, will
provide them. So, I am moving the VAS code into arch/powerpc/kernel.

Please let me know of any comments/concerns.

Thanks
Suka

Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Michael Neuling [mi...@neuling.org] wrote:
> On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> > Define macros for the VAS hardware registers and bit-fields as well
> > as couple of data structures needed by the VAS driver.
> > 
> > > Signed-off-by: Sukadev Bhattiprolu 
> > ---
> > Changelog[v3]
> > - Rename winctx->pid to winctx->pidr to reflect that its a value
> >   from the PID register (SPRN_PID), not the linux process id.
> > - Make it easier to split header into kernel/user parts
> > - To keep user interface simple, use macros rather than enum for
> >   the threshold-control modes.
> > - Add a pid field to struct vas_window - needed for user space
> >   send windows.
> > 
> > Changelog[v2]
> > - Add an overview of VAS in vas-internal.h
> > - Get window context parameters from device tree and drop
> >   unnecessary macros.
> > ---
> >  MAINTAINERS |   6 +
> >  arch/powerpc/include/asm/vas.h  |  43 +
> >  drivers/misc/vas/vas-internal.h | 392 
> > 
> 
> This is going to have to go through gregkh/lkml if it's drivers/misc.  you'll 
> at
> least need gregkh's ack/ok before mpe will take them (which is what we did for
> CAPI).
> 
> We might want to keep this in arch/powerpc but I'm not sure.
> 

We will have device nodes accessible to user space so put it here and can
copy Gregkh next time. But let me know if we should move to arch/powerpc.

> >  3 files changed, 441 insertions(+)
> >  create mode 100644 arch/powerpc/include/asm/vas.h
> >  create mode 100644 drivers/misc/vas/vas-internal.h
> > 
> 
> > +
> > +/*
> > + * Overview of Virtual Accelerator Switchboard (VAS).
> > + *
> > + * VAS is a hardware "switchboard" that allows senders and receivers to
> > + * exchange messages with _minimal_ kernel involvment. The receivers are
> > + * typically NX coprocessor engines that perform compression or encryption
> > + * in hardware, but receivers can also be other software threads.
> > + *
> > + * Senders are user/kernel threads that submit compression/encryption or
> > + * other requests to the receivers. Senders must format their messages as
> > + * Coprocessor Request Blocks (CRB)s and submit them using the instructions
> > + * "copy" and "paste" which were introduced in Power9.
> > + *
> > + * A Power node can have (upto?) 8 Power chips. There is one instance of
> > + * VAS in each Power9 chip. Each instance of VAS has 64K windows or ports,
> > + * Senders and receivers must each connect to a separate window before they
> > + * can exchange messages through the switchboard.
> > + *
> > + * Each window is described by two types of window contexts:
> > + *
> > > + *   Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
> > + *
> > > + *   OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
> > + *
> > + * A window context can be viewed as a set of 64-bit registers. The 
> > settings
> > + * in these registers configure/control/determine the behavior of the VAS
> > + * hardware when messages are sent/received through the window. The 
> > registers
> > + * in the HVWC are configured by the kernel while the registers in the UWC 
> > can
> > + * be configured by the kernel or by the user space application that is 
> > using
> > + * the window.
> > + *
> > + * The HVWCs for all windows on a specific instance of VAS are in a 
> > contiguous
> > + * range of hardware addresses or Base address region (BAR) referred to as 
> > the
> > + * HVWC BAR for the instance. Similarly the UWCs for all windows on an 
> > instance
> > + * are referred to as the UWC BAR for the instance. The two BARs for each
> > + * instance are defined Power9 MMIO Ranges spreadsheet and available to the
> > + * kernel the device tree as follows:
> > + *
> > > + *   /proc/device-tree/xscom@.../vas@.../hvwc-bar-start
> > > + *   /proc/device-tree/xscom@.../vas@.../hvwc-bar-size
> > > + *   /proc/device-tree/xscom@.../vas@.../uwc-bar-start
> > + * /proc/device-tree/xscom@.../vas@.../uwc-bar-size
> 
> should these just be reg properties?

I guess they could. Will try that
> 
> > + *
> > + * The kernel maps these two hardware address regions into the kernel 
> > address
> > + * space (hvwc_map and uwc_map) and accesses the window contexts of a 
> > specific
> > + * window using:
> > + *
> > > + *    hvwc = hvwc_map + winid * VAS_HVWC_SIZE.
> > > + *    uwc = uwc_map + winid * VAS_UWC_SIZE.
> > + *
> > + * where winid is the window index (0..64K).
> > + *
> > + * Note that the window contexts are used to "configure" the windows. In
> > + * addition to this configuration address, each _send_ window also has a
> > + * unique hardware address, referred to as the "paste-address" to which the
> > + * sender must "paste" the message (CRB) they wish to submit. This hardware
> > + * paste address for window can be computed from the following nodes in the
> > + * device 

Re: [PATCH v3 01/10] VAS: Define macros, register fields and structures

[Michael Neuling]

On Thu, 2017-03-16 at 20:33 -0700, Sukadev Bhattiprolu wrote:
> Define macros for the VAS hardware registers and bit-fields as well
> as couple of data structures needed by the VAS driver.
> 
> > Signed-off-by: Sukadev Bhattiprolu 
> ---
> Changelog[v3]
>   - Rename winctx->pid to winctx->pidr to reflect that its a value
>     from the PID register (SPRN_PID), not the linux process id.
>   - Make it easier to split header into kernel/user parts
>   - To keep user interface simple, use macros rather than enum for
>     the threshold-control modes.
>   - Add a pid field to struct vas_window - needed for user space
>     send windows.
> 
> Changelog[v2]
>   - Add an overview of VAS in vas-internal.h
>   - Get window context parameters from device tree and drop
>     unnecessary macros.
> ---
>  MAINTAINERS |   6 +
>  arch/powerpc/include/asm/vas.h  |  43 +
>  drivers/misc/vas/vas-internal.h | 392 
> 

This is going to have to go through gregkh/lkml if it's drivers/misc.  you'll at
least need gregkh's ack/ok before mpe will take them (which is what we did for
CAPI).

We might want to keep this in arch/powerpc but I'm not sure.

>  3 files changed, 441 insertions(+)
>  create mode 100644 arch/powerpc/include/asm/vas.h
>  create mode 100644 drivers/misc/vas/vas-internal.h
> 

> +
> +/*
> + * Overview of Virtual Accelerator Switchboard (VAS).
> + *
> + * VAS is a hardware "switchboard" that allows senders and receivers to
> + * exchange messages with _minimal_ kernel involvment. The receivers are
> + * typically NX coprocessor engines that perform compression or encryption
> + * in hardware, but receivers can also be other software threads.
> + *
> + * Senders are user/kernel threads that submit compression/encryption or
> + * other requests to the receivers. Senders must format their messages as
> + * Coprocessor Request Blocks (CRB)s and submit them using the instructions
> + * "copy" and "paste" which were introduced in Power9.
> + *
> + * A Power node can have (upto?) 8 Power chips. There is one instance of
> + * VAS in each Power9 chip. Each instance of VAS has 64K windows or ports,
> + * Senders and receivers must each connect to a separate window before they
> + * can exchange messages through the switchboard.
> + *
> + * Each window is described by two types of window contexts:
> + *
> > + * Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
> + *
> > + * OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
> + *
> + * A window context can be viewed as a set of 64-bit registers. The settings
> + * in these registers configure/control/determine the behavior of the VAS
> + * hardware when messages are sent/received through the window. The registers
> + * in the HVWC are configured by the kernel while the registers in the UWC 
> can
> + * be configured by the kernel or by the user space application that is using
> + * the window.
> + *
> + * The HVWCs for all windows on a specific instance of VAS are in a 
> contiguous
> + * range of hardware addresses or Base address region (BAR) referred to as 
> the
> + * HVWC BAR for the instance. Similarly the UWCs for all windows on an 
> instance
> + * are referred to as the UWC BAR for the instance. The two BARs for each
> + * instance are defined Power9 MMIO Ranges spreadsheet and available to the
> + * kernel the device tree as follows:
> + *
> > + * /proc/device-tree/xscom@.../vas@.../hvwc-bar-start
> > + * /proc/device-tree/xscom@.../vas@.../hvwc-bar-size
> > + * /proc/device-tree/xscom@.../vas@.../uwc-bar-start
> + *   /proc/device-tree/xscom@.../vas@.../uwc-bar-size

should these just be reg properties?

> + *
> + * The kernel maps these two hardware address regions into the kernel address
> + * space (hvwc_map and uwc_map) and accesses the window contexts of a 
> specific
> + * window using:
> + *
> > + *  hvwc = hvwc_map + winid * VAS_HVWC_SIZE.
> > + *  uwc = uwc_map + winid * VAS_UWC_SIZE.
> + *
> + * where winid is the window index (0..64K).
> + *
> + * Note that the window contexts are used to "configure" the windows. In
> + * addition to this configuration address, each _send_ window also has a
> + * unique hardware address, referred to as the "paste-address" to which the
> + * sender must "paste" the message (CRB) they wish to submit. This hardware
> + * paste address for window can be computed from the following nodes in the
> + * device tree:
> + *
> > + * /proc/device-tree/xscom@.../vas@.../window-base
> + *   /proc/device-tree/xscom@.../vas@.../window-shift

Same here with reg properties.

 
> +struct vas_winctx {

> > +   int lpid;
> + int pidr;   /* value from SPRN_PID, not linux pid */

I'm surprised we have a copy of these here.  They should be accessed from the
context we are attaching to, rather than copied here... but I've not looked at
the rest of the code yet...

[PATCH v3 01/10] VAS: Define macros, register fields and structures

[Sukadev Bhattiprolu]

Define macros for the VAS hardware registers and bit-fields as well
as couple of data structures needed by the VAS driver.

Signed-off-by: Sukadev Bhattiprolu 
---
Changelog[v3]
- Rename winctx->pid to winctx->pidr to reflect that its a value
  from the PID register (SPRN_PID), not the linux process id.
- Make it easier to split header into kernel/user parts
- To keep user interface simple, use macros rather than enum for
  the threshold-control modes.
- Add a pid field to struct vas_window - needed for user space
  send windows.

Changelog[v2]
- Add an overview of VAS in vas-internal.h
- Get window context parameters from device tree and drop
  unnecessary macros.
---
 MAINTAINERS |   6 +
 arch/powerpc/include/asm/vas.h  |  43 +
 drivers/misc/vas/vas-internal.h | 392 
 3 files changed, 441 insertions(+)
 create mode 100644 arch/powerpc/include/asm/vas.h
 create mode 100644 drivers/misc/vas/vas-internal.h

diff --git a/MAINTAINERS b/MAINTAINERS
index c265a5f..2a910c9 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -13213,6 +13213,12 @@ S: Maintained
 F: Documentation/fb/uvesafb.txt
 F: drivers/video/fbdev/uvesafb.*
 
+VAS (IBM Virtual Accelerator Switchboard) DRIVER
+M: Sukadev Bhattiprolu 
+L: linuxppc-dev@lists.ozlabs.org
+S: Supported
+F: drivers/misc/vas/*
+
 VF610 NAND DRIVER
 M: Stefan Agner 
 L: linux-...@lists.infradead.org
diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h
new file mode 100644
index 000..6d35ce6
--- /dev/null
+++ b/arch/powerpc/include/asm/vas.h
@@ -0,0 +1,43 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef VAS_H
+#define VAS_H
+
+/*
+ * Threshold Control Mode: Have paste operation fail if the number of
+ * requests in receive FIFO exceeds a threshold.
+ *
+ * NOTE: No special error code yet if paste is rejected because of these
+ *  limits. So users can't distinguish between this and other errors.
+ */
+#define VAS_THRESH_DISABLED0
+#define VAS_THRESH_FIFO_GT_HALF_FULL   1
+#define VAS_THRESH_FIFO_GT_QTR_FULL2
+#define VAS_THRESH_FIFO_GT_EIGHTH_FULL 3
+
+#ifdef __KERNEL__
+
+#define VAS_RX_FIFO_SIZE_MAX   (8 << 20)   /* 8MB */
+/*
+ * Co-processor Engine type.
+ */
+enum vas_cop_type {
+   VAS_COP_TYPE_FAULT,
+   VAS_COP_TYPE_842,
+   VAS_COP_TYPE_842_HIPRI,
+   VAS_COP_TYPE_GZIP,
+   VAS_COP_TYPE_GZIP_HIPRI,
+   VAS_COP_TYPE_MAX,
+};
+
+
+#endif /* __KERNEL__ */
+
+#endif
diff --git a/drivers/misc/vas/vas-internal.h b/drivers/misc/vas/vas-internal.h
new file mode 100644
index 000..ce48f14
--- /dev/null
+++ b/drivers/misc/vas/vas-internal.h
@@ -0,0 +1,392 @@
+/*
+ * Copyright 2016 IBM Corp.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef VAS_INTERNAL_H
+#define VAS_INTERNAL_H
+#include 
+#include 
+#include 
+
+#ifdef CONFIG_PPC_4K_PAGES
+#  error "TODO: Compute RMA/Paste-address for 4K pages."
+#else
+#ifndef CONFIG_PPC_64K_PAGES
+#  error "Unexpected Page size."
+#endif
+#endif
+
+/*
+ * Overview of Virtual Accelerator Switchboard (VAS).
+ *
+ * VAS is a hardware "switchboard" that allows senders and receivers to
+ * exchange messages with _minimal_ kernel involvment. The receivers are
+ * typically NX coprocessor engines that perform compression or encryption
+ * in hardware, but receivers can also be other software threads.
+ *
+ * Senders are user/kernel threads that submit compression/encryption or
+ * other requests to the receivers. Senders must format their messages as
+ * Coprocessor Request Blocks (CRB)s and submit them using the instructions
+ * "copy" and "paste" which were introduced in Power9.
+ *
+ * A Power node can have (upto?) 8 Power chips. There is one instance of
+ * VAS in each Power9 chip. Each instance of VAS has 64K windows or ports,
+ * Senders and receivers must each connect to a separate window before they
+ * can exchange messages through the switchboard.
+ *
+ * Each window is described by two types of window contexts:
+ *
+ * Hypervisor Window Context (HVWC) of size VAS_HVWC_SIZE bytes
+ *
+ * OS/User Window Context (UWC) of size VAS_UWC_SIZE bytes.
+ *
+ * A window context can be viewed as a set of 64-bit registers. The settings
+ * in these registers configure/control/determine the behavior of the VAS
+ * hardware when