[PATCH 07/15] powerpc/powernv: Add support for the cxl kernel api on the real phb

2016-07-13 Thread Ian Munsie 
From: Ian Munsie 

This adds support for the peer model of the cxl kernel api to the
PowerNV PHB, in which physical function 0 represents the cxl function on
the card (an XSL in the case of the CX4), which other physical functions
will use for memory access and interrupt services. It is referred to as
the peer model as these functions are peers of one another, as opposed
to the Virtual PHB model which forms a hierarchy.

This patch exports APIs to enable the peer mode, check if a PCI device
is attached to a PHB in this mode, and to set and get the peer AFU for
this mode.

The cxl driver will enable this mode for supported cards by calling
pnv_cxl_enable_phb_kernel_api(). This will set a flag in the PHB to note
that this mode is enabled, and switch out it's controller_ops for the
cxl version.

The cxl version of the controller_ops struct implements it's own
versions of the enable_device_hook and release_device to handle
refcounting on the peer AFU and to allocate a default context for the
device.

Once enabled, the cxl kernel API may not be disabled on a PHB. Currently
there is no safe way to disable cxl mode short of a reboot, so until
that changes there is no reason to support the disable path.

Signed-off-by: Ian Munsie 
Reviewed-by: Andrew Donnellan 

---

V1->V2:
- Add an explanation of the peer model to the commit message,
  and a comment above the pnv_cxl_enable_device_hook function.
V2->V3
Addressed comments from Andrew Donnellan:
- Fix typo in comment
- Changed two exported symbols to EXPORT_SYMBOL_GPL
- Undid change to remove static from pnv_pci_release_device and
  pci_controller_ops and declare them in the header, both of
  which were left over from an earlier cut.
---
 arch/powerpc/include/asm/pnv-pci.h|   7 ++
 arch/powerpc/platforms/powernv/pci-cxl.c  | 120 ++
 arch/powerpc/platforms/powernv/pci-ioda.c |  18 -
 arch/powerpc/platforms/powernv/pci.h  |  14 
 4 files changed, 158 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/pnv-pci.h 
b/arch/powerpc/include/asm/pnv-pci.h
index 791db1b..c47097f 100644
--- a/arch/powerpc/include/asm/pnv-pci.h
+++ b/arch/powerpc/include/asm/pnv-pci.h
@@ -38,6 +38,13 @@ int pnv_cxl_alloc_hwirq_ranges(struct cxl_irq_ranges *irqs,
   struct pci_dev *dev, int num);
 void pnv_cxl_release_hwirq_ranges(struct cxl_irq_ranges *irqs,
  struct pci_dev *dev);
+
+/* Support for the cxl kernel api on the real PHB (instead of vPHB) */
+int pnv_cxl_enable_phb_kernel_api(struct pci_controller *hose, bool enable);
+bool pnv_pci_on_cxl_phb(struct pci_dev *dev);
+struct cxl_afu *pnv_cxl_phb_to_afu(struct pci_controller *hose);
+void pnv_cxl_phb_set_peer_afu(struct pci_dev *dev, struct cxl_afu *afu);
+
 #endif
 
 #endif
diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c 
b/arch/powerpc/platforms/powernv/pci-cxl.c
index e0eeb00..831bbfb 100644
--- a/arch/powerpc/platforms/powernv/pci-cxl.c
+++ b/arch/powerpc/platforms/powernv/pci-cxl.c
@@ -7,8 +7,11 @@
  * 2 of the License, or (at your option) any later version.
  */
 
+#include 
+#include 
 #include 
 #include 
+#include 
 
 #include "pci.h"
 
@@ -161,3 +164,120 @@ int pnv_cxl_ioda_msi_setup(struct pci_dev *dev, unsigned 
int hwirq,
return 0;
 }
 EXPORT_SYMBOL(pnv_cxl_ioda_msi_setup);
+
+/*
+ * Sets flags and switches the controller ops to enable the cxl kernel api.
+ * Originally the cxl kernel API operated on a virtual PHB, but certain cards
+ * such as the Mellanox CX4 use a peer model instead and for these cards the
+ * cxl kernel api will operate on the real PHB.
+ */
+int pnv_cxl_enable_phb_kernel_api(struct pci_controller *hose, bool enable)
+{
+   struct pnv_phb *phb = hose->private_data;
+   struct module *cxl_module;
+
+   if (!enable) {
+   /*
+* Once cxl mode is enabled on the PHB, there is currently no
+* known safe method to disable it again, and trying risks a
+* checkstop. If we can find a way to safely disable cxl mode
+* in the future we can revisit this, but for now the only sane
+* thing to do is to refuse to disable cxl mode:
+*/
+   return -EPERM;
+   }
+
+   /*
+* Hold a reference to the cxl module since several PHB operations now
+* depend on it, and it would be insane to allow it to be removed so
+* long as we are in this mode (and since we can't safely disable this
+* mode once enabled...).
+*/
+   mutex_lock(_mutex);
+   cxl_module = find_module("cxl");
+   if (cxl_module)
+   __module_get(cxl_module);
+   mutex_unlock(_mutex);
+   if (!cxl_module)
+   return -ENODEV;
+
+   phb->flags |= PNV_PHB_FLAG_CXL;
+

Re: [PATCH 07/15] powerpc/powernv: Add support for the cxl kernel api on the real phb

2016-07-13 Thread Ian Munsie 
Excerpts from andrew.donnellan's message of 2016-07-12 20:39:13 +1000:
> Some comments below - with those addressed:
> 
> Reviewed-by: Andrew Donnellan 

Thanks for the review :)

> > V1->V2:
> > - Add an explanation of the peer model to the commit message,
> >   and a comment above the pnv_cxl_enable_device_hook function.
> 
> The comments are good!

Thanks :)

> > +/*
> > + * Sets flags and switches the controller ops to enable the cxl kernel api.
> > + * Original the cxl kernel API operated on a virtual PHB, but certain cards
> 
> Originally

Will fix.

> > +EXPORT_SYMBOL(pnv_cxl_enable_phb_kernel_api);
...
> > +EXPORT_SYMBOL(pnv_pci_on_cxl_phb);
> 
> Should these two exports be _GPL as well?

Yep, they should be - will fix in V3.

Looks like we have several more in this file that should be _GPL as
well, but since they weren't introduced in this series (just moved from
another file) I might send a separate patch for those after this has
been merged.

> > -static void pnv_pci_release_device(struct pci_dev *pdev)
> > +void pnv_pci_release_device(struct pci_dev *pdev)
> 
> Why is this being unstatic-ified? I don't see us introducing any new 
> uses of it.

Left over from an earlier cut - will undo this for V3.

> > -static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
> > +const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
> 
> It looks like we don't currently use this - is this being unstatic-ised 
> in view of allowing a switch back to regular mode in future?

Oops, again left over from an earlier cut (for that reason). Will undo
this for V3 - if we allow switching back in the future we can remove the
static then.

> > +extern void pnv_pci_release_device(struct pci_dev *pdev);

Will also remove this declaration of pnv_pci_release_device.

> > +extern const struct pci_controller_ops pnv_pci_ioda_controller_ops;
> 
> See applicable comments about static.

Will remove.

Cheers,
-Ian

___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [PATCH 07/15] powerpc/powernv: Add support for the cxl kernel api on the real phb

2016-07-12 Thread Andrew Donnellan 

(reviewed late in the evening, apologies if I've missed something)

On 11/07/16 21:50, Ian Munsie wrote:

From: Ian Munsie 

This adds support for the peer model of the cxl kernel api to the
PowerNV PHB, in which physical function 0 represents the cxl function on
the card (an XSL in the case of the CX4), which other physical functions
will use for memory access and interrupt services. It is referred to as
the peer model as these functions are peers of one another, as opposed
to the Virtual PHB model which forms a hierarchy.

This patch exports APIs to enable the peer mode, check if a PCI device
is attached to a PHB in this mode, and to set and get the peer AFU for
this mode.

The cxl driver will enable this mode for supported cards by calling
pnv_cxl_enable_phb_kernel_api(). This will set a flag in the PHB to note
that this mode is enabled, and switch out it's controller_ops for the
cxl version.

The cxl version of the controller_ops struct implements it's own
versions of the enable_device_hook and release_device to handle
refcounting on the peer AFU and to allocate a default context for the
device.

Once enabled, the cxl kernel API may not be disabled on a PHB. Currently
there is no safe way to disable cxl mode short of a reboot, so until
that changes there is no reason to support the disable path.

Signed-off-by: Ian Munsie 


Some comments below - with those addressed:

Reviewed-by: Andrew Donnellan 



---

V1->V2:
- Add an explanation of the peer model to the commit message,
  and a comment above the pnv_cxl_enable_device_hook function.


The comments are good!


+/*
+ * Sets flags and switches the controller ops to enable the cxl kernel api.
+ * Original the cxl kernel API operated on a virtual PHB, but certain cards


Originally


+ * such as the Mellanox CX4 use a peer model instead and for these cards the
+ * cxl kernel api will operate on the real PHB.
+ */
+int pnv_cxl_enable_phb_kernel_api(struct pci_controller *hose, bool enable)
+{
+   struct pnv_phb *phb = hose->private_data;
+   struct module *cxl_module;
+
+   if (!enable) {
+   /*
+* Once cxl mode is enabled on the PHB, there is currently no
+* known safe method to disable it again, and trying risks a
+* checkstop. If we can find a way to safely disable cxl mode
+* in the future we can revisit this, but for now the only sane
+* thing to do is to refuse to disable cxl mode:
+*/
+   return -EPERM;
+   }
+
+   /*
+* Hold a reference to the cxl module since several PHB operations now
+* depend on it, and it would be insane to allow it to be removed so
+* long as we are in this mode (and since we can't safely disable this
+* mode once enabled...).
+*/
+   mutex_lock(_mutex);
+   cxl_module = find_module("cxl");
+   if (cxl_module)
+   __module_get(cxl_module);
+   mutex_unlock(_mutex);
+   if (!cxl_module)
+   return -ENODEV;
+
+   phb->flags |= PNV_PHB_FLAG_CXL;
+   hose->controller_ops = pnv_cxl_cx4_ioda_controller_ops;
+
+   return 0;
+}
+EXPORT_SYMBOL(pnv_cxl_enable_phb_kernel_api);
+
+bool pnv_pci_on_cxl_phb(struct pci_dev *dev)
+{
+   struct pci_controller *hose = pci_bus_to_host(dev->bus);
+   struct pnv_phb *phb = hose->private_data;
+
+   return !!(phb->flags & PNV_PHB_FLAG_CXL);
+}
+EXPORT_SYMBOL(pnv_pci_on_cxl_phb);


Should these two exports be _GPL as well?


-static void pnv_pci_release_device(struct pci_dev *pdev)
+void pnv_pci_release_device(struct pci_dev *pdev)


Why is this being unstatic-ified? I don't see us introducing any new 
uses of it.



 {
struct pci_controller *hose = pci_bus_to_host(pdev->bus);
struct pnv_phb *phb = hose->private_data;
@@ -3423,7 +3423,7 @@ static void pnv_pci_ioda_shutdown(struct pci_controller 
*hose)
   OPAL_ASSERT_RESET);
 }

-static const struct pci_controller_ops pnv_pci_ioda_controller_ops = {
+const struct pci_controller_ops pnv_pci_ioda_controller_ops = {


It looks like we don't currently use this - is this being unstatic-ised 
in view of allowing a switch back to regular mode in future?



 extern struct pci_ops pnv_pci_ops;
@@ -218,6 +222,8 @@ extern int pnv_setup_msi_irqs(struct pci_dev *pdev, int 
nvec, int type);
 extern void pnv_teardown_msi_irqs(struct pci_dev *pdev);
 extern struct pnv_ioda_pe *pnv_ioda_get_pe(struct pci_dev *dev);
 extern void pnv_set_msi_irq_chip(struct pnv_phb *phb, unsigned int virq);
+extern bool pnv_pci_enable_device_hook(struct pci_dev *dev);
+extern void pnv_pci_release_device(struct pci_dev *pdev);

 extern void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
const char *fmt, ...);
@@ -238,4 +244,14 @@ extern long pnv_npu_unset_window(struct

[PATCH 07/15] powerpc/powernv: Add support for the cxl kernel api on the real phb

2016-07-11 Thread Ian Munsie 
From: Ian Munsie 

This adds support for the peer model of the cxl kernel api to the
PowerNV PHB, in which physical function 0 represents the cxl function on
the card (an XSL in the case of the CX4), which other physical functions
will use for memory access and interrupt services. It is referred to as
the peer model as these functions are peers of one another, as opposed
to the Virtual PHB model which forms a hierarchy.

This patch exports APIs to enable the peer mode, check if a PCI device
is attached to a PHB in this mode, and to set and get the peer AFU for
this mode.

The cxl driver will enable this mode for supported cards by calling
pnv_cxl_enable_phb_kernel_api(). This will set a flag in the PHB to note
that this mode is enabled, and switch out it's controller_ops for the
cxl version.

The cxl version of the controller_ops struct implements it's own
versions of the enable_device_hook and release_device to handle
refcounting on the peer AFU and to allocate a default context for the
device.

Once enabled, the cxl kernel API may not be disabled on a PHB. Currently
there is no safe way to disable cxl mode short of a reboot, so until
that changes there is no reason to support the disable path.

Signed-off-by: Ian Munsie 

---

V1->V2:
- Add an explanation of the peer model to the commit message,
  and a comment above the pnv_cxl_enable_device_hook function.
---
 arch/powerpc/include/asm/pnv-pci.h|   7 ++
 arch/powerpc/platforms/powernv/pci-cxl.c  | 120 ++
 arch/powerpc/platforms/powernv/pci-ioda.c |  22 +-
 arch/powerpc/platforms/powernv/pci.h  |  16 
 4 files changed, 162 insertions(+), 3 deletions(-)

diff --git a/arch/powerpc/include/asm/pnv-pci.h 
b/arch/powerpc/include/asm/pnv-pci.h
index 791db1b..c47097f 100644
--- a/arch/powerpc/include/asm/pnv-pci.h
+++ b/arch/powerpc/include/asm/pnv-pci.h
@@ -38,6 +38,13 @@ int pnv_cxl_alloc_hwirq_ranges(struct cxl_irq_ranges *irqs,
   struct pci_dev *dev, int num);
 void pnv_cxl_release_hwirq_ranges(struct cxl_irq_ranges *irqs,
  struct pci_dev *dev);
+
+/* Support for the cxl kernel api on the real PHB (instead of vPHB) */
+int pnv_cxl_enable_phb_kernel_api(struct pci_controller *hose, bool enable);
+bool pnv_pci_on_cxl_phb(struct pci_dev *dev);
+struct cxl_afu *pnv_cxl_phb_to_afu(struct pci_controller *hose);
+void pnv_cxl_phb_set_peer_afu(struct pci_dev *dev, struct cxl_afu *afu);
+
 #endif
 
 #endif
diff --git a/arch/powerpc/platforms/powernv/pci-cxl.c 
b/arch/powerpc/platforms/powernv/pci-cxl.c
index e0eeb00..3c4caf0 100644
--- a/arch/powerpc/platforms/powernv/pci-cxl.c
+++ b/arch/powerpc/platforms/powernv/pci-cxl.c
@@ -7,8 +7,11 @@
  * 2 of the License, or (at your option) any later version.
  */
 
+#include 
+#include 
 #include 
 #include 
+#include 
 
 #include "pci.h"
 
@@ -161,3 +164,120 @@ int pnv_cxl_ioda_msi_setup(struct pci_dev *dev, unsigned 
int hwirq,
return 0;
 }
 EXPORT_SYMBOL(pnv_cxl_ioda_msi_setup);
+
+/*
+ * Sets flags and switches the controller ops to enable the cxl kernel api.
+ * Original the cxl kernel API operated on a virtual PHB, but certain cards
+ * such as the Mellanox CX4 use a peer model instead and for these cards the
+ * cxl kernel api will operate on the real PHB.
+ */
+int pnv_cxl_enable_phb_kernel_api(struct pci_controller *hose, bool enable)
+{
+   struct pnv_phb *phb = hose->private_data;
+   struct module *cxl_module;
+
+   if (!enable) {
+   /*
+* Once cxl mode is enabled on the PHB, there is currently no
+* known safe method to disable it again, and trying risks a
+* checkstop. If we can find a way to safely disable cxl mode
+* in the future we can revisit this, but for now the only sane
+* thing to do is to refuse to disable cxl mode:
+*/
+   return -EPERM;
+   }
+
+   /*
+* Hold a reference to the cxl module since several PHB operations now
+* depend on it, and it would be insane to allow it to be removed so
+* long as we are in this mode (and since we can't safely disable this
+* mode once enabled...).
+*/
+   mutex_lock(_mutex);
+   cxl_module = find_module("cxl");
+   if (cxl_module)
+   __module_get(cxl_module);
+   mutex_unlock(_mutex);
+   if (!cxl_module)
+   return -ENODEV;
+
+   phb->flags |= PNV_PHB_FLAG_CXL;
+   hose->controller_ops = pnv_cxl_cx4_ioda_controller_ops;
+
+   return 0;
+}
+EXPORT_SYMBOL(pnv_cxl_enable_phb_kernel_api);
+
+bool pnv_pci_on_cxl_phb(struct pci_dev *dev)
+{
+   struct pci_controller *hose = pci_bus_to_host(dev->bus);
+   struct pnv_phb *phb = hose->private_data;
+
+   return !!(phb->flags & PNV_PHB_FLAG_CXL);
+}
+EXPORT_SYMBOL(pnv_pci_on_cxl_phb);
+
+struct