Re: [PATCH 1/2] vfio/fsl-mc: return -EFAULT if copy_to_user() fails

[Diana Craciun OSS]

On 11/2/2020 11:45 PM, Alex Williamson wrote:


Thanks, Dan.

Diana, can I get an ack for this?  Thanks,



Yes, sure, I apologize for not doing it earlier.

Thanks,
Diana



Alex

On Fri, 23 Oct 2020 14:34:50 +0300
Dan Carpenter  wrote:


The copy_to_user() function returns the number of bytes remaining to be
copied, but this code should return -EFAULT.

Fixes: df747bcd5b21 ("vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl 
call")
Signed-off-by: Dan Carpenter 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 8 ++--
  1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 0113a980f974..21f22e3da11f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -248,7 +248,9 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
info.size = vdev->regions[info.index].size;
info.flags = vdev->regions[info.index].flags;
  
-		return copy_to_user((void __user *)arg, , minsz);

+   if (copy_to_user((void __user *)arg, , minsz))
+   return -EFAULT;
+   return 0;
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -267,7 +269,9 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
info.flags = VFIO_IRQ_INFO_EVENTFD;
info.count = 1;
  
-		return copy_to_user((void __user *)arg, , minsz);

+   if (copy_to_user((void __user *)arg, , minsz))
+   return -EFAULT;
+   return 0;
}
case VFIO_DEVICE_SET_IRQS:
{

Re: drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:16:5: warning: no previous prototype for function 'vfio_fsl_mc_irqs_allocate'

[Diana Craciun OSS]

Hi,

The warning is fixed by this patch: 
https://www.spinics.net/lists/kvm/msg227575.html


The errors are caused by the fact that the vfio fsl-mc patches have a 
dependency on the fsl-mc bus series. For this particular error the 
missing patch is: bus/fsl-mc: Export IRQ pool handling functions to be 
used by VFIO. But the entire series should be merged before the 
vfio-fsl-mc patches.


I have verified compilation with clang on the master branch and there 
are no errors.


Diana


On 11/3/2020 9:54 AM, kernel test robot wrote:

Hi Diana,

FYI, the error/warning still remains.

tree:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
master
head:   b7cbaf59f62f8ab8f157698f9e31642bff525bd0
commit: cc0ee20bd96971c10eba9a83ecf1c0733078a083 vfio/fsl-mc: trigger an 
interrupt via eventfd
date:   3 weeks ago
config: arm64-randconfig-r026-20201101 (attached as .config)
compiler: clang version 12.0.0 (https://github.com/llvm/llvm-project 
235dfcf70abca65dba5d80f1a42d1485bab8980c)
reproduce (this is a W=1 build):
 wget 
https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O 
~/bin/make.cross
 chmod +x ~/bin/make.cross
 # install arm64 cross compiling tool for clang build
 # apt-get install binutils-aarch64-linux-gnu
 # 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=cc0ee20bd96971c10eba9a83ecf1c0733078a083
 git remote add linus 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
 git fetch --no-tags linus master
 git checkout cc0ee20bd96971c10eba9a83ecf1c0733078a083
 # save the attached .config to linux build tree
 COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=arm64

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot 

All warnings (new ones prefixed by >>):


drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:16:5: warning: no previous prototype for 
function 'vfio_fsl_mc_irqs_allocate' [-Wmissing-prototypes]

int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
^
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:16:1: note: declare 'static' if the 
function is not intended to be used outside of this translation unit
int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
^
static
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:121:8: error: implicit declaration 
of function 'fsl_mc_populate_irq_pool' [-Werror,-Wimplicit-function-declaration]
ret = fsl_mc_populate_irq_pool(mc_cont,
  ^
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:122:4: error: use of undeclared 
identifier 'FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS'
FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
^
1 warning and 2 errors generated.

vim +/vfio_fsl_mc_irqs_allocate +16 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

 15 
   > 16  int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
 17 {
 18 struct fsl_mc_device *mc_dev = vdev->mc_dev;
 19 struct vfio_fsl_mc_irq *mc_irq;
 20 int irq_count;
 21 int ret, i;
 22 
 23 /* Device does not support any interrupt */
 24 if (mc_dev->obj_desc.irq_count == 0)
 25 return 0;
 26 
 27 /* interrupts were already allocated for this device */
 28 if (vdev->mc_irqs)
 29 return 0;
 30 
 31 irq_count = mc_dev->obj_desc.irq_count;
 32 
 33 mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
 34 if (!mc_irq)
 35 return -ENOMEM;
 36 
 37 /* Allocate IRQs */
 38 ret = fsl_mc_allocate_irqs(mc_dev);
 39 if (ret) {
 40 kfree(mc_irq);
 41 return ret;
 42 }
 43 
 44 for (i = 0; i < irq_count; i++) {
 45 mc_irq[i].count = 1;
 46 mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
 47 }
 48 
 49 vdev->mc_irqs = mc_irq;
 50 
 51 return 0;
 52 }
 53 

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-...@lists.01.org

Re: [PATCH 2/2] vfio/fsl-mc: prevent underflow in vfio_fsl_mc_mmap()

[Diana Craciun OSS]

Acked-by: Diana Craciun 

On 10/23/2020 2:29 PM, Dan Carpenter wrote:

My static analsysis tool complains that the "index" can be negative.
There are some checks in do_mmap() which try to prevent underflows but
I don't know if they are sufficient for this situation.  Either way,
making "index" unsigned is harmless so let's do it just to be safe.

Fixes: 67247289688d ("vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO 
regions")
Signed-off-by: Dan Carpenter 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 21f22e3da11f..f27e25112c40 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -472,7 +472,7 @@ static int vfio_fsl_mc_mmap(void *device_data, struct 
vm_area_struct *vma)
  {
struct vfio_fsl_mc_device *vdev = device_data;
struct fsl_mc_device *mc_dev = vdev->mc_dev;
-   int index;
+   unsigned int index;
  
  	index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
  

Re: [PATCH 1/2] vfio/fsl-mc: return -EFAULT if copy_to_user() fails

[Diana Craciun OSS]

Acked-by: Diana Craciun 

On 10/23/2020 2:34 PM, Dan Carpenter wrote:

The copy_to_user() function returns the number of bytes remaining to be
copied, but this code should return -EFAULT.

Fixes: df747bcd5b21 ("vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl 
call")
Signed-off-by: Dan Carpenter 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 8 ++--
  1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 0113a980f974..21f22e3da11f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -248,7 +248,9 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
info.size = vdev->regions[info.index].size;
info.flags = vdev->regions[info.index].flags;
  
-		return copy_to_user((void __user *)arg, , minsz);

+   if (copy_to_user((void __user *)arg, , minsz))
+   return -EFAULT;
+   return 0;
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -267,7 +269,9 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
info.flags = VFIO_IRQ_INFO_EVENTFD;
info.count = 1;
  
-		return copy_to_user((void __user *)arg, , minsz);

+   if (copy_to_user((void __user *)arg, , minsz))
+   return -EFAULT;
+   return 0;
}
case VFIO_DEVICE_SET_IRQS:
{

[PATCH] vfio/fsl-mc: Make vfio_fsl_mc_irqs_allocate static

[Diana Craciun]

Fixed compiler warning:
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:16:5: warning: no previous
prototype for function 'vfio_fsl_mc_irqs_allocate' [-Wmissing-prototypes]
   ^
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:16:1: note: declare 'static'
if the function is not intended to be used outside of this translation unit
int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)

Reported-by: kernel test robot 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index c80dceb46f79..0d9f3002df7f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -13,7 +13,7 @@
 #include "linux/fsl/mc.h"
 #include "vfio_fsl_mc_private.h"
 
-int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
+static int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
 {
struct fsl_mc_device *mc_dev = vdev->mc_dev;
struct vfio_fsl_mc_irq *mc_irq;
-- 
2.17.1

[PATCH v2] vfio/fsl-mc: fix the return of the uninitialized variable ret

[Diana Craciun]

The vfio_fsl_mc_reflck_attach function may return, on success path,
an uninitialized variable. Fix the problem by initializing the return
variable to 0.

Addresses-Coverity: ("Uninitialized scalar variable")
Fixes: f2ba7e8c947b ("vfio/fsl-mc: Added lock support in preparation for 
interrupt handling")
Reported-by: Colin Ian King 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 80fc7f4ed343..0113a980f974 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -58,7 +58,7 @@ static struct vfio_fsl_mc_reflck 
*vfio_fsl_mc_reflck_alloc(void)
 
 static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
 {
-   int ret;
+   int ret = 0;
 
mutex_lock(_lock);
if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
-- 
2.17.1

Re: [PATCH] vfio/fsl-mc: fix the return of the uninitialized variable ret

[Diana Craciun OSS]

On 10/15/2020 9:52 PM, Alex Williamson wrote:

On Thu, 15 Oct 2020 13:22:26 +0100
Colin King  wrote:


From: Colin Ian King 

Currently the success path in function vfio_fsl_mc_reflck_attach is
returning an uninitialized value in variable ret. Fix this by setting
this to zero to indicate success.

Addresses-Coverity: ("Uninitialized scalar variable")
Fixes: f2ba7e8c947b ("vfio/fsl-mc: Added lock support in preparation for interrupt 
handling")
Signed-off-by: Colin Ian King 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 1 +
  1 file changed, 1 insertion(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 80fc7f4ed343..42a5decb78d1 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -84,6 +84,7 @@ static int vfio_fsl_mc_reflck_attach(struct 
vfio_fsl_mc_device *vdev)
vfio_fsl_mc_reflck_get(cont_vdev->reflck);
vdev->reflck = cont_vdev->reflck;
vfio_device_put(device);
+   ret = 0;
}
  
  unlock:


Looks correct to me, unless Diana would rather set the initial value to
zero instead.  Thanks,

Alex




I prefer to initialize the variable to 0 when it's defined. I'll send a 
patch for it.


Thanks,
Diana

[PATCH] vfio/fsl-mc: Fix the dead code in vfio_fsl_mc_set_irq_trigger

[Diana Craciun]

Static analysis discovered that some code in vfio_fsl_mc_set_irq_trigger
is dead code. Fixed the code by changing the conditions order.

Fixes: cc0ee20bd969 ("vfio/fsl-mc: trigger an interrupt via eventfd")
Reported-by: Colin Ian King 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 2ce2acad3461..c80dceb46f79 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -114,6 +114,9 @@ static int vfio_fsl_mc_set_irq_trigger(struct 
vfio_fsl_mc_device *vdev,
struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
 
+   if (!count && (flags & VFIO_IRQ_SET_DATA_NONE))
+   return vfio_set_trigger(vdev, index, -1);
+
if (start != 0 || count != 1)
return -EINVAL;
 
@@ -128,9 +131,6 @@ static int vfio_fsl_mc_set_irq_trigger(struct 
vfio_fsl_mc_device *vdev,
goto unlock;
mutex_unlock(>reflck->lock);
 
-   if (!count && (flags & VFIO_IRQ_SET_DATA_NONE))
-   return vfio_set_trigger(vdev, index, -1);
-
if (flags & VFIO_IRQ_SET_DATA_EVENTFD) {
s32 fd = *(s32 *)data;
 
-- 
2.17.1

Re: linux-next: build failure after merge of the vfio tree

[Diana Craciun OSS]

Thank you. The errors indicate that, most likely, the fsl-mc-bus patches 
from char-misc-next are missing at this point. I have added the vfio 
patches on top of linux-next (which already contains the fsl-mc-bus 
patches) and built x86_64 allmodconfig. There were no errors.


Thanks,
Diana

On 10/14/2020 1:16 AM, Stephen Rothwell wrote:

Hi Diana,

On Tue, 13 Oct 2020 18:56:07 +0300 Diana Craciun OSS 
 wrote:


Hi,

How does it fail? What's the error?


Sorry about that:

drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c: In function 
'vfio_fsl_mc_set_irq_trigger':
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:121:8: error: implicit declaration of 
function 'fsl_mc_populate_irq_pool' [-Werror=implicit-function-declaration]
   121 |  ret = fsl_mc_populate_irq_pool(mc_cont,
   |^~~~
drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c:122:4: error: 
'FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS' undeclared (first use in this function)
   122 |FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
   |^~
drivers/vfio/fsl-mc/vfio_fsl_mc.c: In function 'vfio_fsl_mc_release':
drivers/vfio/fsl-mc/vfio_fsl_mc.c:178:9: error: implicit declaration of 
function 'dprc_reset_container' [-Werror=implicit-function-declaration]
   178 |   ret = dprc_reset_container(mc_cont->mc_io, 0,
   | ^~~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:181:6: error: 
'DPRC_RESET_OPTION_NON_RECURSIVE' undeclared (first use in this function)
   181 |  DPRC_RESET_OPTION_NON_RECURSIVE);
   |  ^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:181:6: note: each undeclared identifier is 
reported only once for each function it appears in
drivers/vfio/fsl-mc/vfio_fsl_mc.c:191:3: error: implicit declaration of 
function 'fsl_mc_cleanup_irq_pool' [-Werror=implicit-function-declaration]
   191 |   fsl_mc_cleanup_irq_pool(mc_cont);
   |   ^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c: In function 'vfio_fsl_mc_ioctl':
drivers/vfio/fsl-mc/vfio_fsl_mc.c:316:9: error: 
'DPRC_RESET_OPTION_NON_RECURSIVE' undeclared (first use in this function)
   316 | DPRC_RESET_OPTION_NON_RECURSIVE);
   | ^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c: In function 'vfio_fsl_mc_mmap_mmio':
drivers/vfio/fsl-mc/vfio_fsl_mc.c:455:36: error: 'FSL_MC_REGION_CACHEABLE' 
undeclared (first use in this function)
   455 |  region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) &&
   |^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:456:22: error: 'FSL_MC_REGION_SHAREABLE' 
undeclared (first use in this function)
   456 |   (region.type & FSL_MC_REGION_SHAREABLE);
   |  ^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c: In function 'vfio_fsl_mc_bus_notifier':
drivers/vfio/fsl-mc/vfio_fsl_mc.c:522:9: error: 'struct fsl_mc_device' has no 
member named 'driver_override'
   522 |   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
   | ^~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:524:14: error: 'struct fsl_mc_device' has no 
member named 'driver_override'
   524 |   if (!mc_dev->driver_override)
   |  ^~
drivers/vfio/fsl-mc/vfio_fsl_mc.c: In function 'vfio_fsl_mc_init_device':
drivers/vfio/fsl-mc/vfio_fsl_mc.c:561:8: error: implicit declaration of 
function 'dprc_setup' [-Werror=implicit-function-declaration]
   561 |  ret = dprc_setup(mc_dev);
   |^~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:567:8: error: implicit declaration of 
function 'dprc_scan_container' [-Werror=implicit-function-declaration]
   567 |  ret = dprc_scan_container(mc_dev, false);
   |^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:576:2: error: implicit declaration of 
function 'dprc_remove_devices' [-Werror=implicit-function-declaration]
   576 |  dprc_remove_devices(mc_dev, NULL, 0);
   |  ^~~
drivers/vfio/fsl-mc/vfio_fsl_mc.c:577:2: error: implicit declaration of 
function 'dprc_cleanup' [-Werror=implicit-function-declaration]
   577 |  dprc_cleanup(mc_dev);
   |  ^~~~

Re: linux-next: build failure after merge of the vfio tree

[Diana Craciun OSS]

Hi,

How does it fail? What's the error?

Thanks,
Diana


On 10/13/2020 6:07 AM, Stephen Rothwell wrote:

Hi all,

After merging the vfio tree, today's linux-next build (x86_64
allmodconfig) failed like this:


Caused by commit

   cc0ee20bd969 ("vfio/fsl-mc: trigger an interrupt via eventfd")
   ac93ab2bf69a ("vfio/fsl-mc: Add support for device reset")

I have used the vfio tree from next-20201012 for today.

[PATCH v2] vfio/fsl-mc: Fixed vfio-fsl-mc driver compilation on 32 bit

[Diana Craciun]

The FSL_MC_BUS on which the VFIO-FSL-MC driver is dependent on
can be compiled on other architectures as well (not only ARM64)
including 32 bit architectures.
Include linux/io-64-nonatomic-hi-lo.h to make writeq/readq used
in the driver available on 32bit platforms.

Signed-off-by: Diana Craciun 
---
v1 --> v2
 - Added prefix to patch description

 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index d009f873578c..80fc7f4ed343 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -13,6 +13,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
-- 
2.17.1

[PATCH] Fixed vfio-fsl-mc driver compilation on 32 bit

[Diana Craciun]

The FSL_MC_BUS on which the VFIO-FSL-MC driver is dependent on
can be compiled on other architectures as well (not only ARM64)
including 32 bit architectures.
Include linux/io-64-nonatomic-hi-lo.h to make writeq/readq used
in the driver available on 32bit platforms.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index d009f873578c..80fc7f4ed343 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -13,6 +13,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
-- 
2.17.1

[PATCH v6 05/10] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions

[Diana Craciun]

Allow userspace to mmap device regions for direct access of
fsl-mc devices.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 68 ++-
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 05dace5ddc2c..55190a2730fb 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -30,11 +30,20 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
for (i = 0; i < count; i++) {
struct resource *res = _dev->regions[i];
+   int no_mmap = is_fsl_mc_bus_dprc(mc_dev);
 
vdev->regions[i].addr = res->start;
vdev->regions[i].size = resource_size(res);
-   vdev->regions[i].flags = 0;
vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   /*
+* Only regions addressed with PAGE granularity may be
+* MMAPed securely.
+*/
+   if (!no_mmap && !(vdev->regions[i].addr & ~PAGE_MASK) &&
+   !(vdev->regions[i].size & ~PAGE_MASK))
+   vdev->regions[i].flags |=
+   VFIO_REGION_INFO_FLAG_MMAP;
+
}
 
return 0;
@@ -163,9 +172,64 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const 
char __user *buf,
return -EINVAL;
 }
 
+static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,
+struct vm_area_struct *vma)
+{
+   u64 size = vma->vm_end - vma->vm_start;
+   u64 pgoff, base;
+   u8 region_cacheable;
+
+   pgoff = vma->vm_pgoff &
+   ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+   base = pgoff << PAGE_SHIFT;
+
+   if (region.size < PAGE_SIZE || base + size > region.size)
+   return -EINVAL;
+
+   region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) &&
+  (region.type & FSL_MC_REGION_SHAREABLE);
+   if (!region_cacheable)
+   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+   vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff;
+
+   return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+  size, vma->vm_page_prot);
+}
+
 static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int index;
+
+   index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
+
+   if (vma->vm_end < vma->vm_start)
+   return -EINVAL;
+   if (vma->vm_start & ~PAGE_MASK)
+   return -EINVAL;
+   if (vma->vm_end & ~PAGE_MASK)
+   return -EINVAL;
+   if (!(vma->vm_flags & VM_SHARED))
+   return -EINVAL;
+   if (index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ)
+   && (vma->vm_flags & VM_READ))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)
+   && (vma->vm_flags & VM_WRITE))
+   return -EINVAL;
+
+   vma->vm_private_data = mc_dev;
+
+   return vfio_fsl_mc_mmap_mmio(vdev->regions[index], vma);
 }
 
 static const struct vfio_device_ops vfio_fsl_mc_ops = {
-- 
2.17.1

[PATCH v6 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun]

The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 91 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  1 +
 2 files changed, 92 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index a7a483a1e90b..594760203268 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
 
 #include "vfio_fsl_mc_private.h"
 
+static struct fsl_mc_driver vfio_fsl_mc_driver;
+
 static int vfio_fsl_mc_open(void *device_data)
 {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,80 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
 };
 
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,
+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   if (!mc_dev->driver_override)
+   dev_warn(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s failed\n",
+dev_name(_cont->dev));
+   else
+   dev_info(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s\n",
+dev_name(_cont->dev));
+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "VFIO_FSL_MC: Object %s bound to driver 
%s while DPRC bound to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Failed to setup DPRC 
(%d)\n", ret);
+   goto out_nc_unreg;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Container scanning failed 
(%d)\n", ret);
+   goto out_dprc_cleanup;
+   }
+
+   return 0;
+
+out_dprc_cleanup:
+   dprc_remove_devices(mc_dev, NULL, 0);
+   dprc_cleanup(mc_dev);
+out_nc_unreg:
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   vdev->nb.notifier_call = NULL;
+
+   return ret;
+}
+
 static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
 {
struct iommu_group *group;
@@ -110,8 +186,15 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
dev_err(dev, "VFIO_FSL_MC: Failed to add to vfio group\n");
goto out_group_put;
}
+
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret)
+   goto out_group_dev;
+
return 0;
 
+out_group_dev:
+   vfio_del_group_dev(dev);
 out_group_put:
vfio_iommu_group_put(group, dev);
return ret;
@@ -126,6 +209,14 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   if (is_fsl_mc_bus_dprc(mc_dev)) {
+   dprc_remove_devices(mc_dev, NULL, 0);
+   dprc_cleanup(mc_dev);
+   }
+
+   if (vdev->nb.notifier_call)
+   bus_unregister_notifier(_mc_bus_type, >nb);
+
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
return 0;
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_m

[PATCH v6 06/10] vfio/fsl-mc: Added lock support in preparation for interrupt handling

[Diana Craciun]

Only the DPRC object allocates interrupts from the MSI
interrupt domain. The interrupts are managed by the DPRC in
a pool of interrupts. The access to this pool of interrupts
has to be protected with a lock.
This patch extends the current lock implementation to have a
lock per DPRC.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 92 +--
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  7 +-
 2 files changed, 90 insertions(+), 9 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 55190a2730fb..b52407c4e1ea 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,6 +17,78 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static DEFINE_MUTEX(reflck_lock);
+
+static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_get(>kref);
+}
+
+static void vfio_fsl_mc_reflck_release(struct kref *kref)
+{
+   struct vfio_fsl_mc_reflck *reflck = container_of(kref,
+ struct vfio_fsl_mc_reflck,
+ kref);
+
+   mutex_destroy(>lock);
+   kfree(reflck);
+   mutex_unlock(_lock);
+}
+
+static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_put_mutex(>kref, vfio_fsl_mc_reflck_release, _lock);
+}
+
+static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void)
+{
+   struct vfio_fsl_mc_reflck *reflck;
+
+   reflck = kzalloc(sizeof(*reflck), GFP_KERNEL);
+   if (!reflck)
+   return ERR_PTR(-ENOMEM);
+
+   kref_init(>kref);
+   mutex_init(>lock);
+
+   return reflck;
+}
+
+static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
+{
+   int ret;
+
+   mutex_lock(_lock);
+   if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
+   vdev->reflck = vfio_fsl_mc_reflck_alloc();
+   ret = PTR_ERR_OR_ZERO(vdev->reflck);
+   } else {
+   struct device *mc_cont_dev = vdev->mc_dev->dev.parent;
+   struct vfio_device *device;
+   struct vfio_fsl_mc_device *cont_vdev;
+
+   device = vfio_device_get_from_dev(mc_cont_dev);
+   if (!device) {
+   ret = -ENODEV;
+   goto unlock;
+   }
+
+   cont_vdev = vfio_device_data(device);
+   if (!cont_vdev || !cont_vdev->reflck) {
+   vfio_device_put(device);
+   ret = -ENODEV;
+   goto unlock;
+   }
+   vfio_fsl_mc_reflck_get(cont_vdev->reflck);
+   vdev->reflck = cont_vdev->reflck;
+   vfio_device_put(device);
+   }
+
+unlock:
+   mutex_unlock(_lock);
+   return ret;
+}
+
 static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
 {
struct fsl_mc_device *mc_dev = vdev->mc_dev;
@@ -62,7 +134,7 @@ static int vfio_fsl_mc_open(void *device_data)
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
if (!vdev->refcnt) {
ret = vfio_fsl_mc_regions_init(vdev);
if (ret)
@@ -70,12 +142,12 @@ static int vfio_fsl_mc_open(void *device_data)
}
vdev->refcnt++;
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
return 0;
 
 err_reg_init:
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
module_put(THIS_MODULE);
return ret;
 }
@@ -84,12 +156,12 @@ static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
 
if (!(--vdev->refcnt))
vfio_fsl_mc_regions_cleanup(vdev);
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
 }
@@ -343,14 +415,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
goto out_group_put;
}
 
-   ret = vfio_fsl_mc_init_device(vdev);
+   ret = vfio_fsl_mc_reflck_attach(vdev);
if (ret)
goto out_group_dev;
 
-   mutex_init(>driver_lock);
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret)
+   goto out_reflck;
 
return 0;
 
+out_reflck:
+   vfio_fsl_mc_reflck_put(vdev->reflck);
 out_group_dev:
vfio_del_group_dev(dev);
 out_group_put:
@@ -367,7 +443,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
-   mutex_destroy(>driver_lock);
+   vfio_fsl_mc_reflck_put(vdev->reflck);
 
if (is_fsl_mc_bus_dprc(mc_dev)) {
dprc

[PATCH v6 07/10] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices

[Diana Craciun]

This patch adds the skeleton for interrupt support
for fsl-mc devices. The interrupts are not yet functional,
the functionality will be added by subsequent patches.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/Makefile  |  2 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 52 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 34 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  6 +++
 4 files changed, 91 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
index 0c6e5d2ddaae..cad6dbf0b735 100644
--- a/drivers/vfio/fsl-mc/Makefile
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 
-vfio-fsl-mc-y := vfio_fsl_mc.o
+vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o
 obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index b52407c4e1ea..7803a9d6bfd9 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -217,11 +217,55 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
-   return -ENOTTY;
+   struct vfio_irq_info info;
+
+   minsz = offsetofend(struct vfio_irq_info, count);
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   info.flags = VFIO_IRQ_INFO_EVENTFD;
+   info.count = 1;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_SET_IRQS:
{
-   return -ENOTTY;
+   struct vfio_irq_set hdr;
+   u8 *data = NULL;
+   int ret = 0;
+   size_t data_size = 0;
+
+   minsz = offsetofend(struct vfio_irq_set, count);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   ret = vfio_set_irqs_validate_and_prepare(, 
mc_dev->obj_desc.irq_count,
+   mc_dev->obj_desc.irq_count, _size);
+   if (ret)
+   return ret;
+
+   if (data_size) {
+   data = memdup_user((void __user *)(arg + minsz),
+  data_size);
+   if (IS_ERR(data))
+   return PTR_ERR(data);
+   }
+
+   mutex_lock(>igate);
+   ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags,
+hdr.index, hdr.start,
+hdr.count, data);
+   mutex_unlock(>igate);
+   kfree(data);
+
+   return ret;
}
case VFIO_DEVICE_RESET:
{
@@ -423,6 +467,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
if (ret)
goto out_reflck;
 
+   mutex_init(>igate);
+
return 0;
 
 out_reflck:
@@ -443,6 +489,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   mutex_destroy(>igate);
+
vfio_fsl_mc_reflck_put(vdev->reflck);
 
if (is_fsl_mc_bus_dprc(mc_dev)) {
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
new file mode 100644
index ..5232f208e361
--- /dev/null
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
+/*
+ * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2019 NXP
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "linux/fsl/mc.h"
+#include "vfio_fsl_mc_private.h"
+
+static int vfio_fsl_mc_set_irq_trigger(struct vfio_fsl_mc_device *vdev,
+  unsigned int index, unsigned int start,
+  unsigned int count, u32 flags,
+  void *data)
+{
+   return -EINVAL;
+}
+
+int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev,
+  u32 flags, unsigned int index,
+  unsigned int start, unsigned int count,
+  void *data)
+{
+   if (flags & VFIO_IRQ_SET_ACTION_TRIGGER)
+   return  vfio_fsl_mc_set_irq_trigger(vdev, index, start,
+ count, flags, data);
+   else
+   return -EINVAL;
+}
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_

[PATCH v6 01/10] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

[Diana Craciun]

From: Bharat Bhushan 

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

This patch adds the infrastructure for VFIO support for fsl-mc
devices. Subsequent patches will add support for binding and secure
assigning these devices using VFIO.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 ++
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 157 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  14 ++
 include/uapi/linux/vfio.h |   1 +
 8 files changed, 193 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 33b27e62ce19..1046f4065ac1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18258,6 +18258,12 @@ F: drivers/vfio/
 F: include/linux/vfio.h
 F: include/uapi/linux/vfio.h
 
+VFIO FSL-MC DRIVER
+M: Diana Craciun 
+L: k...@vger.kernel.org
+S: Maintained
+F: drivers/vfio/fsl-mc/
+
 VFIO MEDIATED DEVICE DRIVERS
 M: Kirti Wankhede 
 L: k...@vger.kernel.org
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index fd17db9b432f..5533df91b257 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
 source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/fsl-mc/Kconfig"
 source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..fee73f3d9480 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
 obj-$(CONFIG_VFIO_PCI) += pci/
 obj-$(CONFIG_VFIO_PLATFORM) += platform/
 obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/
diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig
new file mode 100644
index ..b1a527d6b6f2
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Kconfig
@@ -0,0 +1,9 @@
+config VFIO_FSL_MC
+   tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices"
+   depends on VFIO && FSL_MC_BUS && EVENTFD
+   help
+ Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc
+ (Management Complex) devices. This is required to passthrough
+ fsl-mc bus devices using the VFIO framework.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
new file mode 100644
index ..0c6e5d2ddaae
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -0,0 +1,4 @@
+# SPDX-Lic

[PATCH v6 08/10] vfio/fsl-mc: trigger an interrupt via eventfd

[Diana Craciun]

This patch allows to set an eventfd for fsl-mc device interrupts
and also to trigger the interrupt eventfd from userspace for testing.

All fsl-mc device interrupts are MSIs. The MSIs are allocated from
the MSI domain only once per DPRC and used by all the DPAA2 objects.
The interrupts are managed by the DPRC in a pool of interrupts. Each
device requests interrupts from this pool. The pool is allocated
when the first virtual device is setting the interrupts.
The pool of interrupts is protected by a lock.

The DPRC has an interrupt of its own which indicates if the DPRC
contents have changed. However, currently, the contents of a DPRC
assigned to the guest cannot be changed at runtime, so this interrupt
is not configured.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c |  24 +++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 162 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  10 ++
 3 files changed, 194 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 7803a9d6bfd9..feb5dafd4bf5 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -155,12 +155,34 @@ static int vfio_fsl_mc_open(void *device_data)
 static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
 
mutex_lock(>reflck->lock);
 
-   if (!(--vdev->refcnt))
+   if (!(--vdev->refcnt)) {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
+
vfio_fsl_mc_regions_cleanup(vdev);
 
+   /* reset the device before cleaning up the interrupts */
+   ret = dprc_reset_container(mc_cont->mc_io, 0,
+ mc_cont->mc_handle,
+ mc_cont->obj_desc.id,
+ DPRC_RESET_OPTION_NON_RECURSIVE);
+
+   if (ret) {
+   dev_warn(_cont->dev, "VFIO_FLS_MC: reset device has 
failed (%d)\n",
+ret);
+   WARN_ON(1);
+   }
+
+   vfio_fsl_mc_irqs_cleanup(vdev);
+
+   fsl_mc_cleanup_irq_pool(mc_cont);
+   }
+
mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 5232f208e361..2ce2acad3461 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -13,12 +13,150 @@
 #include "linux/fsl/mc.h"
 #include "vfio_fsl_mc_private.h"
 
+int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_irq *mc_irq;
+   int irq_count;
+   int ret, i;
+
+   /* Device does not support any interrupt */
+   if (mc_dev->obj_desc.irq_count == 0)
+   return 0;
+
+   /* interrupts were already allocated for this device */
+   if (vdev->mc_irqs)
+   return 0;
+
+   irq_count = mc_dev->obj_desc.irq_count;
+
+   mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
+   if (!mc_irq)
+   return -ENOMEM;
+
+   /* Allocate IRQs */
+   ret = fsl_mc_allocate_irqs(mc_dev);
+   if (ret) {
+   kfree(mc_irq);
+   return ret;
+   }
+
+   for (i = 0; i < irq_count; i++) {
+   mc_irq[i].count = 1;
+   mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
+   }
+
+   vdev->mc_irqs = mc_irq;
+
+   return 0;
+}
+
+static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg)
+{
+   struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg;
+
+   eventfd_signal(mc_irq->trigger, 1);
+   return IRQ_HANDLED;
+}
+
+static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev,
+  int index, int fd)
+{
+   struct vfio_fsl_mc_irq *irq = >mc_irqs[index];
+   struct eventfd_ctx *trigger;
+   int hwirq;
+   int ret;
+
+   hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq;
+   if (irq->trigger) {
+   free_irq(hwirq, irq);
+   kfree(irq->name);
+   eventfd_ctx_put(irq->trigger);
+   irq->trigger = NULL;
+   }
+
+   if (fd < 0) /* Disable only */
+   return 0;
+
+   irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)",
+   hwirq, dev_name(>mc_dev->dev));
+   if (!irq->name)
+   return -ENOMEM;
+
+   trigger = eventfd_ctx_fdget(fd);
+   if (IS_ERR(trigger)) {
+

[PATCH v6 04/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call

[Diana Craciun]

Expose to userspace information about the memory regions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 79 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 18 ++
 2 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 161c2cbe07dc..05dace5ddc2c 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,16 +17,71 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int count = mc_dev->obj_desc.region_count;
+   int i;
+
+   vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region),
+   GFP_KERNEL);
+   if (!vdev->regions)
+   return -ENOMEM;
+
+   for (i = 0; i < count; i++) {
+   struct resource *res = _dev->regions[i];
+
+   vdev->regions[i].addr = res->start;
+   vdev->regions[i].size = resource_size(res);
+   vdev->regions[i].flags = 0;
+   vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   }
+
+   return 0;
+}
+
+static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
+{
+   kfree(vdev->regions);
+}
+
 static int vfio_fsl_mc_open(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
+
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
+   mutex_lock(>driver_lock);
+   if (!vdev->refcnt) {
+   ret = vfio_fsl_mc_regions_init(vdev);
+   if (ret)
+   goto err_reg_init;
+   }
+   vdev->refcnt++;
+
+   mutex_unlock(>driver_lock);
+
return 0;
+
+err_reg_init:
+   mutex_unlock(>driver_lock);
+   module_put(THIS_MODULE);
+   return ret;
 }
 
 static void vfio_fsl_mc_release(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+
+   mutex_lock(>driver_lock);
+
+   if (!(--vdev->refcnt))
+   vfio_fsl_mc_regions_cleanup(vdev);
+
+   mutex_unlock(>driver_lock);
+
module_put(THIS_MODULE);
 }
 
@@ -59,7 +114,25 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-   return -ENOTTY;
+   struct vfio_region_info info;
+
+   minsz = offsetofend(struct vfio_region_info, offset);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   /* map offset to the physical address  */
+   info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
+   info.size = vdev->regions[info.index].size;
+   info.flags = vdev->regions[info.index].flags;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -210,6 +283,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
if (ret)
goto out_group_dev;
 
+   mutex_init(>driver_lock);
+
return 0;
 
 out_group_dev:
@@ -228,6 +303,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   mutex_destroy(>driver_lock);
+
if (is_fsl_mc_bus_dprc(mc_dev)) {
dprc_remove_devices(mc_dev, NULL, 0);
dprc_cleanup(mc_dev);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index 37d61eaa58c8..be60f41af30f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -7,9 +7,27 @@
 #ifndef VFIO_FSL_MC_PRIVATE_H
 #define VFIO_FSL_MC_PRIVATE_H
 
+#define VFIO_FSL_MC_OFFSET_SHIFT40
+#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1)
+
+#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT)
+
+#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \
+   ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT)
+
+struct vfio_fsl_mc_region {
+   u32 flags;
+   u32 type;
+   u64 addr;
+   resource_size_t size;
+};
+
 struct vfio_fsl_mc_device {
struct fsl_mc_device*mc_dev;
struct notifier_blocknb;
+   int refcnt;
+   struct vfio_fsl_mc_region   *regions;
+   struct mutex driver_lock;
 };
 
 #endif /* VFIO_FSL_MC_PRIVATE_H */
-- 
2.17.1

[PATCH v6 10/10] vfio/fsl-mc: Add support for device reset

[Diana Craciun]

Currently only resetting the DPRC container is supported which
will reset all the objects inside it. Resetting individual
objects is possible from the userspace by issueing commands
towards MC firmware.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index d95568cd8021..d009f873578c 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -217,6 +217,10 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
return -EINVAL;
 
info.flags = VFIO_DEVICE_FLAGS_FSL_MC;
+
+   if (is_fsl_mc_bus_dprc(mc_dev))
+   info.flags |= VFIO_DEVICE_FLAGS_RESET;
+
info.num_regions = mc_dev->obj_desc.region_count;
info.num_irqs = mc_dev->obj_desc.irq_count;
 
@@ -299,7 +303,19 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_RESET:
{
-   return -ENOTTY;
+   int ret;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
+   /* reset is supported only for the DPRC */
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -ENOTTY;
+
+   ret = dprc_reset_container(mc_dev->mc_io, 0,
+  mc_dev->mc_handle,
+  mc_dev->obj_desc.id,
+  DPRC_RESET_OPTION_NON_RECURSIVE);
+   return ret;
+
}
default:
return -ENOTTY;
-- 
2.17.1

[PATCH v6 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun]

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 118 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
 2 files changed, 116 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index feb5dafd4bf5..d95568cd8021 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
@@ -115,7 +116,9 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
!(vdev->regions[i].size & ~PAGE_MASK))
vdev->regions[i].flags |=
VFIO_REGION_INFO_FLAG_MMAP;
-
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
 
return 0;
@@ -123,6 +126,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
 static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
 {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int i;
+
+   for (i = 0; i < mc_dev->obj_desc.region_count; i++)
+   iounmap(vdev->regions[i].ioaddr);
kfree(vdev->regions);
 }
 
@@ -301,13 +309,117 @@ static long vfio_fsl_mc_ioctl(void *device_data, 
unsigned int cmd,
 static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;
+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+   int i;
+   enum mc_cmd_status status;
+   unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+   /* Write at command parameter into portal */
+   for (i = 7; i >= 1; i--)
+   writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+   /* Write command header in the end */
+   writeq(cmd_data[0], ioaddr);
+
+   /* Wait for response before returning to user-space
+* This can be optimized in future to even prepare response
+* before returning to user-space and avoid read ioctl.
+*/
+   for (;;) {
+   u64 header;
+   struct mc_cmd_header *resp_hdr;
+
+   header = cpu_to_le64(readq_relaxed(ioaddr));
+
+   resp_hdr = (struct mc_cmd_header *)
+   status = (enum mc_cmd_status)resp_hdr->status;
+   if (status != MC_CMD_STATUS_READY)
+   break;
+
+   udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS);
+   timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS;
+   if (timeout_usecs == 0)
+   return -ETIMEDOUT;
+   }
+
+   return 0;
 }
 
 static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf,
 size_t count, loff_t *ppos)
 {
-  

[PATCH v6 03/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl

[Diana Craciun]

Allow userspace to get fsl-mc device info (number of regions
and irqs).

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
Reviewed-by: Eric Auger 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 21 -
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 594760203268..161c2cbe07dc 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,10 +33,29 @@ static void vfio_fsl_mc_release(void *device_data)
 static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
  unsigned long arg)
 {
+   unsigned long minsz;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
switch (cmd) {
case VFIO_DEVICE_GET_INFO:
{
-   return -ENOTTY;
+   struct vfio_device_info info;
+
+   minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   info.flags = VFIO_DEVICE_FLAGS_FSL_MC;
+   info.num_regions = mc_dev->obj_desc.region_count;
+   info.num_irqs = mc_dev->obj_desc.irq_count;
+
+   return copy_to_user((void __user *)arg, , minsz) ?
+   -EFAULT : 0;
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-- 
2.17.1

[PATCH v6 00/10] vfio/fsl-mc: VFIO support for FSL-MC device

[Diana Craciun]

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
driver. The changes were needed in order to re-use code and to export
some more functions that are needed by the VFIO driver.
Currenlty the mc-bus patches were queued for merging.
https://www.spinics.net/lists/kernel/msg3680670.html

v5 --> v6
- style fixes
- review fixes

v4 --> v5
- do not allow mmap for DPRCs
- style fixes

v3 --> v4
- use bus provided functions to tear down the DPRC
- added reset support

v2 --> v3
- There is no need to align region size to page size
- read/write implemented for all DPAA2 objects
- review fixes

v1 --> v2
- Fixed the container reset, a new flag added to the firmware command
- Implement a bus notifier for setting driver_override


Bharat Bhushan (1):
  vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

Diana Craciun (9):
  vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
  vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
  vfio/fsl-mc: Added lock support in preparation for interrupt handling
  vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
  vfio/fsl-mc: trigger an interrupt via eventfd
  vfio/fsl-mc: Add read/write support for fsl-mc devices
  vfio/fsl-mc: Add support for device reset

 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 +
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 682 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 194 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  55 ++
 include/uapi/linux/vfio.h |   1 +
 9 files changed, 953 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

-- 
2.17.1

Re: [PATCH v5 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun OSS]

On 10/2/2020 11:50 PM, Alex Williamson wrote:

On Tue, 29 Sep 2020 12:03:38 +0300
Diana Craciun  wrote:


The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 118 +-
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
  2 files changed, 116 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 82157837f37a..0aff99cdf722 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #include "vfio_fsl_mc_private.h"
  
@@ -115,7 +116,9 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)

!(vdev->regions[i].size & ~PAGE_MASK))
vdev->regions[i].flags |=
VFIO_REGION_INFO_FLAG_MMAP;
-
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
  
  	return 0;

@@ -123,6 +126,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
  
  static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)

  {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int i;
+
+   for (i = 0; i < mc_dev->obj_desc.region_count; i++)
+   iounmap(vdev->regions[i].ioaddr);
kfree(vdev->regions);
  }
  
@@ -301,13 +309,117 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,

  static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
  {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;


Nit, there are no regions w/o read access according to the regions_init
code above.  Maybe this is just for symmetry with write?  Keep it if
you prefer.  Thanks,


I would prefer to keep it like this for symmetry with write.

Thanks,
Diana



Alex


+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+   int i;
+   enum mc_cmd_status status;
+   unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+   /* Write at command parameter into portal */
+   for (i = 7; i >= 1; i--)
+   writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+   /* Write command header in the end */
+   writeq(cmd_data[0], ioaddr);
+
+   /* Wait for response before returning to user-space
+* This can be optimized in future to even prepare response
+* before returning to user-space and avoid read ioctl.
+*/
+   for (;;) {
+   u64 header;
+   struct mc_cmd_header *resp_hdr;
+
+   header = cpu_to_le64(readq_relaxed(ioaddr));
+
+   resp_hdr = (struct mc_cmd_header *)
+   status = (enum mc_cmd_status)resp_hdr->status;
+   if (status != MC_CMD_STATUS_READY)
+   break;
+
+   udelay(MC_CMD_COMPLETION_PO

Re: [PATCH v5 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun OSS]

On 10/2/2020 8:24 PM, Alex Williamson wrote:

On Tue, 29 Sep 2020 12:03:31 +0300
Diana Craciun  wrote:


The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 90 +++
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  1 +
  2 files changed, 91 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index a7a483a1e90b..ba44d6d01cc9 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
  
  #include "vfio_fsl_mc_private.h"
  
+static struct fsl_mc_driver vfio_fsl_mc_driver;

+
  static int vfio_fsl_mc_open(void *device_data)
  {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,79 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
  };
  
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,

+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   if (!mc_dev->driver_override)
+   dev_warn(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s failed\n",
+dev_name(_cont->dev));
+   else
+   dev_info(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s\n",
+dev_name(_cont->dev));


Nit, some whitespace inconsistencies on the second line of each of
these.  I can fixup on commit if we don't find anything else worth a
respin.


+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "VFIO_FSL_MC: Object %s bound to driver %s 
while DPRC bound to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Failed to setup DPRC 
(%d)\n", ret);
+   goto out_nc_unreg;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Container scanning failed 
(%d)\n", ret);
+   goto out_dprc_cleanup;
+   }


If I understand this correctly, we've setup the bus notifier to write
the driver override as each sub-devices appear on the bus from this
scan.  When non-dprc devices are removed below, it appears we remove all
their sub-devices.  Is there a chance here that an error from the scan
leaves residual sub-devices, ie. should we proceed the below
dprc_cleanup() with a call to dprc_remove_devices() to be certain none
remain?  Thanks,

Alex


Right, we should call dprc_remove_devices as well. I will respin another 
version.






+
+   return 0;
+
+out_dprc_cleanup:
+   dprc_cleanup(mc_dev);
+out_nc_unreg:
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   vdev->nb.notifier_call = NULL;
+
+   return ret;
+}
+
  static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
  {
struct iommu_group *group;
@@ -110,8 +185,15 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
dev_err(dev, "VFIO_FSL_MC: Failed to add to vfio group\n");
goto out_group_put;
}
+
+   ret = vfio_fsl_mc

[PATCH v5 00/10] vfio/fsl-mc: VFIO support for FSL-MC device

[Diana Craciun]

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
driver. The changes were needed in order to re-use code and to export
some more functions that are needed by the VFIO driver.
Currenlty the mc-bus patches are under review:
https://www.spinics.net/lists/kernel/msg3680670.html

v4 --> v5
- do not allow mmap for DPRCs
- style fixes

v3 --> v4
- use bus provided functions to tear down the DPRC
- added reset support

v2 --> v3
- There is no need to align region size to page size
- read/write implemented for all DPAA2 objects
- review fixes

v1 --> v2
- Fixed the container reset, a new flag added to the firmware command
- Implement a bus notifier for setting driver_override

Bharat Bhushan (1):
  vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

Diana Craciun (9):
  vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
  vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
  vfio/fsl-mc: Added lock support in preparation for interrupt handling
  vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
  vfio/fsl-mc: trigger an interrupt via eventfd
  vfio/fsl-mc: Add read/write support for fsl-mc devices
  vfio/fsl-mc: Add support for device reset

 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 +
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 677 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 193 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  55 ++
 include/uapi/linux/vfio.h |   1 +
 9 files changed, 947 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

-- 
2.17.1

[PATCH v5 04/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call

[Diana Craciun]

Expose to userspace information about the memory regions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 79 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 18 ++
 2 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index fa46676c735b..802f3a00337f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,16 +17,71 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int count = mc_dev->obj_desc.region_count;
+   int i;
+
+   vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region),
+   GFP_KERNEL);
+   if (!vdev->regions)
+   return -ENOMEM;
+
+   for (i = 0; i < count; i++) {
+   struct resource *res = _dev->regions[i];
+
+   vdev->regions[i].addr = res->start;
+   vdev->regions[i].size = resource_size(res);
+   vdev->regions[i].flags = 0;
+   vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   }
+
+   return 0;
+}
+
+static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
+{
+   kfree(vdev->regions);
+}
+
 static int vfio_fsl_mc_open(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
+
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
+   mutex_lock(>driver_lock);
+   if (!vdev->refcnt) {
+   ret = vfio_fsl_mc_regions_init(vdev);
+   if (ret)
+   goto err_reg_init;
+   }
+   vdev->refcnt++;
+
+   mutex_unlock(>driver_lock);
+
return 0;
+
+err_reg_init:
+   mutex_unlock(>driver_lock);
+   module_put(THIS_MODULE);
+   return ret;
 }
 
 static void vfio_fsl_mc_release(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+
+   mutex_lock(>driver_lock);
+
+   if (!(--vdev->refcnt))
+   vfio_fsl_mc_regions_cleanup(vdev);
+
+   mutex_unlock(>driver_lock);
+
module_put(THIS_MODULE);
 }
 
@@ -59,7 +114,25 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-   return -ENOTTY;
+   struct vfio_region_info info;
+
+   minsz = offsetofend(struct vfio_region_info, offset);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   /* map offset to the physical address  */
+   info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
+   info.size = vdev->regions[info.index].size;
+   info.flags = vdev->regions[info.index].flags;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -209,6 +282,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
if (ret)
goto out_group_dev;
 
+   mutex_init(>driver_lock);
+
return 0;
 
 out_group_dev:
@@ -227,6 +302,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   mutex_destroy(>driver_lock);
+
if (is_fsl_mc_bus_dprc(mc_dev)) {
dprc_remove_devices(mc_dev, NULL, 0);
dprc_cleanup(mc_dev);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index 37d61eaa58c8..be60f41af30f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -7,9 +7,27 @@
 #ifndef VFIO_FSL_MC_PRIVATE_H
 #define VFIO_FSL_MC_PRIVATE_H
 
+#define VFIO_FSL_MC_OFFSET_SHIFT40
+#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1)
+
+#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT)
+
+#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \
+   ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT)
+
+struct vfio_fsl_mc_region {
+   u32 flags;
+   u32 type;
+   u64 addr;
+   resource_size_t size;
+};
+
 struct vfio_fsl_mc_device {
struct fsl_mc_device*mc_dev;
struct notifier_blocknb;
+   int refcnt;
+   struct vfio_fsl_mc_region   *regions;
+   struct mutex driver_lock;
 };
 
 #endif /* VFIO_FSL_MC_PRIVATE_H */
-- 
2.17.1

[PATCH v5 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun]

The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 90 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  1 +
 2 files changed, 91 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index a7a483a1e90b..ba44d6d01cc9 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
 
 #include "vfio_fsl_mc_private.h"
 
+static struct fsl_mc_driver vfio_fsl_mc_driver;
+
 static int vfio_fsl_mc_open(void *device_data)
 {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,79 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
 };
 
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,
+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   if (!mc_dev->driver_override)
+   dev_warn(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s failed\n",
+dev_name(_cont->dev));
+   else
+   dev_info(dev, "VFIO_FSL_MC: Setting driver override for 
device in dprc %s\n",
+dev_name(_cont->dev));
+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "VFIO_FSL_MC: Object %s bound to driver 
%s while DPRC bound to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Failed to setup DPRC 
(%d)\n", ret);
+   goto out_nc_unreg;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret) {
+   dev_err(_dev->dev, "VFIO_FSL_MC: Container scanning failed 
(%d)\n", ret);
+   goto out_dprc_cleanup;
+   }
+
+   return 0;
+
+out_dprc_cleanup:
+   dprc_cleanup(mc_dev);
+out_nc_unreg:
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   vdev->nb.notifier_call = NULL;
+
+   return ret;
+}
+
 static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
 {
struct iommu_group *group;
@@ -110,8 +185,15 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
dev_err(dev, "VFIO_FSL_MC: Failed to add to vfio group\n");
goto out_group_put;
}
+
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret)
+   goto out_group_dev;
+
return 0;
 
+out_group_dev:
+   vfio_del_group_dev(dev);
 out_group_put:
vfio_iommu_group_put(group, dev);
return ret;
@@ -126,6 +208,14 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   if (is_fsl_mc_bus_dprc(mc_dev)) {
+   dprc_remove_devices(mc_dev, NULL, 0);
+   dprc_cleanup(mc_dev);
+   }
+
+   if (vdev->nb.notifier_call)
+   bus_unregister_notifier(_mc_bus_type, >nb);
+
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
return 0;
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index e79c

[PATCH v5 01/10] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

[Diana Craciun]

From: Bharat Bhushan 

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

This patch adds the infrastructure for VFIO support for fsl-mc
devices. Subsequent patches will add support for binding and secure
assigning these devices using VFIO.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 ++
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 157 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  14 ++
 include/uapi/linux/vfio.h |   1 +
 8 files changed, 193 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 9350506a1127..cf2c7b21fefd 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18257,6 +18257,12 @@ F: drivers/vfio/
 F: include/linux/vfio.h
 F: include/uapi/linux/vfio.h
 
+VFIO FSL-MC DRIVER
+M: Diana Craciun 
+L: k...@vger.kernel.org
+S: Maintained
+F: drivers/vfio/fsl-mc/
+
 VFIO MEDIATED DEVICE DRIVERS
 M: Kirti Wankhede 
 L: k...@vger.kernel.org
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index fd17db9b432f..5533df91b257 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
 source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/fsl-mc/Kconfig"
 source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..fee73f3d9480 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
 obj-$(CONFIG_VFIO_PCI) += pci/
 obj-$(CONFIG_VFIO_PLATFORM) += platform/
 obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/
diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig
new file mode 100644
index ..b1a527d6b6f2
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Kconfig
@@ -0,0 +1,9 @@
+config VFIO_FSL_MC
+   tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices"
+   depends on VFIO && FSL_MC_BUS && EVENTFD
+   help
+ Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc
+ (Management Complex) devices. This is required to passthrough
+ fsl-mc bus devices using the VFIO framework.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
new file mode 100644
index ..0c6e5d2ddaae
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -0,0 +1,4 @@
+# SPDX-Lic

[PATCH v5 08/10] vfio/fsl-mc: trigger an interrupt via eventfd

[Diana Craciun]

This patch allows to set an eventfd for fsl-mc device interrupts
and also to trigger the interrupt eventfd from userspace for testing.

All fsl-mc device interrupts are MSIs. The MSIs are allocated from
the MSI domain only once per DPRC and used by all the DPAA2 objects.
The interrupts are managed by the DPRC in a pool of interrupts. Each
device requests interrupts from this pool. The pool is allocated
when the first virtual device is setting the interrupts.
The pool of interrupts is protected by a lock.

The DPRC has an interrupt of its own which indicates if the DPRC
contents have changed. However, currently, the contents of a DPRC
assigned to the guest cannot be changed at runtime, so this interrupt
is not configured.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c |  24 +++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 161 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  10 ++
 3 files changed, 193 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 2919e2d0041b..82157837f37a 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -155,12 +155,34 @@ static int vfio_fsl_mc_open(void *device_data)
 static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
 
mutex_lock(>reflck->lock);
 
-   if (!(--vdev->refcnt))
+   if (!(--vdev->refcnt)) {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
+
vfio_fsl_mc_regions_cleanup(vdev);
 
+   /* reset the device before cleaning up the interrupts */
+   ret = dprc_reset_container(mc_cont->mc_io, 0,
+ mc_cont->mc_handle,
+ mc_cont->obj_desc.id,
+ DPRC_RESET_OPTION_NON_RECURSIVE);
+
+   if (ret) {
+   dev_warn(_cont->dev, "VFIO_FLS_MC: reset device has 
failed (%d)\n",
+ret);
+   WARN_ON(1);
+   }
+
+   vfio_fsl_mc_irqs_cleanup(vdev);
+
+   fsl_mc_cleanup_irq_pool(mc_cont);
+   }
+
mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 5232f208e361..992ee18f1f6f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -13,12 +13,150 @@
 #include "linux/fsl/mc.h"
 #include "vfio_fsl_mc_private.h"
 
+int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_irq *mc_irq;
+   int irq_count;
+   int ret, i;
+
+   /* Device does not support any interrupt */
+   if (mc_dev->obj_desc.irq_count == 0)
+   return 0;
+
+   /* interrupts were already allocated for this device */
+   if (vdev->mc_irqs)
+   return 0;
+
+   irq_count = mc_dev->obj_desc.irq_count;
+
+   mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
+   if (!mc_irq)
+   return -ENOMEM;
+
+   /* Allocate IRQs */
+   ret = fsl_mc_allocate_irqs(mc_dev);
+   if (ret) {
+   kfree(mc_irq);
+   return ret;
+   }
+
+   for (i = 0; i < irq_count; i++) {
+   mc_irq[i].count = 1;
+   mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
+   }
+
+   vdev->mc_irqs = mc_irq;
+
+   return 0;
+}
+
+static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg)
+{
+   struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg;
+
+   eventfd_signal(mc_irq->trigger, 1);
+   return IRQ_HANDLED;
+}
+
+static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev,
+  int index, int fd)
+{
+   struct vfio_fsl_mc_irq *irq = >mc_irqs[index];
+   struct eventfd_ctx *trigger;
+   int hwirq;
+   int ret;
+
+   hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq;
+   if (irq->trigger) {
+   free_irq(hwirq, irq);
+   kfree(irq->name);
+   eventfd_ctx_put(irq->trigger);
+   irq->trigger = NULL;
+   }
+
+   if (fd < 0) /* Disable only */
+   return 0;
+
+   irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)",
+   hwirq, dev_name(>mc_dev->dev));
+   if (!irq->name)
+   return -ENOMEM;
+
+   trigger = eventfd_ctx_fdget(fd);
+   if (IS_ERR(trigger)) {
+

[PATCH v5 06/10] vfio/fsl-mc: Added lock support in preparation for interrupt handling

[Diana Craciun]

Only the DPRC object allocates interrupts from the MSI
interrupt domain. The interrupts are managed by the DPRC in
a pool of interrupts. The access to this pool of interrupts
has to be protected with a lock.
This patch extends the current lock implementation to have a
lock per DPRC.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 92 +--
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  7 +-
 2 files changed, 90 insertions(+), 9 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 7ceadb801082..548fb4d80f3f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,6 +17,78 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static DEFINE_MUTEX(reflck_lock);
+
+static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_get(>kref);
+}
+
+static void vfio_fsl_mc_reflck_release(struct kref *kref)
+{
+   struct vfio_fsl_mc_reflck *reflck = container_of(kref,
+ struct vfio_fsl_mc_reflck,
+ kref);
+
+   mutex_destroy(>lock);
+   kfree(reflck);
+   mutex_unlock(_lock);
+}
+
+static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_put_mutex(>kref, vfio_fsl_mc_reflck_release, _lock);
+}
+
+static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void)
+{
+   struct vfio_fsl_mc_reflck *reflck;
+
+   reflck = kzalloc(sizeof(*reflck), GFP_KERNEL);
+   if (!reflck)
+   return ERR_PTR(-ENOMEM);
+
+   kref_init(>kref);
+   mutex_init(>lock);
+
+   return reflck;
+}
+
+static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
+{
+   int ret;
+
+   mutex_lock(_lock);
+   if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
+   vdev->reflck = vfio_fsl_mc_reflck_alloc();
+   ret = PTR_ERR_OR_ZERO(vdev->reflck);
+   } else {
+   struct device *mc_cont_dev = vdev->mc_dev->dev.parent;
+   struct vfio_device *device;
+   struct vfio_fsl_mc_device *cont_vdev;
+
+   device = vfio_device_get_from_dev(mc_cont_dev);
+   if (!device) {
+   ret = -ENODEV;
+   goto unlock;
+   }
+
+   cont_vdev = vfio_device_data(device);
+   if (!cont_vdev || !cont_vdev->reflck) {
+   vfio_device_put(device);
+   ret = -ENODEV;
+   goto unlock;
+   }
+   vfio_fsl_mc_reflck_get(cont_vdev->reflck);
+   vdev->reflck = cont_vdev->reflck;
+   vfio_device_put(device);
+   }
+
+unlock:
+   mutex_unlock(_lock);
+   return ret;
+}
+
 static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
 {
struct fsl_mc_device *mc_dev = vdev->mc_dev;
@@ -62,7 +134,7 @@ static int vfio_fsl_mc_open(void *device_data)
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
if (!vdev->refcnt) {
ret = vfio_fsl_mc_regions_init(vdev);
if (ret)
@@ -70,12 +142,12 @@ static int vfio_fsl_mc_open(void *device_data)
}
vdev->refcnt++;
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
return 0;
 
 err_reg_init:
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
module_put(THIS_MODULE);
return ret;
 }
@@ -84,12 +156,12 @@ static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
 
if (!(--vdev->refcnt))
vfio_fsl_mc_regions_cleanup(vdev);
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
 }
@@ -342,14 +414,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
goto out_group_put;
}
 
-   ret = vfio_fsl_mc_init_device(vdev);
+   ret = vfio_fsl_mc_reflck_attach(vdev);
if (ret)
goto out_group_dev;
 
-   mutex_init(>driver_lock);
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret)
+   goto out_reflck;
 
return 0;
 
+out_reflck:
+   vfio_fsl_mc_reflck_put(vdev->reflck);
 out_group_dev:
vfio_del_group_dev(dev);
 out_group_put:
@@ -366,7 +442,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
-   mutex_destroy(>driver_lock);
+   vfio_fsl_mc_reflck_put(vdev->reflck);
 
if (is_fsl_mc_bus_dprc(mc_dev)) {
dprc

[PATCH v5 05/10] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions

[Diana Craciun]

Allow userspace to mmap device regions for direct access of
fsl-mc devices.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 68 ++-
 1 file changed, 66 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 802f3a00337f..7ceadb801082 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -30,11 +30,20 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
for (i = 0; i < count; i++) {
struct resource *res = _dev->regions[i];
+   int no_mmap = is_fsl_mc_bus_dprc(mc_dev);
 
vdev->regions[i].addr = res->start;
vdev->regions[i].size = resource_size(res);
-   vdev->regions[i].flags = 0;
vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   /*
+* Only regions addressed with PAGE granularity may be
+* MMAPed securely.
+*/
+   if (!no_mmap && !(vdev->regions[i].addr & ~PAGE_MASK) &&
+   !(vdev->regions[i].size & ~PAGE_MASK))
+   vdev->regions[i].flags |=
+   VFIO_REGION_INFO_FLAG_MMAP;
+
}
 
return 0;
@@ -163,9 +172,64 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const 
char __user *buf,
return -EINVAL;
 }
 
+static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,
+struct vm_area_struct *vma)
+{
+   u64 size = vma->vm_end - vma->vm_start;
+   u64 pgoff, base;
+   u8 region_cacheable;
+
+   pgoff = vma->vm_pgoff &
+   ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+   base = pgoff << PAGE_SHIFT;
+
+   if (region.size < PAGE_SIZE || base + size > region.size)
+   return -EINVAL;
+
+   region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) &&
+  (region.type & FSL_MC_REGION_SHAREABLE);
+   if (!region_cacheable)
+   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+   vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff;
+
+   return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+  size, vma->vm_page_prot);
+}
+
 static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int index;
+
+   index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
+
+   if (vma->vm_end < vma->vm_start)
+   return -EINVAL;
+   if (vma->vm_start & ~PAGE_MASK)
+   return -EINVAL;
+   if (vma->vm_end & ~PAGE_MASK)
+   return -EINVAL;
+   if (!(vma->vm_flags & VM_SHARED))
+   return -EINVAL;
+   if (index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ)
+   && (vma->vm_flags & VM_READ))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)
+   && (vma->vm_flags & VM_WRITE))
+   return -EINVAL;
+
+   vma->vm_private_data = mc_dev;
+
+   return vfio_fsl_mc_mmap_mmio(vdev->regions[index], vma);
 }
 
 static const struct vfio_device_ops vfio_fsl_mc_ops = {
-- 
2.17.1

[PATCH v5 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun]

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 118 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
 2 files changed, 116 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 82157837f37a..0aff99cdf722 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
@@ -115,7 +116,9 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
!(vdev->regions[i].size & ~PAGE_MASK))
vdev->regions[i].flags |=
VFIO_REGION_INFO_FLAG_MMAP;
-
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
 
return 0;
@@ -123,6 +126,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
 static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
 {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int i;
+
+   for (i = 0; i < mc_dev->obj_desc.region_count; i++)
+   iounmap(vdev->regions[i].ioaddr);
kfree(vdev->regions);
 }
 
@@ -301,13 +309,117 @@ static long vfio_fsl_mc_ioctl(void *device_data, 
unsigned int cmd,
 static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= mc_dev->obj_desc.region_count)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;
+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+   int i;
+   enum mc_cmd_status status;
+   unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+   /* Write at command parameter into portal */
+   for (i = 7; i >= 1; i--)
+   writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+   /* Write command header in the end */
+   writeq(cmd_data[0], ioaddr);
+
+   /* Wait for response before returning to user-space
+* This can be optimized in future to even prepare response
+* before returning to user-space and avoid read ioctl.
+*/
+   for (;;) {
+   u64 header;
+   struct mc_cmd_header *resp_hdr;
+
+   header = cpu_to_le64(readq_relaxed(ioaddr));
+
+   resp_hdr = (struct mc_cmd_header *)
+   status = (enum mc_cmd_status)resp_hdr->status;
+   if (status != MC_CMD_STATUS_READY)
+   break;
+
+   udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS);
+   timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS;
+   if (timeout_usecs == 0)
+   return -ETIMEDOUT;
+   }
+
+   return 0;
 }
 
 static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf,
 size_t count, loff_t *ppos)
 {
-  

[PATCH v5 03/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl

[Diana Craciun]

Allow userspace to get fsl-mc device info (number of regions
and irqs).

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
Reviewed-by: Eric Auger 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 21 -
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index ba44d6d01cc9..fa46676c735b 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,10 +33,29 @@ static void vfio_fsl_mc_release(void *device_data)
 static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
  unsigned long arg)
 {
+   unsigned long minsz;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
switch (cmd) {
case VFIO_DEVICE_GET_INFO:
{
-   return -ENOTTY;
+   struct vfio_device_info info;
+
+   minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   info.flags = VFIO_DEVICE_FLAGS_FSL_MC;
+   info.num_regions = mc_dev->obj_desc.region_count;
+   info.num_irqs = mc_dev->obj_desc.irq_count;
+
+   return copy_to_user((void __user *)arg, , minsz) ?
+   -EFAULT : 0;
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-- 
2.17.1

[PATCH v5 10/10] vfio/fsl-mc: Add support for device reset

[Diana Craciun]

Currently only resetting the DPRC container is supported which
will reset all the objects inside it. Resetting individual
objects is possible from the userspace by issueing commands
towards MC firmware.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 0aff99cdf722..e1b2dea8a5fe 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -299,7 +299,19 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_RESET:
{
-   return -ENOTTY;
+   int ret;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
+   /* reset is supported only for the DPRC */
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -ENOTTY;
+
+   ret = dprc_reset_container(mc_dev->mc_io, 0,
+  mc_dev->mc_handle,
+  mc_dev->obj_desc.id,
+  DPRC_RESET_OPTION_NON_RECURSIVE);
+   return ret;
+
}
default:
return -ENOTTY;
-- 
2.17.1

[PATCH v5 07/10] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices

[Diana Craciun]

This patch adds the skeleton for interrupt support
for fsl-mc devices. The interrupts are not yet functional,
the functionality will be added by subsequent patches.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/Makefile  |  2 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 52 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 34 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  6 +++
 4 files changed, 91 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
index 0c6e5d2ddaae..cad6dbf0b735 100644
--- a/drivers/vfio/fsl-mc/Makefile
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 
-vfio-fsl-mc-y := vfio_fsl_mc.o
+vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o
 obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 548fb4d80f3f..2919e2d0041b 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -217,11 +217,55 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
-   return -ENOTTY;
+   struct vfio_irq_info info;
+
+   minsz = offsetofend(struct vfio_irq_info, count);
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   info.flags = VFIO_IRQ_INFO_EVENTFD;
+   info.count = 1;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_SET_IRQS:
{
-   return -ENOTTY;
+   struct vfio_irq_set hdr;
+   u8 *data = NULL;
+   int ret = 0;
+   size_t data_size = 0;
+
+   minsz = offsetofend(struct vfio_irq_set, count);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   ret = vfio_set_irqs_validate_and_prepare(, 
mc_dev->obj_desc.irq_count,
+   mc_dev->obj_desc.irq_count, _size);
+   if (ret)
+   return ret;
+
+   if (data_size) {
+   data = memdup_user((void __user *)(arg + minsz),
+  data_size);
+   if (IS_ERR(data))
+   return PTR_ERR(data);
+   }
+
+   mutex_lock(>igate);
+   ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags,
+hdr.index, hdr.start,
+hdr.count, data);
+   mutex_unlock(>igate);
+   kfree(data);
+
+   return ret;
}
case VFIO_DEVICE_RESET:
{
@@ -422,6 +466,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
if (ret)
goto out_reflck;
 
+   mutex_init(>igate);
+
return 0;
 
 out_reflck:
@@ -442,6 +488,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   mutex_destroy(>igate);
+
vfio_fsl_mc_reflck_put(vdev->reflck);
 
if (is_fsl_mc_bus_dprc(mc_dev)) {
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
new file mode 100644
index ..5232f208e361
--- /dev/null
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
+/*
+ * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2019 NXP
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "linux/fsl/mc.h"
+#include "vfio_fsl_mc_private.h"
+
+static int vfio_fsl_mc_set_irq_trigger(struct vfio_fsl_mc_device *vdev,
+  unsigned int index, unsigned int start,
+  unsigned int count, u32 flags,
+  void *data)
+{
+   return -EINVAL;
+}
+
+int vfio_fsl_mc_set_irqs_ioctl(struct vfio_fsl_mc_device *vdev,
+  u32 flags, unsigned int index,
+  unsigned int start, unsigned int count,
+  void *data)
+{
+   if (flags & VFIO_IRQ_SET_ACTION_TRIGGER)
+   return  vfio_fsl_mc_set_irq_trigger(vdev, index, start,
+ count, flags, data);
+   else
+   return -EINVAL;
+}
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_

[PATCH v5 04/13] bus/fsl-mc: Set the QMAN/BMAN region flags

[Diana Craciun]

The QMAN region is memory mapped, so it should be of type
IORESOURCE_MEM. The region flags bits were wrongly used to
pass additional information. Use the bus specific bits for
this purpose.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/fsl-mc-bus.c | 7 ++-
 drivers/bus/fsl-mc/fsl-mc-private.h | 6 --
 include/linux/fsl/mc.h  | 8 
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 9a884936e53e..1bb70a92167e 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -615,11 +615,8 @@ static int fsl_mc_device_get_mmio_regions(struct 
fsl_mc_device *mc_dev,
 
regions[i].end = regions[i].start + region_desc.size - 1;
regions[i].name = "fsl-mc object MMIO region";
-   regions[i].flags = IORESOURCE_IO;
-   if (region_desc.flags & DPRC_REGION_CACHEABLE)
-   regions[i].flags |= IORESOURCE_CACHEABLE;
-   if (region_desc.flags & DPRC_REGION_SHAREABLE)
-   regions[i].flags |= IORESOURCE_MEM;
+   regions[i].flags = region_desc.flags & IORESOURCE_BITS;
+   regions[i].flags |= IORESOURCE_MEM;
}
 
mc_dev->regions = regions;
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 7a46a12eb747..9f200731b274 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -358,12 +358,6 @@ int dprc_set_obj_irq(struct fsl_mc_io *mc_io,
 int obj_id,
 u8 irq_index,
 struct dprc_irq_cfg *irq_cfg);
-
-/* Region flags */
-/* Cacheable - Indicates that region should be mapped as cacheable */
-#define DPRC_REGION_CACHEABLE  0x0001
-#define DPRC_REGION_SHAREABLE  0x0002
-
 /**
  * enum dprc_region_type - Region type
  * @DPRC_REGION_TYPE_MC_PORTAL: MC portal region
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 3b5f0c98636d..03a5d16dde73 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -3,6 +3,7 @@
  * Freescale Management Complex (MC) bus public interface
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -148,6 +149,13 @@ struct fsl_mc_obj_desc {
  */
 #define FSL_MC_IS_DPRC 0x0001
 
+/* Region flags */
+/* Indicates that region can be mapped as cacheable */
+#define FSL_MC_REGION_CACHEABLE0x0001
+
+/* Indicates that region can be mapped as shareable */
+#define FSL_MC_REGION_SHAREABLE0x0002
+
 /**
  * struct fsl_mc_device - MC object device object
  * @dev: Linux driver model device object
-- 
2.17.1

[PATCH v5 05/13] bus/fsl-mc: Cache the DPRC API version

[Diana Craciun]

There are already firmware API commands that have multiple
versions. For each multiple version command, another command
to retrieve the API version is issued. This may introduce an important
overhead. The version does not change while the system is running,
so the DPRC API version can be safely cached.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c | 30 +-
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 602f030d84eb..e76f2c76f4c8 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 /*
  * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2020 NXP
  *
  */
 #include 
@@ -8,6 +9,13 @@
 
 #include "fsl-mc-private.h"
 
+/*
+ * cache the DPRC version to reduce the number of commands
+ * towards the mc firmware
+ */
+static u16 dprc_major_ver;
+static u16 dprc_minor_ver;
+
 /**
  * dprc_open() - Open DPRC object for use
  * @mc_io: Pointer to MC portal's I/O object
@@ -443,15 +451,19 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
struct fsl_mc_command cmd = { 0 };
struct dprc_cmd_get_obj_region *cmd_params;
struct dprc_rsp_get_obj_region *rsp_params;
-   u16 major_ver, minor_ver;
int err;
 
-   /* prepare command */
-   err = dprc_get_api_version(mc_io, 0,
-_ver,
-_ver);
-   if (err)
-   return err;
+/*
+ * If the DPRC object version was not yet cached, cache it now.
+ * Otherwise use the already cached value.
+ */
+   if (!dprc_major_ver && !dprc_minor_ver) {
+   err = dprc_get_api_version(mc_io, 0,
+ _major_ver,
+ _minor_ver);
+   if (err)
+   return err;
+   }
 
/**
 * MC API version 6.3 introduced a new field to the region
@@ -459,7 +471,7 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
 * address is set to zero to indicate it needs to be obtained elsewhere
 * (typically the device tree).
 */
-   if (major_ver > 6 || (major_ver == 6 && minor_ver >= 3))
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
cmd.header =
mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
 cmd_flags, token);
@@ -483,7 +495,7 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
rsp_params = (struct dprc_rsp_get_obj_region *)cmd.params;
region_desc->base_offset = le64_to_cpu(rsp_params->base_offset);
region_desc->size = le32_to_cpu(rsp_params->size);
-   if (major_ver > 6 || (major_ver == 6 && minor_ver >= 3))
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
region_desc->base_address = le64_to_cpu(rsp_params->base_addr);
else
region_desc->base_address = 0;
-- 
2.17.1

[PATCH v5 06/13] bus/fsl-mc: Add dprc-reset-container support

[Diana Craciun]

From: Bharat Bhushan 

DPRC reset is required by VFIO-mc in order to stop a device
to further generate DMA transactions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 71 +
 drivers/bus/fsl-mc/fsl-mc-private.h |  7 +++
 include/linux/fsl/mc.h  |  7 +++
 3 files changed, 85 insertions(+)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index e76f2c76f4c8..2448a723eb28 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -80,6 +80,77 @@ int dprc_close(struct fsl_mc_io *mc_io,
 }
 EXPORT_SYMBOL_GPL(dprc_close);
 
+/**
+ * dprc_reset_container - Reset child container.
+ * @mc_io: Pointer to MC portal's I/O object
+ * @cmd_flags: Command flags; one or more of 'MC_CMD_FLAG_'
+ * @token: Token of DPRC object
+ * @child_container_id:ID of the container to reset
+ * @options: 32 bit options:
+ *   - 0 (no bits set) - all the objects inside the container are
+ * reset. The child containers are entered recursively and the
+ * objects reset. All the objects (including the child containers)
+ * are closed.
+ *   - bit 0 set - all the objects inside the container are reset.
+ * However the child containers are not entered recursively.
+ * This option is supported for API versions >= 6.5
+ * In case a software context crashes or becomes non-responsive, the parent
+ * may wish to reset its resources container before the software context is
+ * restarted.
+ *
+ * This routine informs all objects assigned to the child container that the
+ * container is being reset, so they may perform any cleanup operations that 
are
+ * needed. All objects handles that were owned by the child container shall be
+ * closed.
+ *
+ * Note that such request may be submitted even if the child software context
+ * has not crashed, but the resulting object cleanup operations will not be
+ * aware of that.
+ *
+ * Return: '0' on Success; Error code otherwise.
+ */
+int dprc_reset_container(struct fsl_mc_io *mc_io,
+u32 cmd_flags,
+u16 token,
+int child_container_id,
+u32 options)
+{
+   struct fsl_mc_command cmd = { 0 };
+   struct dprc_cmd_reset_container *cmd_params;
+   u32 cmdid = DPRC_CMDID_RESET_CONT;
+   int err;
+
+   /*
+* If the DPRC object version was not yet cached, cache it now.
+* Otherwise use the already cached value.
+*/
+   if (!dprc_major_ver && !dprc_minor_ver) {
+   err = dprc_get_api_version(mc_io, 0,
+   _major_ver,
+   _minor_ver);
+   if (err)
+   return err;
+   }
+
+   /*
+* MC API 6.5 introduced a new field in the command used to pass
+* some flags.
+* Bit 0 indicates that the child containers are not recursively reset.
+*/
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 5))
+   cmdid = DPRC_CMDID_RESET_CONT_V2;
+
+   /* prepare command */
+   cmd.header = mc_encode_cmd_header(cmdid, cmd_flags, token);
+   cmd_params = (struct dprc_cmd_reset_container *)cmd.params;
+   cmd_params->child_container_id = cpu_to_le32(child_container_id);
+   cmd_params->options = cpu_to_le32(options);
+
+   /* send command to mc*/
+   return mc_send_command(mc_io, );
+}
+EXPORT_SYMBOL_GPL(dprc_reset_container);
+
 /**
  * dprc_set_irq() - Set IRQ information for the DPRC to trigger an interrupt.
  * @mc_io: Pointer to MC portal's I/O object
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 9f200731b274..5f7e762d517c 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -91,6 +91,8 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 #define DPRC_CMDID_GET_API_VERSION  DPRC_CMD(0xa05)
 
 #define DPRC_CMDID_GET_ATTR DPRC_CMD(0x004)
+#define DPRC_CMDID_RESET_CONT   DPRC_CMD(0x005)
+#define DPRC_CMDID_RESET_CONT_V2DPRC_CMD_V2(0x005)
 
 #define DPRC_CMDID_SET_IRQ  DPRC_CMD(0x010)
 #define DPRC_CMDID_SET_IRQ_ENABLE   DPRC_CMD(0x012)
@@ -111,6 +113,11 @@ struct dprc_cmd_open {
__le32 container_id;
 };
 
+struct dprc_cmd_reset_container {
+   __le32 child_container_id;
+   __le32 options;
+};
+
 struct dprc_cmd_set_irq {
/* cmd word 0 */
__le32 irq_val;
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 03a5d16dde73..1d8800acf21f 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -524,6 +524,13 @@ static inline bool is_fsl_mc_bus_dpdmai(const struct 
fsl_mc_device *mc_dev)
return mc_dev->dev.type == 

[PATCH v5 03/13] bus/fsl-mc: add support for 'driver_override' in the mc-bus

[Diana Craciun]

From: Bharat Bhushan 

This patch is required for vfio-fsl-mc meta driver to successfully bind
layerscape container devices for device passthrough. This patch adds
a mechanism to allow a layerscape device to specify a driver rather than
a layerscape driver provide a device match.

Example to allow a device (dprc.1) to specifically bind
with driver (vfio-fsl-mc):-
 - echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.1/driver_override
 - echo dprc.1 > /sys/bus/fsl-mc/drivers/fsl_mc_dprc/unbind
 - echo dprc.1 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---

The patch does not address the comment regarding moving driver_override
in the core code. I prefer not to tie these patches on that change and
address that separately.

 drivers/bus/fsl-mc/fsl-mc-bus.c | 54 +
 include/linux/fsl/mc.h  |  2 ++
 2 files changed, 56 insertions(+)

diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index b69794e7364d..9a884936e53e 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -3,6 +3,7 @@
  * Freescale Management Complex (MC) bus driver
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -78,6 +79,12 @@ static int fsl_mc_bus_match(struct device *dev, struct 
device_driver *drv)
struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(drv);
bool found = false;
 
+   /* When driver_override is set, only bind to the matching driver */
+   if (mc_dev->driver_override) {
+   found = !strcmp(mc_dev->driver_override, mc_drv->driver.name);
+   goto out;
+   }
+
if (!mc_drv->match_id_table)
goto out;
 
@@ -147,8 +154,52 @@ static ssize_t modalias_show(struct device *dev, struct 
device_attribute *attr,
 }
 static DEVICE_ATTR_RO(modalias);
 
+static ssize_t driver_override_store(struct device *dev,
+struct device_attribute *attr,
+const char *buf, size_t count)
+{
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   char *driver_override, *old = mc_dev->driver_override;
+   char *cp;
+
+   if (WARN_ON(dev->bus != _mc_bus_type))
+   return -EINVAL;
+
+   if (count >= (PAGE_SIZE - 1))
+   return -EINVAL;
+
+   driver_override = kstrndup(buf, count, GFP_KERNEL);
+   if (!driver_override)
+   return -ENOMEM;
+
+   cp = strchr(driver_override, '\n');
+   if (cp)
+   *cp = '\0';
+
+   if (strlen(driver_override)) {
+   mc_dev->driver_override = driver_override;
+   } else {
+   kfree(driver_override);
+   mc_dev->driver_override = NULL;
+   }
+
+   kfree(old);
+
+   return count;
+}
+
+static ssize_t driver_override_show(struct device *dev,
+   struct device_attribute *attr, char *buf)
+{
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+
+   return snprintf(buf, PAGE_SIZE, "%s\n", mc_dev->driver_override);
+}
+static DEVICE_ATTR_RW(driver_override);
+
 static struct attribute *fsl_mc_dev_attrs[] = {
_attr_modalias.attr,
+   _attr_driver_override.attr,
NULL,
 };
 
@@ -748,6 +799,9 @@ EXPORT_SYMBOL_GPL(fsl_mc_device_add);
  */
 void fsl_mc_device_remove(struct fsl_mc_device *mc_dev)
 {
+   kfree(mc_dev->driver_override);
+   mc_dev->driver_override = NULL;
+
/*
 * The device-specific remove callback will get invoked by device_del()
 */
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index a428c61ead6e..3b5f0c98636d 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -161,6 +161,7 @@ struct fsl_mc_obj_desc {
  * @regions: pointer to array of MMIO region entries
  * @irqs: pointer to array of pointers to interrupts allocated to this device
  * @resource: generic resource associated with this MC object device, if any.
+ * @driver_override: driver name to force a match
  *
  * Generic device object for MC object devices that are "attached" to a
  * MC bus.
@@ -194,6 +195,7 @@ struct fsl_mc_device {
struct fsl_mc_device_irq **irqs;
struct fsl_mc_resource *resource;
struct device_link *consumer_link;
+   char   *driver_override;
 };
 
 #define to_fsl_mc_device(_dev) \
-- 
2.17.1

[PATCH v5 11/13] bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO

[Diana Craciun]

The IRQ pool handling functions can be used by both DPRC
driver and VFIO. Adapt and export those functions.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c  |  8 +++-
 drivers/bus/fsl-mc/fsl-mc-allocator.c | 12 
 drivers/bus/fsl-mc/fsl-mc-private.h   | 10 --
 include/linux/fsl/mc.h| 11 +++
 4 files changed, 22 insertions(+), 19 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 9e4088209cfe..91dc015963a8 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -327,8 +327,8 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
}
 
if (alloc_interrupts && !mc_bus->irq_resources) {
-   error = fsl_mc_populate_irq_pool(mc_bus,
-   FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+   error = fsl_mc_populate_irq_pool(mc_bus_dev,
+FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
if (error < 0)
return error;
}
@@ -766,8 +766,6 @@ static void dprc_teardown_irq(struct fsl_mc_device *mc_dev)
 int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
-   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
-
 
/* this function should be called only for DPRCs, it
 * is an error to call it for regular objects
@@ -776,7 +774,7 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev)
return -EINVAL;
 
if (dev_get_msi_domain(_dev->dev)) {
-   fsl_mc_cleanup_irq_pool(mc_bus);
+   fsl_mc_cleanup_irq_pool(mc_dev);
dev_set_msi_domain(_dev->dev, NULL);
}
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-allocator.c 
b/drivers/bus/fsl-mc/fsl-mc-allocator.c
index cc7bb900f524..e71a6f52ea0c 100644
--- a/drivers/bus/fsl-mc/fsl-mc-allocator.c
+++ b/drivers/bus/fsl-mc/fsl-mc-allocator.c
@@ -344,7 +344,7 @@ EXPORT_SYMBOL_GPL(fsl_mc_object_free);
  * Initialize the interrupt pool associated with an fsl-mc bus.
  * It allocates a block of IRQs from the GIC-ITS.
  */
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+int fsl_mc_populate_irq_pool(struct fsl_mc_device *mc_bus_dev,
 unsigned int irq_count)
 {
unsigned int i;
@@ -352,10 +352,14 @@ int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
struct fsl_mc_device_irq *irq_resources;
struct fsl_mc_device_irq *mc_dev_irq;
int error;
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
+   /* do nothing if the IRQ pool is already populated */
+   if (mc_bus->irq_resources)
+   return 0;
+
if (irq_count == 0 ||
irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS)
return -EINVAL;
@@ -407,9 +411,9 @@ EXPORT_SYMBOL_GPL(fsl_mc_populate_irq_pool);
  * Teardown the interrupt pool associated with an fsl-mc bus.
  * It frees the IRQs that were allocated to the pool, back to the GIC-ITS.
  */
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus)
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_device *mc_bus_dev)
 {
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 5f7e762d517c..ffe709a3f0f8 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -519,11 +519,6 @@ struct dpcon_cmd_set_notification {
__le64 user_ctx;
 };
 
-/**
- * Maximum number of total IRQs that can be pre-allocated for an MC bus'
- * IRQ pool
- */
-#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS 256
 
 /**
  * struct fsl_mc_resource_pool - Pool of MC resources of a given
@@ -598,11 +593,6 @@ void fsl_mc_msi_domain_free_irqs(struct device *dev);
 
 struct irq_domain *fsl_mc_find_msi_domain(struct device *dev);
 
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
-unsigned int irq_count);
-
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
-
 int __must_check fsl_create_mc_io(struct device *dev,
  phys_addr_t mc_portal_phys_addr,
  u32 mc_portal_size,
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index e99d181ee4cd..f791fe38c251 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -542,6 +542,17 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev);
 
 int dprc_setup(struct fsl_mc_device *mc_dev);
 
+/**
+ * Maximum number of total IRQs t

[PATCH v5 00/13] bus/fsl-mc: Extend mc-bus driver functionalities in preparation for mc-bus VFIO support

[Diana Craciun]

The vfio-mc bus driver needs some additional services to be exported by the
mc-bus driver like:
- a way to reset the DPRC container
- driver_override support
- functions to setup/tear down a DPRC
- functions for allocating the pool of interrupts. In case of VFIO the
interrupts are not configured at probe time, but later by userspace
request

v4 -> v5
- dprc_celanup should not fail

v3 -> v4
- Rebased on the latest kernel.
- Exported a dprc_remove function

v2 -> v3
- Add a new version for dprc_get_obj_region
- Export the cacheability bus specific bits defines

v1 -> v2
- Remove driver_override propagation through various functions
- Cache the DPRC API version

The patches are related with "vfio/fsl-mc: VFIO support for FSL-MC
devices" patches, but the series were split because they are targeting
different subsystems. However, the mc-bus patches may suffer changes
when addressing the VFIO review comments.

The patches do not address the comment regarding moving driver_override
in the core code. I prefer not to tie these patches on that change and
address that separately.

Bharat Bhushan (3):
  bus/fsl-mc: add support for 'driver_override' in the mc-bus
  bus/fsl-mc: Add dprc-reset-container support
  bus/fsl-mc: Extend ICID size from 16bit to 32bit

Diana Craciun (10):
  bus/fsl-mc: Do no longer export the total number of irqs outside
dprc_scan_objects
  bus/fsl-mc: Add a new parameter to dprc_scan_objects function
  bus/fsl-mc: Set the QMAN/BMAN region flags
  bus/fsl-mc: Cache the DPRC API version
  bus/fsl-mc: Export dprc_scan/dprc_remove functions to be used by
multiple entities
  bus/fsl-mc: Export a cleanup function for DPRC
  bus/fsl-mc: Add a container setup function
  bus/fsl_mc: Do not rely on caller to provide non NULL mc_io
  bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO
  bus/fsl-mc: Add a new version for dprc_get_obj_region command

 drivers/bus/fsl-mc/dprc-driver.c  | 190 --
 drivers/bus/fsl-mc/dprc.c | 141 +++
 drivers/bus/fsl-mc/fsl-mc-allocator.c |  12 +-
 drivers/bus/fsl-mc/fsl-mc-bus.c   |  64 -
 drivers/bus/fsl-mc/fsl-mc-private.h   |  31 ++---
 drivers/bus/fsl-mc/mc-io.c|   7 +-
 include/linux/fsl/mc.h|  41 +-
 7 files changed, 359 insertions(+), 127 deletions(-)

-- 
2.17.1

[PATCH v5 07/13] bus/fsl-mc: Export dprc_scan/dprc_remove functions to be used by multiple entities

[Diana Craciun]

Currently the DPRC scan function is used only by the bus driver.
But the same functionality will be needed by the VFIO driver.
To support this, the dprc scan function was exported and a little
bit adjusted to fit both scenarios. Also the scan mutex initialization
is done when the bus object is created, not in dprc_probe in order
to be used by both VFIO and bus driver.
Similarily dprc_remove_devices is exported to be used by VFIO.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 26 +++---
 drivers/bus/fsl-mc/fsl-mc-bus.c  |  1 +
 include/linux/fsl/mc.h   |  7 +++
 3 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 3d850515e3e5..dde8fe965e30 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -81,9 +81,9 @@ static int __fsl_mc_device_remove(struct device *dev, void 
*data)
  * the MC by removing devices that represent MC objects that have
  * been dynamically removed in the physical DPRC.
  */
-static void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
-   struct fsl_mc_obj_desc *obj_desc_array,
-   int num_child_objects_in_mc)
+void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
+struct fsl_mc_obj_desc *obj_desc_array,
+int num_child_objects_in_mc)
 {
if (num_child_objects_in_mc != 0) {
/*
@@ -105,6 +105,7 @@ static void dprc_remove_devices(struct fsl_mc_device 
*mc_bus_dev,
  __fsl_mc_device_remove);
}
 }
+EXPORT_SYMBOL_GPL(dprc_remove_devices);
 
 static int __fsl_mc_device_match(struct device *dev, void *data)
 {
@@ -354,9 +355,10 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
  * bus driver with the actual state of the MC by adding and removing
  * devices as appropriate.
  */
-static int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
+int dprc_scan_container(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts)
 {
-   int error;
+   int error = 0;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
 
fsl_mc_init_all_resource_pools(mc_bus_dev);
@@ -365,16 +367,12 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev, true);
+   error = dprc_scan_objects(mc_bus_dev, alloc_interrupts);
mutex_unlock(_bus->scan_mutex);
-   if (error < 0) {
-   fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
-   return error;
-   }
 
-   return 0;
+   return error;
 }
-
+EXPORT_SYMBOL_GPL(dprc_scan_container);
 /**
  * dprc_irq0_handler - Regular ISR for DPRC interrupt 0
  *
@@ -683,12 +681,10 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
goto error_cleanup_open;
}
 
-   mutex_init(_bus->scan_mutex);
-
/*
 * Discover MC objects in DPRC object:
 */
-   error = dprc_scan_container(mc_dev);
+   error = dprc_scan_container(mc_dev, true);
if (error < 0)
goto error_cleanup_open;
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 1bb70a92167e..32f194814b08 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -678,6 +678,7 @@ int fsl_mc_device_add(struct fsl_mc_obj_desc *obj_desc,
if (!mc_bus)
return -ENOMEM;
 
+   mutex_init(_bus->scan_mutex);
mc_dev = _bus->mc_dev;
} else {
/*
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 1d8800acf21f..da11171bc38f 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -531,6 +531,13 @@ int dprc_reset_container(struct fsl_mc_io *mc_io,
 int child_container_id,
 u32 options);
 
+int dprc_scan_container(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts);
+
+void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
+struct fsl_mc_obj_desc *obj_desc_array,
+int num_child_objects_in_mc);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v5 09/13] bus/fsl-mc: Add a container setup function

[Diana Craciun]

Both DPRC driver and VFIO driver use the same initialization code
for the DPRC. Introduced a new function which groups this
initialization code. The function is exported and may be
used by VFIO as well.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 71 ++--
 include/linux/fsl/mc.h   |  2 +
 2 files changed, 51 insertions(+), 22 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 516119c8121a..9e4088209cfe 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -588,25 +588,24 @@ static int dprc_setup_irq(struct fsl_mc_device *mc_dev)
 }
 
 /**
- * dprc_probe - callback invoked when a DPRC is being bound to this driver
+ * dprc_setup - opens and creates a mc_io for DPRC
  *
  * @mc_dev: Pointer to fsl-mc device representing a DPRC
  *
  * It opens the physical DPRC in the MC.
- * It scans the DPRC to discover the MC objects contained in it.
- * It creates the interrupt pool for the MC bus associated with the DPRC.
- * It configures the interrupts for the DPRC device itself.
+ * It configures the DPRC portal used to communicate with MC
  */
-static int dprc_probe(struct fsl_mc_device *mc_dev)
+
+int dprc_setup(struct fsl_mc_device *mc_dev)
 {
-   int error;
-   size_t region_size;
struct device *parent_dev = mc_dev->dev.parent;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+   struct irq_domain *mc_msi_domain;
bool mc_io_created = false;
bool msi_domain_set = false;
u16 major_ver, minor_ver;
-   struct irq_domain *mc_msi_domain;
+   size_t region_size;
+   int error;
 
if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
@@ -681,35 +680,63 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
goto error_cleanup_open;
}
 
+   return 0;
+
+error_cleanup_open:
+   (void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
+
+error_cleanup_msi_domain:
+   if (msi_domain_set)
+   dev_set_msi_domain(_dev->dev, NULL);
+
+   if (mc_io_created) {
+   fsl_destroy_mc_io(mc_dev->mc_io);
+   mc_dev->mc_io = NULL;
+   }
+
+   return error;
+}
+EXPORT_SYMBOL_GPL(dprc_setup);
+
+/**
+ * dprc_probe - callback invoked when a DPRC is being bound to this driver
+ *
+ * @mc_dev: Pointer to fsl-mc device representing a DPRC
+ *
+ * It opens the physical DPRC in the MC.
+ * It scans the DPRC to discover the MC objects contained in it.
+ * It creates the interrupt pool for the MC bus associated with the DPRC.
+ * It configures the interrupts for the DPRC device itself.
+ */
+static int dprc_probe(struct fsl_mc_device *mc_dev)
+{
+   int error;
+
+   error = dprc_setup(mc_dev);
+   if (error < 0)
+   return error;
+
/*
 * Discover MC objects in DPRC object:
 */
error = dprc_scan_container(mc_dev, true);
if (error < 0)
-   goto error_cleanup_open;
+   goto dprc_cleanup;
 
/*
 * Configure interrupt for the DPRC object associated with this MC bus:
 */
error = dprc_setup_irq(mc_dev);
if (error < 0)
-   goto error_cleanup_open;
+   goto scan_cleanup;
 
dev_info(_dev->dev, "DPRC device bound to driver");
return 0;
 
-error_cleanup_open:
-   (void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
-
-error_cleanup_msi_domain:
-   if (msi_domain_set)
-   dev_set_msi_domain(_dev->dev, NULL);
-
-   if (mc_io_created) {
-   fsl_destroy_mc_io(mc_dev->mc_io);
-   mc_dev->mc_io = NULL;
-   }
-
+scan_cleanup:
+   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
+dprc_cleanup:
+   dprc_cleanup(mc_dev);
return error;
 }
 
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 5519a510b8c9..e99d181ee4cd 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -540,6 +540,8 @@ void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
 
 int dprc_cleanup(struct fsl_mc_device *mc_dev);
 
+int dprc_setup(struct fsl_mc_device *mc_dev);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v5 08/13] bus/fsl-mc: Export a cleanup function for DPRC

[Diana Craciun]

Create and export a cleanup function for DPRC. The function
is used by the DPRC driver, but it will be used by the VFIO
driver as well.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 62 
 include/linux/fsl/mc.h   |  2 ++
 2 files changed, 49 insertions(+), 15 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index dde8fe965e30..516119c8121a 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -728,33 +728,26 @@ static void dprc_teardown_irq(struct fsl_mc_device 
*mc_dev)
 }
 
 /**
- * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ * dprc_cleanup - function that cleanups a DPRC
  *
  * @mc_dev: Pointer to fsl-mc device representing the DPRC
  *
- * It removes the DPRC's child objects from Linux (not from the MC) and
- * closes the DPRC device in the MC.
- * It tears down the interrupts that were configured for the DPRC device.
+ * It closes the DPRC device in the MC.
  * It destroys the interrupt pool associated with this MC bus.
  */
-static int dprc_remove(struct fsl_mc_device *mc_dev)
+
+int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
-   if (!is_fsl_mc_bus_dprc(mc_dev))
-   return -EINVAL;
-   if (!mc_dev->mc_io)
-   return -EINVAL;
 
-   if (!mc_bus->irq_resources)
+   /* this function should be called only for DPRCs, it
+* is an error to call it for regular objects
+*/
+   if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
 
-   if (dev_get_msi_domain(_dev->dev))
-   dprc_teardown_irq(mc_dev);
-
-   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
-
if (dev_get_msi_domain(_dev->dev)) {
fsl_mc_cleanup_irq_pool(mc_bus);
dev_set_msi_domain(_dev->dev, NULL);
@@ -762,6 +755,14 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
 
fsl_mc_cleanup_all_resource_pools(mc_dev);
 
+   /* if this step fails we cannot go further with cleanup as there is no 
way of
+* communicating with the firmware
+*/
+   if (!mc_dev->mc_io) {
+   dev_err(_dev->dev, "mc_io is NULL, tear down cannot be 
performed in firmware\n");
+   return -EINVAL;
+   }
+
error = dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
if (error < 0)
dev_err(_dev->dev, "dprc_close() failed: %d\n", error);
@@ -771,6 +772,37 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
mc_dev->mc_io = NULL;
}
 
+   return 0;
+}
+EXPORT_SYMBOL_GPL(dprc_cleanup);
+
+/**
+ * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ *
+ * @mc_dev: Pointer to fsl-mc device representing the DPRC
+ *
+ * It removes the DPRC's child objects from Linux (not from the MC) and
+ * closes the DPRC device in the MC.
+ * It tears down the interrupts that were configured for the DPRC device.
+ * It destroys the interrupt pool associated with this MC bus.
+ */
+static int dprc_remove(struct fsl_mc_device *mc_dev)
+{
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -EINVAL;
+
+   if (!mc_bus->irq_resources)
+   return -EINVAL;
+
+   if (dev_get_msi_domain(_dev->dev))
+   dprc_teardown_irq(mc_dev);
+
+   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
+
+   dprc_cleanup(mc_dev);
+
dev_info(_dev->dev, "DPRC device unbound from driver");
return 0;
 }
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index da11171bc38f..5519a510b8c9 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -538,6 +538,8 @@ void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
 struct fsl_mc_obj_desc *obj_desc_array,
 int num_child_objects_in_mc);
 
+int dprc_cleanup(struct fsl_mc_device *mc_dev);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v5 13/13] bus/fsl-mc: Add a new version for dprc_get_obj_region command

[Diana Craciun]

The region size reported by the firmware for mc and software
portals was less than allocated by the hardware. This may be
problematic when mmapping the region in user space because the
region size is less than page size. However the size as reserved
by the hardware is 64K.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 38 ++---
 drivers/bus/fsl-mc/fsl-mc-private.h |  3 +++
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 650808799a29..57b097caf255 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -536,20 +536,30 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
return err;
}
 
-   /**
-* MC API version 6.3 introduced a new field to the region
-* descriptor: base_address. If the older API is in use then the base
-* address is set to zero to indicate it needs to be obtained elsewhere
-* (typically the device tree).
-*/
-   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
-cmd_flags, token);
-   else
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
-cmd_flags, token);
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 6)) 
{
+   /*
+* MC API version 6.6 changed the size of the MC portals and 
software
+* portals to 64K (as implemented by hardware). If older API is 
in use the
+* size reported is less (64 bytes for mc portals and 4K for 
software
+* portals).
+*/
+
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V3,
+ cmd_flags, token);
+
+   } else if (dprc_major_ver == 6 && dprc_minor_ver >= 3) {
+   /*
+* MC API version 6.3 introduced a new field to the region
+* descriptor: base_address. If the older API is in use then 
the base
+* address is set to zero to indicate it needs to be obtained 
elsewhere
+* (typically the device tree).
+*/
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
+ cmd_flags, token);
+   } else {
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
+ cmd_flags, token);
+   }
 
cmd_params = (struct dprc_cmd_get_obj_region *)cmd.params;
cmd_params->obj_id = cpu_to_le32(obj_id);
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 3e9f9c778631..85ca5fdee581 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -80,10 +80,12 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 /* DPRC command versioning */
 #define DPRC_CMD_BASE_VERSION  1
 #define DPRC_CMD_2ND_VERSION   2
+#define DPRC_CMD_3RD_VERSION   3
 #define DPRC_CMD_ID_OFFSET 4
 
 #define DPRC_CMD(id)   (((id) << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)
 #define DPRC_CMD_V2(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_2ND_VERSION)
+#define DPRC_CMD_V3(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_3RD_VERSION)
 
 /* DPRC command IDs */
 #define DPRC_CMDID_CLOSEDPRC_CMD(0x800)
@@ -105,6 +107,7 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 #define DPRC_CMDID_GET_OBJ  DPRC_CMD(0x15A)
 #define DPRC_CMDID_GET_OBJ_REG  DPRC_CMD(0x15E)
 #define DPRC_CMDID_GET_OBJ_REG_V2   DPRC_CMD_V2(0x15E)
+#define DPRC_CMDID_GET_OBJ_REG_V3   DPRC_CMD_V3(0x15E)
 #define DPRC_CMDID_SET_OBJ_IRQ  DPRC_CMD(0x15F)
 
 #define DPRC_CMDID_GET_CONNECTION   DPRC_CMD(0x16C)
-- 
2.17.1

[PATCH v5 10/13] bus/fsl_mc: Do not rely on caller to provide non NULL mc_io

[Diana Craciun]

Before destroying the mc_io, check first that it was
allocated.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/mc-io.c | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/fsl-mc/mc-io.c b/drivers/bus/fsl-mc/mc-io.c
index a30b53f1d87d..305015486b91 100644
--- a/drivers/bus/fsl-mc/mc-io.c
+++ b/drivers/bus/fsl-mc/mc-io.c
@@ -129,7 +129,12 @@ int __must_check fsl_create_mc_io(struct device *dev,
  */
 void fsl_destroy_mc_io(struct fsl_mc_io *mc_io)
 {
-   struct fsl_mc_device *dpmcp_dev = mc_io->dpmcp_dev;
+   struct fsl_mc_device *dpmcp_dev;
+
+   if (!mc_io)
+   return;
+
+   dpmcp_dev = mc_io->dpmcp_dev;
 
if (dpmcp_dev)
fsl_mc_io_unset_dpmcp(mc_io);
-- 
2.17.1

[PATCH v5 12/13] bus/fsl-mc: Extend ICID size from 16bit to 32bit

[Diana Craciun]

From: Bharat Bhushan 

In virtual machines the device-id range is defined
between 0x1-0x2. The reason for using such a
large range is to avoid overlapping with the PCI range.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 2 +-
 drivers/bus/fsl-mc/fsl-mc-bus.c | 2 +-
 drivers/bus/fsl-mc/fsl-mc-private.h | 5 ++---
 include/linux/fsl/mc.h  | 2 +-
 4 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 2448a723eb28..650808799a29 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -360,7 +360,7 @@ int dprc_get_attributes(struct fsl_mc_io *mc_io,
/* retrieve response parameters */
rsp_params = (struct dprc_rsp_get_attributes *)cmd.params;
attr->container_id = le32_to_cpu(rsp_params->container_id);
-   attr->icid = le16_to_cpu(rsp_params->icid);
+   attr->icid = le32_to_cpu(rsp_params->icid);
attr->options = le32_to_cpu(rsp_params->options);
attr->portal_id = le32_to_cpu(rsp_params->portal_id);
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 32f194814b08..d375bb585749 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -503,7 +503,7 @@ static int get_dprc_attr(struct fsl_mc_io *mc_io,
 }
 
 static int get_dprc_icid(struct fsl_mc_io *mc_io,
-int container_id, u16 *icid)
+int container_id, u32 *icid)
 {
struct dprc_attributes attr;
int error;
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index ffe709a3f0f8..3e9f9c778631 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -159,8 +159,7 @@ struct dprc_cmd_clear_irq_status {
 struct dprc_rsp_get_attributes {
/* response word 0 */
__le32 container_id;
-   __le16 icid;
-   __le16 pad;
+   __le32 icid;
/* response word 1 */
__le32 options;
__le32 portal_id;
@@ -337,7 +336,7 @@ int dprc_clear_irq_status(struct fsl_mc_io *mc_io,
  */
 struct dprc_attributes {
int container_id;
-   u16 icid;
+   u32 icid;
int portal_id;
u64 options;
 };
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index f791fe38c251..db244874e834 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -195,7 +195,7 @@ struct fsl_mc_device {
struct device dev;
u64 dma_mask;
u16 flags;
-   u16 icid;
+   u32 icid;
u16 mc_handle;
struct fsl_mc_io *mc_io;
struct fsl_mc_obj_desc obj_desc;
-- 
2.17.1

[PATCH v5 02/13] bus/fsl-mc: Add a new parameter to dprc_scan_objects function

[Diana Craciun]

Prepare the dprc_scan_objects function to be used by
the VFIO mc driver code. The function is used to scan the mc
objects by the bus driver. The same functionality is
needed by the VFIO mc driver, but in this case the
interrupt configuration is delayed until the userspace
configures the interrupts. In order to use the same function
in both drivers add a new parameter.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 54c576d68122..3d850515e3e5 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -221,6 +221,8 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
+ * @alloc_interrupts: if true the function allocates the interrupt pool,
+ * otherwise the interrupt allocation is delayed
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -234,7 +236,8 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
+static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts)
 {
int num_child_objects;
int dprc_get_obj_failures;
@@ -322,7 +325,7 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev)
 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
}
 
-   if (!mc_bus->irq_resources) {
+   if (alloc_interrupts && !mc_bus->irq_resources) {
error = fsl_mc_populate_irq_pool(mc_bus,
FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
if (error < 0)
@@ -362,7 +365,7 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev);
+   error = dprc_scan_objects(mc_bus_dev, true);
mutex_unlock(_bus->scan_mutex);
if (error < 0) {
fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
@@ -432,7 +435,7 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
  DPRC_IRQ_EVENT_OBJ_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_CREATED)) {
 
-   error = dprc_scan_objects(mc_dev);
+   error = dprc_scan_objects(mc_dev, true);
if (error < 0) {
/*
 * If the error is -ENXIO, we ignore it, as it indicates
-- 
2.17.1

[PATCH v5 01/13] bus/fsl-mc: Do no longer export the total number of irqs outside dprc_scan_objects

[Diana Craciun]

The total number of interrupts is only used for some checks
outside the dprc_scan_objects function. Furthermore, in some
situations the check is made twice. Move the bounds check inside
the function for all situations.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 30 ++
 1 file changed, 10 insertions(+), 20 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 2a473c09bc33..54c576d68122 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -3,6 +3,7 @@
  * Freescale data path resource container (DPRC) driver
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -220,8 +221,6 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
- * @total_irq_count: If argument is provided the function populates the
- * total number of IRQs created by objects in the DPRC.
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -235,8 +234,7 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
-unsigned int *total_irq_count)
+static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 {
int num_child_objects;
int dprc_get_obj_failures;
@@ -317,22 +315,21 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
 * Allocate IRQ's before binding the scanned devices with their
 * respective drivers.
 */
-   if (dev_get_msi_domain(_bus_dev->dev) && !mc_bus->irq_resources) {
+   if (dev_get_msi_domain(_bus_dev->dev)) {
if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
dev_warn(_bus_dev->dev,
 "IRQs needed (%u) exceed IRQs preallocated 
(%u)\n",
 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
}
 
-   error = fsl_mc_populate_irq_pool(mc_bus,
+   if (!mc_bus->irq_resources) {
+   error = fsl_mc_populate_irq_pool(mc_bus,
FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
-   if (error < 0)
-   return error;
+   if (error < 0)
+   return error;
+   }
}
 
-   if (total_irq_count)
-   *total_irq_count = irq_count;
-
dprc_remove_devices(mc_bus_dev, child_obj_desc_array,
num_child_objects);
 
@@ -365,7 +362,7 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev, NULL);
+   error = dprc_scan_objects(mc_bus_dev);
mutex_unlock(_bus->scan_mutex);
if (error < 0) {
fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
@@ -434,9 +431,8 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
  DPRC_IRQ_EVENT_CONTAINER_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_CREATED)) {
-   unsigned int irq_count;
 
-   error = dprc_scan_objects(mc_dev, _count);
+   error = dprc_scan_objects(mc_dev);
if (error < 0) {
/*
 * If the error is -ENXIO, we ignore it, as it indicates
@@ -451,12 +447,6 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
 
goto out;
}
-
-   if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
-   dev_warn(dev,
-"IRQs needed (%u) exceed IRQs preallocated 
(%u)\n",
-irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
-   }
}
 
 out:
-- 
2.17.1

Re: [PATCH v4 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun OSS]

Hi Eric,

On 9/10/2020 11:20 AM, Auger Eric wrote:

Hi Diana,

On 9/7/20 4:34 PM, Diana Craciun OSS wrote:

Hi Eric,

On 9/4/2020 11:18 AM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

What I don't get is all the regions are mmappable too.
And this patch does not seem to introduce special handling with respect
to MSIs. Please could you clarify?


The device can be controlled using commands issued towards a firmware.
Most of the commands can be passthrough, among exceptions is the command
that configures the interrupts. In a guest the interrupts are emulated
and for the hardware the numbers in the guest mean nothing. So, in a
virtual machine scenario, the DPMCP and DPRC regions are emulated in
qemu such that the command which configures the interrupts will not go
to hardware with the information set by the guest.
However there are other scenarios apart from virtual machines like DPDK
in which the interrupt configuration command is not used. The problem
might be that the userspace could issue the command because there is no
restriction in the VFIO, but in that case the worst thing that may
happen is for the interrupts for the device not to work.
However it is possible to restrict the command for this scenario as well
if I change the code and not allow the DPRC region to be mmapable. In
practice it proved that it might not gain much by direct assigning that
area. Also the interrupt configuration command was restricted from the
firmware to be issued only from the DPRC device region to help such a
scenario.

Yes actually I meant that the region used to configure MSIs should not
be mmappable then?




That region is not used only for the MSI configuration. The way it works 
is through commands that are written at a certain memory address. And 
the commands can be different. Applications like DPDK do not use the 
command to configure interrupts, so that is the reason that the region 
is mmapable. But at a second thought, I think that I can restrict the 
DPRC region not to be mmapable. And that is the only region that can be 
used to configure interrupts.


Thanks,

Diana


Thanks

Eric





This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
   drivers/vfio/fsl-mc/vfio_fsl_mc.c | 115 +-
   drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
   2 files changed, 114 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 73834f488a94..27713aa86878 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
   #include 
   #include 
   #include 
+#include 
     #include "vfio_fsl_mc_private.h"
   @@ -106,6 +107,9 @@ static int vfio_fsl_mc_regions_init(struct
vfio_fsl_mc_device *vdev)
   vdev->regions[i].size = resource_size(res);
   vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
   vdev->regions[i].type = mc_dev->regions[i].flags &
IORESOURCE_BITS;
+    vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+    if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+    vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
   }
     vdev->num_regions = mc_dev->obj_desc.region_count;
@@ -114,6 +118,11 @@ static int vfio_fsl_mc_regions_init(struct
vfio_fsl_mc_device *vdev)
     static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device
*vdev)
   {
+    int i;
+
+    for (i = 0; i < vdev->num_regions; i++)
+    iounmap(vdev->regions[i].ioaddr);
+
   vdev->num_regions = 0;
   kfree(vdev->regions);
   }
@@ -311,13 +320,115 @@ static long vfio_fsl_mc_ioctl(void
*device_data, unsigned int cmd,
   static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
   size_t count, loff_t *ppos)
   {
-    return -EINVAL;
+    struct vfio_fsl_mc_device *vdev = device_data;
+    unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+    loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+    struct vfio_fsl_mc_region *region;
+    u64 data[8];
+    int i;
+
+    if (index >= vdev->num_regions)
+    return -EINVAL;
+
+    region = >regions[index];
+
+    if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+    return -EINVAL;
+
+    if 

Re: [PATCH v4 07/10] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices

[Diana Craciun OSS]

Hi Eric,

On 9/3/2020 11:15 PM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

This patch adds the skeleton for interrupt support
for fsl-mc devices. The interrupts are not yet functional,
the functionality will be added by subsequent patches.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/Makefile  |  2 +-
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 75 ++-
  drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 63 +++
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  7 ++-
  4 files changed, 143 insertions(+), 4 deletions(-)
  create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
index 0c6e5d2ddaae..cad6dbf0b735 100644
--- a/drivers/vfio/fsl-mc/Makefile
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -1,4 +1,4 @@
  # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
  
-vfio-fsl-mc-y := vfio_fsl_mc.o

+vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o
  obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index bbd3365e877e..42014297b484 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -209,11 +209,79 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
-   return -ENOTTY;
+   struct vfio_irq_info info;
+
+   minsz = offsetofend(struct vfio_irq_info, count);
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   info.flags = VFIO_IRQ_INFO_EVENTFD;

shouldn't it be MASKABLE as well? I see skeletons for MASK.


The skeletons for mask are not implemented. Maybe I should just remove 
the skeletons.



+   info.count = 1;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_SET_IRQS:
{
-   return -ENOTTY;
+   struct vfio_irq_set hdr;
+   u8 *data = NULL;
+   int ret = 0;
+
+   minsz = offsetofend(struct vfio_irq_set, count);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (hdr.argsz < minsz)
+   return -EINVAL;
+
+   if (hdr.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   if (hdr.start != 0 || hdr.count > 1)
+   return -EINVAL;
+
+   if (hdr.count == 0 &&
+   (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE) ||
+   !(hdr.flags & VFIO_IRQ_SET_ACTION_TRIGGER)))
+   return -EINVAL;
+
+   if (hdr.flags & ~(VFIO_IRQ_SET_DATA_TYPE_MASK |
+ VFIO_IRQ_SET_ACTION_TYPE_MASK))
+   return -EINVAL;
+
+   if (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE)) {
+   size_t size;
+
+   if (hdr.flags & VFIO_IRQ_SET_DATA_BOOL)
+   size = sizeof(uint8_t);
+   else if (hdr.flags & VFIO_IRQ_SET_DATA_EVENTFD)
+   size = sizeof(int32_t);
+   else
+   return -EINVAL;
+
+   if (hdr.argsz - minsz < hdr.count * size)
+   return -EINVAL;
+
+   data = memdup_user((void __user *)(arg + minsz),
+  hdr.count * size);
+   if (IS_ERR(data))
+   return PTR_ERR(data);
+   }

can't you reuse vfio_set_irqs_validate_and_prepare()?


Yes, I think I can reuse it.


+
+   mutex_lock(>igate);
+   ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags,
+hdr.index, hdr.start,
+hdr.count, data);
+   mutex_unlock(>igate);
+   kfree(data);
+
+   return ret;
}
case VFIO_DEVICE_RESET:
{
@@ -413,6 +481,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
  
+	mutex_init(>igate);

+
return ret;
  }
  
@@ -436,6 +506,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)

mc_dev->mc_io = NULL;
  
  	vfio_fsl_mc_reflck_put(vdev->reflck);

+   mutex_destroy(>igate);
  
  	vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
  
diff --git a/drivers/vfio

Re: [PATCH v4 08/10] vfio/fsl-mc: trigger an interrupt via eventfd

[Diana Craciun OSS]

Hi Eric,

On 9/4/2020 11:02 AM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

This patch allows to set an eventfd for fsl-mc device interrupts
and also to trigger the interrupt eventfd from userspace for testing.

All fsl-mc device interrupts are MSIs. The MSIs are allocated from
the MSI domain only once per DPRC and used by all the DPAA2 objects.
The interrupts are managed by the DPRC in a pool of interrupts. Each
device requests interrupts from this pool. The pool is allocated
when the first virtual device is setting the interrupts.
The pool of interrupts is protected by a lock.

The DPRC has an interrupt of its own which indicates if the DPRC
contents have changed. However, currently, the contents of a DPRC
assigned to the guest cannot be changed at runtime, so this interrupt
is not configured.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c |  18 ++-
  drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 160 +-
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  10 ++
  3 files changed, 186 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 42014297b484..73834f488a94 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -147,12 +147,28 @@ static int vfio_fsl_mc_open(void *device_data)
  static void vfio_fsl_mc_release(void *device_data)
  {
struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
  
  	mutex_lock(>reflck->lock);
  
-	if (!(--vdev->refcnt))

+   if (!(--vdev->refcnt)) {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
+
vfio_fsl_mc_regions_cleanup(vdev);
  
+		/* reset the device before cleaning up the interrupts */

+   ret = dprc_reset_container(mc_cont->mc_io, 0,
+ mc_cont->mc_handle,
+ mc_cont->obj_desc.id,
+ DPRC_RESET_OPTION_NON_RECURSIVE);

shouldn't you test ret?

+
+   vfio_fsl_mc_irqs_cleanup(vdev);
+
+   fsl_mc_cleanup_irq_pool(mc_cont);
+   }
+
mutex_unlock(>reflck->lock);
  
  	module_put(THIS_MODULE);

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 058aa97aa54a..409f3507fcf3 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -29,12 +29,149 @@ static int vfio_fsl_mc_irq_unmask(struct 
vfio_fsl_mc_device *vdev,
return -EINVAL;
  }
  
+int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)

+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_irq *mc_irq;
+   int irq_count;
+   int ret, i;
+
+/* Device does not support any interrupt */

indent needs to be fixed

+   if (mc_dev->obj_desc.irq_count == 0)
+   return 0;
+
+   /* interrupts were already allocated for this device */
+   if (vdev->mc_irqs)
+   return 0;
+
+   irq_count = mc_dev->obj_desc.irq_count;
+
+   mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
+   if (!mc_irq)
+   return -ENOMEM;
+
+   /* Allocate IRQs */
+   ret = fsl_mc_allocate_irqs(mc_dev);
+   if (ret) {
+   kfree(mc_irq);
+   return ret;
+   }
+
+   for (i = 0; i < irq_count; i++) {
+   mc_irq[i].count = 1;
+   mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
+   }
+
+   vdev->mc_irqs = mc_irq;
+
+   return 0;
+}
+
+static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg)
+{
+   struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg;
+
+   eventfd_signal(mc_irq->trigger, 1);
+   return IRQ_HANDLED;
+}
+
+static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev,
+  int index, int fd)
+{
+   struct vfio_fsl_mc_irq *irq = >mc_irqs[index];
+   struct eventfd_ctx *trigger;
+   int hwirq;
+   int ret;
+
+   hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq;
+   if (irq->trigger) {
+   free_irq(hwirq, irq);
+   kfree(irq->name);
+   eventfd_ctx_put(irq->trigger);
+   irq->trigger = NULL;
+   }
+
+   if (fd < 0) /* Disable only */
+   return 0;
+
+   irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)",
+   hwirq, dev_name(>mc_dev->dev));
+   if (!irq->name)
+   return -ENOMEM;
+
+   trigger = eventfd_ctx_fdget(fd);
+   if (IS_ERR(trigger)) {
+   kfree(irq->name);
+   return PTR_ERR(trigger);
+   }
+
+  

Re: [PATCH v4 10/10] vfio/fsl-mc: Add support for device reset

[Diana Craciun OSS]

Hi Eric,

On 9/4/2020 11:21 AM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

Currently only resetting the DPRC container is supported which
will reset all the objects inside it. Resetting individual
objects is possible from the userspace by issueing commands
towards MC firmware.

Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 15 ++-
  1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 27713aa86878..d17c5b3148ad 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -310,7 +310,20 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_RESET:
{
-   return -ENOTTY;
+   int ret = 0;

initialization not needed

+

spare empty line

+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
+   /* reset is supported only for the DPRC */
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -ENOTTY;

it is an error case or do we just don't care?



I rather don't care, but shouldn't the userspace know that the reset for 
that device failed?



+
+   ret = dprc_reset_container(mc_dev->mc_io, 0,
+  mc_dev->mc_handle,
+  mc_dev->obj_desc.id,
+  DPRC_RESET_OPTION_NON_RECURSIVE);
+   return ret;
+
}
default:
return -ENOTTY;


Thanks

Eric



Thanks,
Diana

Re: [PATCH v4 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun OSS]

Hi Eric,

On 9/3/2020 5:06 PM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 84 +++
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  1 +
  2 files changed, 85 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 8b53c2a25b32..85e007be3a5d 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
  
  #include "vfio_fsl_mc_private.h"
  
+static struct fsl_mc_driver vfio_fsl_mc_driver;

+
  static int vfio_fsl_mc_open(void *device_data)
  {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,72 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
  };
  
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,

+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   if (!mc_dev->driver_override)
+   dev_warn(dev, "Setting driver override for device in dprc %s 
failed\n",
+dev_name(_cont->dev));
+   dev_info(dev, "Setting driver override for device in dprc %s\n",
+dev_name(_cont->dev));

Don't you miss an else here?

+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "Object %s bound to driver %s while DPRC bound 
to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret < 0) {

if (ret) here and in other places? or are there any > returned values

+   dev_err(_dev->dev, "Failed to setup DPRC (error = %d)\n", 
ret);

nit: maybe align your error messages. Before you were using __func__,
here you don't. Maybe don't? also you may consider using strerror(-ret)

+   bus_unregister_notifier(_mc_bus_type, >nb);
+   return ret;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret < 0) {
+   dev_err(_dev->dev, "Container scanning failed: %d\n", ret);
+   dprc_cleanup(mc_dev);

I see dprc_cleanup is likely to fail. Generally cleanup shouldn't.
https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg2283433.html


Right, I will change the dprc_cleanup not to fail.



+   bus_unregister_notifier(_mc_bus_type, >nb);

nit: here also you can factorize code doing goto unregister;
shouldn't you reset vdev->nb.notifier_call to NULL as well. I see it is
tested in other places.

+   }
+
+   return ret;
+}
+
  static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
  {
struct iommu_group *group;
@@ -112,6 +180,12 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
  
+	ret = vfio_fsl_mc_init_device(vdev);

+   if (ret < 0) {

I think you also need to call vfio_del_group_dev(>dev)

+   vfio_iommu_group_put(group, dev);
+   return ret;

nit: goto put_group;

+   }
+
return ret;
  }
  
@@ -124,6 +198,16 @@ static int vfio_fsl_mc_remove(stru

Re: [PATCH v4 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun OSS]

Hi Eric,

On 9/4/2020 11:18 AM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

What I don't get is all the regions are mmappable too.
And this patch does not seem to introduce special handling with respect
to MSIs. Please could you clarify?


The device can be controlled using commands issued towards a firmware. 
Most of the commands can be passthrough, among exceptions is the command 
that configures the interrupts. In a guest the interrupts are emulated 
and for the hardware the numbers in the guest mean nothing. So, in a 
virtual machine scenario, the DPMCP and DPRC regions are emulated in 
qemu such that the command which configures the interrupts will not go 
to hardware with the information set by the guest.
However there are other scenarios apart from virtual machines like DPDK 
in which the interrupt configuration command is not used. The problem 
might be that the userspace could issue the command because there is no 
restriction in the VFIO, but in that case the worst thing that may 
happen is for the interrupts for the device not to work.
However it is possible to restrict the command for this scenario as well 
if I change the code and not allow the DPRC region to be mmapable. In 
practice it proved that it might not gain much by direct assigning that 
area. Also the interrupt configuration command was restricted from the 
firmware to be issued only from the DPRC device region to help such a 
scenario.





This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 115 +-
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
  2 files changed, 114 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 73834f488a94..27713aa86878 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
  #include 
  #include 
  #include 
+#include 
  
  #include "vfio_fsl_mc_private.h"
  
@@ -106,6 +107,9 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)

vdev->regions[i].size = resource_size(res);
vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
  
  	vdev->num_regions = mc_dev->obj_desc.region_count;

@@ -114,6 +118,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
  
  static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)

  {
+   int i;
+
+   for (i = 0; i < vdev->num_regions; i++)
+   iounmap(vdev->regions[i].ioaddr);
+
vdev->num_regions = 0;
kfree(vdev->regions);
  }
@@ -311,13 +320,115 @@ static long vfio_fsl_mc_ioctl(void *device_data, 
unsigned int cmd,
  static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
  {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;
+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS  

Re: [PATCH v4 05/10] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions

[Diana Craciun OSS]

Hi Eric,

On 9/3/2020 7:05 PM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

Allow userspace to mmap device regions for direct access of
fsl-mc devices.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 60 +--
  1 file changed, 58 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 093b8d68496c..64d5c1fff51f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,7 +33,8 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device 
*vdev)
  
  		vdev->regions[i].addr = res->start;

vdev->regions[i].size = resource_size(res);
-   vdev->regions[i].flags = 0;
+   vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;

Is the region always mmappable or does it depend on the
mc_dev->regions[i].flags. Also on VFIO platform we checked some
alignment addr/size constraints.


The region is mmappable regardless of the region flags. However, I see 
that there are some questions regarding the fact that the regions are 
always mmappable in the following patches, so I'll try to clarify here.


The way the userspace communicates with the hardware is through some 
commands (special data written in the device region). The commands can 
be issued using special channels (devices): dprc and dpmcp. Most of the 
commands can be passthrough, the command that configures the interrupts 
is the most important example. In order to reduce the complexity, the 
command which configures the interrupts was restricted from the firmware 
to be issued only on a single type of device (dprc). However, in the 
current implementation the memory region corresponding to the dpcr can 
be passthorugh as well. The reason is that it can be used (for example) 
by a DPDK application in the userspace. The DPDK application configures 
the interrupts using the VFIO_DEVICE_SET_IRQS system call. When it comes 
to virtual machines the dprc and the dpmc will be emulated in QEMU.


Regarding the alignemnet/size constraints I agree, I will add some checks.


+   vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
}
  
  	vdev->num_regions = mc_dev->obj_desc.region_count;

@@ -164,9 +165,64 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const 
char __user *buf,
return -EINVAL;
  }
  
+static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,

+struct vm_area_struct *vma)
+{
+   u64 size = vma->vm_end - vma->vm_start;
+   u64 pgoff, base;
+   u8 region_cacheable;
+
+   pgoff = vma->vm_pgoff &
+   ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+   base = pgoff << PAGE_SHIFT;
+
+   if (region.size < PAGE_SIZE || base + size > region.size)
+   return -EINVAL;
+
+   region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) &&
+  (region.type & FSL_MC_REGION_SHAREABLE);

I see in fsl-mc-bus.c that IORESOURCE_CACHEABLE and IORESOURCE_MEM are
set on the regions flag?


Yes, initially the two flags were set (IORESOURCE_CACHEABLE and 
IORESOURCE_MEM). However, I have changed the behaviour a little bit (it 
was wrong in the past) and IORESOURCE_MEM is still present, but also 
FSL_MC_REGION_CACHEABLE and FSL_MC_REGION_SHAREABLE which are retrieved 
from the firmware through commands.



+   if (!region_cacheable)
+   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+   vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff;
+
+   return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+  size, vma->vm_page_prot);
+}
+
  static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
  {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int index;
+
+   index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
+
+   if (vma->vm_end < vma->vm_start)
+   return -EINVAL;
+   if (vma->vm_start & ~PAGE_MASK)
+   return -EINVAL;
+   if (vma->vm_end & ~PAGE_MASK)
+   return -EINVAL;
+   if (!(vma->vm_flags & VM_SHARED))
+   return -EINVAL;
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ)
+   && (vma->vm_flags & VM_READ))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION

Re: [PATCH v4 01/10] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

[Diana Craciun OSS]

Hi Eric,

On 9/3/2020 5:06 PM, Auger Eric wrote:

Hi Diana,

On 8/26/20 11:33 AM, Diana Craciun wrote:

From: Bharat Bhushan 

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

This patch adds the infrastructure for VFIO support for fsl-mc
devices. Subsequent patches will add support for binding and secure
assigning these devices using VFIO.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
  MAINTAINERS   |   6 +
  drivers/vfio/Kconfig  |   1 +
  drivers/vfio/Makefile |   1 +
  drivers/vfio/fsl-mc/Kconfig   |   9 ++
  drivers/vfio/fsl-mc/Makefile  |   4 +
  drivers/vfio/fsl-mc/vfio_fsl_mc.c | 160 ++
  drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  14 ++
  include/uapi/linux/vfio.h |   1 +
  8 files changed, 196 insertions(+)
  create mode 100644 drivers/vfio/fsl-mc/Kconfig
  create mode 100644 drivers/vfio/fsl-mc/Makefile
  create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
  create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 3b186ade3597..f3f9ea108588 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18229,6 +18229,12 @@ F: drivers/vfio/
  F:include/linux/vfio.h
  F:include/uapi/linux/vfio.h
  
+VFIO FSL-MC DRIVER

+M: Diana Craciun 
+L: k...@vger.kernel.org
+S: Maintained
+F: drivers/vfio/fsl-mc/
+
  VFIO MEDIATED DEVICE DRIVERS
  M:Kirti Wankhede 
  L:k...@vger.kernel.org
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index fd17db9b432f..5533df91b257 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
  source "drivers/vfio/pci/Kconfig"
  source "drivers/vfio/platform/Kconfig"
  source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/fsl-mc/Kconfig"
  source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..fee73f3d9480 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
  obj-$(CONFIG_VFIO_PCI) += pci/
  obj-$(CONFIG_VFIO_PLATFORM) += platform/
  obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/
diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig
new file mode 100644
index ..b1a527d6b6f2
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Kconfig
@@ -0,0 +1,9 @@
+config VFIO_FSL_MC
+   tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices"
+   depends on VFIO && FSL_MC_BUS && EVENTFD
+   help
+ Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc
+ (Management Complex) devices. This is required to passthrough
+ fsl-mc bus devices using the VFIO framework.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
new file mode 100644

Re: [PATCH v4 00/10] vfio/fsl-mc: VFIO support for FSL-MC device

[Diana Craciun OSS]

Hi Eric,

On 9/3/2020 4:40 PM, Auger Eric wrote:
The patches are dependent on some changes in the mc-bus (bus/fsl-mc) 
driver. The changes were needed in order to re-use code and to export 
some more functions that are needed by the VFIO driver. Currenlty the 
mc-bus patches are under review: 
https://www.spinics.net/lists/kernel/msg3639226.html 
Could you share a branch with both series? This would help the review. 
Thanks Eric 


I have pushed both the series here: 
https://source.codeaurora.org/external/qoriq/qoriq-components/linux-extras/log/?h=dpaa2_direct_assignment 



Regards,

Diana

PS: I apologize if you received the message twice, I have sent it by 
mistake as html first.

[PATCH v4 02/10] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun]

The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 84 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  1 +
 2 files changed, 85 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 8b53c2a25b32..85e007be3a5d 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
 
 #include "vfio_fsl_mc_private.h"
 
+static struct fsl_mc_driver vfio_fsl_mc_driver;
+
 static int vfio_fsl_mc_open(void *device_data)
 {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,72 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
 };
 
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,
+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   if (!mc_dev->driver_override)
+   dev_warn(dev, "Setting driver override for device in 
dprc %s failed\n",
+dev_name(_cont->dev));
+   dev_info(dev, "Setting driver override for device in dprc %s\n",
+dev_name(_cont->dev));
+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "Object %s bound to driver %s while DPRC 
bound to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret < 0) {
+   dev_err(_dev->dev, "Failed to setup DPRC (error = %d)\n", 
ret);
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   return ret;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret < 0) {
+   dev_err(_dev->dev, "Container scanning failed: %d\n", ret);
+   dprc_cleanup(mc_dev);
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   }
+
+   return ret;
+}
+
 static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
 {
struct iommu_group *group;
@@ -112,6 +180,12 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
 
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret < 0) {
+   vfio_iommu_group_put(group, dev);
+   return ret;
+   }
+
return ret;
 }
 
@@ -124,6 +198,16 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   if (vdev->nb.notifier_call)
+   bus_unregister_notifier(_mc_bus_type, >nb);
+
+   if (is_fsl_mc_bus_dprc(mc_dev)) {
+   dprc_remove_devices(mc_dev, NULL, 0);
+   dprc_cleanup(mc_dev);
+   }
+
+   mc_dev->mc_io = NULL;
+
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
return 0;
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index e79cc116f6b8..37d61eaa58c8 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -9,6 +9,7 @@
 
 struct vfio_fsl_mc_device {
struct fsl_mc_device*mc_dev;
+   struct notifier_blocknb;
 };
 
 #endif /* VFIO_FSL_MC_PRIVATE_H */
-- 
2.17.1

[PATCH v4 03/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl

[Diana Craciun]

Allow userspace to get fsl-mc device info (number of regions
and irqs).

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 21 -
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 85e007be3a5d..5a5460d01f00 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,10 +33,29 @@ static void vfio_fsl_mc_release(void *device_data)
 static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
  unsigned long arg)
 {
+   unsigned long minsz;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
switch (cmd) {
case VFIO_DEVICE_GET_INFO:
{
-   return -ENOTTY;
+   struct vfio_device_info info;
+
+   minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   info.flags = VFIO_DEVICE_FLAGS_FSL_MC;
+   info.num_regions = mc_dev->obj_desc.region_count;
+   info.num_irqs = mc_dev->obj_desc.irq_count;
+
+   return copy_to_user((void __user *)arg, , minsz) ?
+   -EFAULT : 0;
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-- 
2.17.1

[PATCH v4 00/10] vfio/fsl-mc: VFIO support for FSL-MC device

[Diana Craciun]

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
driver. The changes were needed in order to re-use code and to export
some more functions that are needed by the VFIO driver.
Currenlty the mc-bus patches are under review:
https://www.spinics.net/lists/kernel/msg3639226.html

v3 --> v4
- use bus provided functions to tear down the DPRC
- added reset support

v2 --> v3
- There is no need to align region size to page size
- read/write implemented for all DPAA2 objects
- review fixes

v1 --> v2
- Fixed the container reset, a new flag added to the firmware command
- Implement a bus notifier for setting driver_override


Bharat Bhushan (1):
  vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

Diana Craciun (9):
  vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
  vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
  vfio/fsl-mc: Added lock support in preparation for interrupt handling
  vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
  vfio/fsl-mc: trigger an interrupt via eventfd
  vfio/fsl-mc: Add read/write support for fsl-mc devices
  vfio/fsl-mc: Add support for device reset

 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 +
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 684 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 221 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  56 ++
 include/uapi/linux/vfio.h |   1 +
 9 files changed, 983 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

-- 
2.17.1

[PATCH v4 05/10] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions

[Diana Craciun]

Allow userspace to mmap device regions for direct access of
fsl-mc devices.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 60 +--
 1 file changed, 58 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 093b8d68496c..64d5c1fff51f 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,7 +33,8 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device 
*vdev)
 
vdev->regions[i].addr = res->start;
vdev->regions[i].size = resource_size(res);
-   vdev->regions[i].flags = 0;
+   vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
+   vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
}
 
vdev->num_regions = mc_dev->obj_desc.region_count;
@@ -164,9 +165,64 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const 
char __user *buf,
return -EINVAL;
 }
 
+static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,
+struct vm_area_struct *vma)
+{
+   u64 size = vma->vm_end - vma->vm_start;
+   u64 pgoff, base;
+   u8 region_cacheable;
+
+   pgoff = vma->vm_pgoff &
+   ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+   base = pgoff << PAGE_SHIFT;
+
+   if (region.size < PAGE_SIZE || base + size > region.size)
+   return -EINVAL;
+
+   region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) &&
+  (region.type & FSL_MC_REGION_SHAREABLE);
+   if (!region_cacheable)
+   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+   vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff;
+
+   return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+  size, vma->vm_page_prot);
+}
+
 static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int index;
+
+   index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
+
+   if (vma->vm_end < vma->vm_start)
+   return -EINVAL;
+   if (vma->vm_start & ~PAGE_MASK)
+   return -EINVAL;
+   if (vma->vm_end & ~PAGE_MASK)
+   return -EINVAL;
+   if (!(vma->vm_flags & VM_SHARED))
+   return -EINVAL;
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ)
+   && (vma->vm_flags & VM_READ))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)
+   && (vma->vm_flags & VM_WRITE))
+   return -EINVAL;
+
+   vma->vm_private_data = mc_dev;
+
+   return vfio_fsl_mc_mmap_mmio(vdev->regions[index], vma);
 }
 
 static const struct vfio_device_ops vfio_fsl_mc_ops = {
-- 
2.17.1

[PATCH v4 09/10] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun]

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 115 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
 2 files changed, 114 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 73834f488a94..27713aa86878 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
@@ -106,6 +107,9 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
vdev->regions[i].size = resource_size(res);
vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
 
vdev->num_regions = mc_dev->obj_desc.region_count;
@@ -114,6 +118,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
 static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
 {
+   int i;
+
+   for (i = 0; i < vdev->num_regions; i++)
+   iounmap(vdev->regions[i].ioaddr);
+
vdev->num_regions = 0;
kfree(vdev->regions);
 }
@@ -311,13 +320,115 @@ static long vfio_fsl_mc_ioctl(void *device_data, 
unsigned int cmd,
 static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;
+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+   int i;
+   enum mc_cmd_status status;
+   unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+   /* Write at command parameter into portal */
+   for (i = 7; i >= 1; i--)
+   writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+   /* Write command header in the end */
+   writeq(cmd_data[0], ioaddr);
+
+   /* Wait for response before returning to user-space
+* This can be optimized in future to even prepare response
+* before returning to user-space and avoid read ioctl.
+*/
+   for (;;) {
+   u64 header;
+   struct mc_cmd_header *resp_hdr;
+
+   header = cpu_to_le64(readq_relaxed(ioaddr));
+
+   resp_hdr = (struct mc_cmd_header *)
+   status = (enum mc_cmd_status)resp_hdr->status;
+   if (status != MC_CMD_STATUS_READY)
+   break;
+
+   udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS);
+   timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS;
+   if (timeout_usecs == 0)
+   return -ETIMEDOUT;
+   }
+
+   return 0;
 }
 
 static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf,
 size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   st

[PATCH v4 07/10] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices

[Diana Craciun]

This patch adds the skeleton for interrupt support
for fsl-mc devices. The interrupts are not yet functional,
the functionality will be added by subsequent patches.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/Makefile  |  2 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 75 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 63 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  7 ++-
 4 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
index 0c6e5d2ddaae..cad6dbf0b735 100644
--- a/drivers/vfio/fsl-mc/Makefile
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 
-vfio-fsl-mc-y := vfio_fsl_mc.o
+vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o
 obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index bbd3365e877e..42014297b484 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -209,11 +209,79 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
-   return -ENOTTY;
+   struct vfio_irq_info info;
+
+   minsz = offsetofend(struct vfio_irq_info, count);
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   info.flags = VFIO_IRQ_INFO_EVENTFD;
+   info.count = 1;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_SET_IRQS:
{
-   return -ENOTTY;
+   struct vfio_irq_set hdr;
+   u8 *data = NULL;
+   int ret = 0;
+
+   minsz = offsetofend(struct vfio_irq_set, count);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (hdr.argsz < minsz)
+   return -EINVAL;
+
+   if (hdr.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   if (hdr.start != 0 || hdr.count > 1)
+   return -EINVAL;
+
+   if (hdr.count == 0 &&
+   (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE) ||
+   !(hdr.flags & VFIO_IRQ_SET_ACTION_TRIGGER)))
+   return -EINVAL;
+
+   if (hdr.flags & ~(VFIO_IRQ_SET_DATA_TYPE_MASK |
+ VFIO_IRQ_SET_ACTION_TYPE_MASK))
+   return -EINVAL;
+
+   if (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE)) {
+   size_t size;
+
+   if (hdr.flags & VFIO_IRQ_SET_DATA_BOOL)
+   size = sizeof(uint8_t);
+   else if (hdr.flags & VFIO_IRQ_SET_DATA_EVENTFD)
+   size = sizeof(int32_t);
+   else
+   return -EINVAL;
+
+   if (hdr.argsz - minsz < hdr.count * size)
+   return -EINVAL;
+
+   data = memdup_user((void __user *)(arg + minsz),
+  hdr.count * size);
+   if (IS_ERR(data))
+   return PTR_ERR(data);
+   }
+
+   mutex_lock(>igate);
+   ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags,
+hdr.index, hdr.start,
+hdr.count, data);
+   mutex_unlock(>igate);
+   kfree(data);
+
+   return ret;
}
case VFIO_DEVICE_RESET:
{
@@ -413,6 +481,8 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
 
+   mutex_init(>igate);
+
return ret;
 }
 
@@ -436,6 +506,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
mc_dev->mc_io = NULL;
 
vfio_fsl_mc_reflck_put(vdev->reflck);
+   mutex_destroy(>igate);
 
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
new file mode 100644
index ..058aa97aa54a
--- /dev/null
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -0,0 +1,63 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
+/*
+ * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2019 NXP
+ */
+
+#inc

[PATCH v4 04/10] vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call

[Diana Craciun]

Expose to userspace information about the memory regions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 79 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 19 ++
 2 files changed, 97 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 5a5460d01f00..093b8d68496c 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,16 +17,72 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int count = mc_dev->obj_desc.region_count;
+   int i;
+
+   vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region),
+   GFP_KERNEL);
+   if (!vdev->regions)
+   return -ENOMEM;
+
+   for (i = 0; i < count; i++) {
+   struct resource *res = _dev->regions[i];
+
+   vdev->regions[i].addr = res->start;
+   vdev->regions[i].size = resource_size(res);
+   vdev->regions[i].flags = 0;
+   }
+
+   vdev->num_regions = mc_dev->obj_desc.region_count;
+   return 0;
+}
+
+static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
+{
+   vdev->num_regions = 0;
+   kfree(vdev->regions);
+}
+
 static int vfio_fsl_mc_open(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
+
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
+   mutex_lock(>driver_lock);
+   if (!vdev->refcnt) {
+   ret = vfio_fsl_mc_regions_init(vdev);
+   if (ret)
+   goto err_reg_init;
+   }
+   vdev->refcnt++;
+
+   mutex_unlock(>driver_lock);
+
return 0;
+
+err_reg_init:
+   mutex_unlock(>driver_lock);
+   module_put(THIS_MODULE);
+   return ret;
 }
 
 static void vfio_fsl_mc_release(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+
+   mutex_lock(>driver_lock);
+
+   if (!(--vdev->refcnt))
+   vfio_fsl_mc_regions_cleanup(vdev);
+
+   mutex_unlock(>driver_lock);
+
module_put(THIS_MODULE);
 }
 
@@ -59,7 +115,25 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-   return -ENOTTY;
+   struct vfio_region_info info;
+
+   minsz = offsetofend(struct vfio_region_info, offset);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= vdev->num_regions)
+   return -EINVAL;
+
+   /* map offset to the physical address  */
+   info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
+   info.size = vdev->regions[info.index].size;
+   info.flags = vdev->regions[info.index].flags;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -204,6 +278,7 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
vfio_iommu_group_put(group, dev);
return ret;
}
+   mutex_init(>driver_lock);
 
return ret;
 }
@@ -227,6 +302,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
 
mc_dev->mc_io = NULL;
 
+   mutex_destroy(>driver_lock);
+
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
return 0;
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index 37d61eaa58c8..818dfd3df4db 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -7,9 +7,28 @@
 #ifndef VFIO_FSL_MC_PRIVATE_H
 #define VFIO_FSL_MC_PRIVATE_H
 
+#define VFIO_FSL_MC_OFFSET_SHIFT40
+#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1)
+
+#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT)
+
+#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \
+   ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT)
+
+struct vfio_fsl_mc_region {
+   u32 flags;
+   u32 type;
+   u64 addr;
+   resource_size_t size;
+};
+
 struct vfio_fsl_mc_device {
struct fsl_mc_device*mc_dev;
struct notifier_blocknb;
+   int refcnt;
+   u32 num_regions;
+   struct vfio_fsl_mc_region   *regions;
+   struct mutex driver_lock;
 };
 
 #endif /* VFIO_FSL_MC_PRIVATE_H */
-- 
2.17.1

[PATCH v4 06/10] vfio/fsl-mc: Added lock support in preparation for interrupt handling

[Diana Craciun]

Only the DPRC object allocates interrupts from the MSI
interrupt domain. The interrupts are managed by the DPRC in
a pool of interrupts. The access to this pool of interrupts
has to be protected with a lock.
This patch extends the current lock implementation to have a
lock per DPRC.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 91 +--
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  8 +-
 2 files changed, 91 insertions(+), 8 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 64d5c1fff51f..bbd3365e877e 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,6 +17,77 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static DEFINE_MUTEX(reflck_lock);
+
+static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_get(>kref);
+}
+
+static void vfio_fsl_mc_reflck_release(struct kref *kref)
+{
+   struct vfio_fsl_mc_reflck *reflck = container_of(kref,
+ struct vfio_fsl_mc_reflck,
+ kref);
+
+   mutex_destroy(>lock);
+   kfree(reflck);
+   mutex_unlock(_lock);
+}
+
+static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_put_mutex(>kref, vfio_fsl_mc_reflck_release, _lock);
+}
+
+static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void)
+{
+   struct vfio_fsl_mc_reflck *reflck;
+
+   reflck = kzalloc(sizeof(*reflck), GFP_KERNEL);
+   if (!reflck)
+   return ERR_PTR(-ENOMEM);
+
+   kref_init(>kref);
+   mutex_init(>lock);
+
+   return reflck;
+}
+
+static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
+{
+   int ret = 0;
+
+   mutex_lock(_lock);
+   if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
+   vdev->reflck = vfio_fsl_mc_reflck_alloc();
+   } else {
+   struct device *mc_cont_dev = vdev->mc_dev->dev.parent;
+   struct vfio_device *device;
+   struct vfio_fsl_mc_device *cont_vdev;
+
+   device = vfio_device_get_from_dev(mc_cont_dev);
+   if (!device) {
+   ret = -ENODEV;
+   goto unlock;
+   }
+
+   cont_vdev = vfio_device_data(device);
+   if (!cont_vdev->reflck) {
+   vfio_device_put(device);
+   ret = -ENODEV;
+   goto unlock;
+   }
+   vfio_fsl_mc_reflck_get(cont_vdev->reflck);
+   vdev->reflck = cont_vdev->reflck;
+   vfio_device_put(device);
+   }
+
+unlock:
+   mutex_unlock(_lock);
+   return ret;
+}
+
 static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
 {
struct fsl_mc_device *mc_dev = vdev->mc_dev;
@@ -55,7 +126,7 @@ static int vfio_fsl_mc_open(void *device_data)
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
if (!vdev->refcnt) {
ret = vfio_fsl_mc_regions_init(vdev);
if (ret)
@@ -63,12 +134,12 @@ static int vfio_fsl_mc_open(void *device_data)
}
vdev->refcnt++;
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
return 0;
 
 err_reg_init:
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
module_put(THIS_MODULE);
return ret;
 }
@@ -77,12 +148,12 @@ static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
 
if (!(--vdev->refcnt))
vfio_fsl_mc_regions_cleanup(vdev);
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
 }
@@ -329,12 +400,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
 
+   ret = vfio_fsl_mc_reflck_attach(vdev);
+   if (ret) {
+   vfio_iommu_group_put(group, dev);
+   return ret;
+   }
+
ret = vfio_fsl_mc_init_device(vdev);
if (ret < 0) {
+   vfio_fsl_mc_reflck_put(vdev->reflck);
vfio_iommu_group_put(group, dev);
return ret;
}
-   mutex_init(>driver_lock);
 
return ret;
 }
@@ -358,7 +435,7 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
 
mc_dev->mc_io = NULL;
 
-   mutex_destroy(>driver_lock);
+   vfio_fsl_mc_reflck_put(vdev->reflck);
 
vfio_iommu_group_put(mc_dev->dev.iommu_group, dev);
 
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_f

[PATCH v4 01/10] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

[Diana Craciun]

From: Bharat Bhushan 

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

This patch adds the infrastructure for VFIO support for fsl-mc
devices. Subsequent patches will add support for binding and secure
assigning these devices using VFIO.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 ++
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 160 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  14 ++
 include/uapi/linux/vfio.h |   1 +
 8 files changed, 196 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 3b186ade3597..f3f9ea108588 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18229,6 +18229,12 @@ F: drivers/vfio/
 F: include/linux/vfio.h
 F: include/uapi/linux/vfio.h
 
+VFIO FSL-MC DRIVER
+M: Diana Craciun 
+L: k...@vger.kernel.org
+S: Maintained
+F: drivers/vfio/fsl-mc/
+
 VFIO MEDIATED DEVICE DRIVERS
 M: Kirti Wankhede 
 L: k...@vger.kernel.org
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index fd17db9b432f..5533df91b257 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
 source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/fsl-mc/Kconfig"
 source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..fee73f3d9480 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
 obj-$(CONFIG_VFIO_PCI) += pci/
 obj-$(CONFIG_VFIO_PLATFORM) += platform/
 obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/
diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig
new file mode 100644
index ..b1a527d6b6f2
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Kconfig
@@ -0,0 +1,9 @@
+config VFIO_FSL_MC
+   tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices"
+   depends on VFIO && FSL_MC_BUS && EVENTFD
+   help
+ Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc
+ (Management Complex) devices. This is required to passthrough
+ fsl-mc bus devices using the VFIO framework.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
new file mode 100644
index ..0c6e5d2ddaae
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -0,0 +1,4 @@
+# SPDX-Lic

[PATCH v4 08/10] vfio/fsl-mc: trigger an interrupt via eventfd

[Diana Craciun]

This patch allows to set an eventfd for fsl-mc device interrupts
and also to trigger the interrupt eventfd from userspace for testing.

All fsl-mc device interrupts are MSIs. The MSIs are allocated from
the MSI domain only once per DPRC and used by all the DPAA2 objects.
The interrupts are managed by the DPRC in a pool of interrupts. Each
device requests interrupts from this pool. The pool is allocated
when the first virtual device is setting the interrupts.
The pool of interrupts is protected by a lock.

The DPRC has an interrupt of its own which indicates if the DPRC
contents have changed. However, currently, the contents of a DPRC
assigned to the guest cannot be changed at runtime, so this interrupt
is not configured.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c |  18 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 160 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  10 ++
 3 files changed, 186 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 42014297b484..73834f488a94 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -147,12 +147,28 @@ static int vfio_fsl_mc_open(void *device_data)
 static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
 
mutex_lock(>reflck->lock);
 
-   if (!(--vdev->refcnt))
+   if (!(--vdev->refcnt)) {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
+
vfio_fsl_mc_regions_cleanup(vdev);
 
+   /* reset the device before cleaning up the interrupts */
+   ret = dprc_reset_container(mc_cont->mc_io, 0,
+ mc_cont->mc_handle,
+ mc_cont->obj_desc.id,
+ DPRC_RESET_OPTION_NON_RECURSIVE);
+
+   vfio_fsl_mc_irqs_cleanup(vdev);
+
+   fsl_mc_cleanup_irq_pool(mc_cont);
+   }
+
mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 058aa97aa54a..409f3507fcf3 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -29,12 +29,149 @@ static int vfio_fsl_mc_irq_unmask(struct 
vfio_fsl_mc_device *vdev,
return -EINVAL;
 }
 
+int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_irq *mc_irq;
+   int irq_count;
+   int ret, i;
+
+/* Device does not support any interrupt */
+   if (mc_dev->obj_desc.irq_count == 0)
+   return 0;
+
+   /* interrupts were already allocated for this device */
+   if (vdev->mc_irqs)
+   return 0;
+
+   irq_count = mc_dev->obj_desc.irq_count;
+
+   mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
+   if (!mc_irq)
+   return -ENOMEM;
+
+   /* Allocate IRQs */
+   ret = fsl_mc_allocate_irqs(mc_dev);
+   if (ret) {
+   kfree(mc_irq);
+   return ret;
+   }
+
+   for (i = 0; i < irq_count; i++) {
+   mc_irq[i].count = 1;
+   mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
+   }
+
+   vdev->mc_irqs = mc_irq;
+
+   return 0;
+}
+
+static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg)
+{
+   struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg;
+
+   eventfd_signal(mc_irq->trigger, 1);
+   return IRQ_HANDLED;
+}
+
+static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev,
+  int index, int fd)
+{
+   struct vfio_fsl_mc_irq *irq = >mc_irqs[index];
+   struct eventfd_ctx *trigger;
+   int hwirq;
+   int ret;
+
+   hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq;
+   if (irq->trigger) {
+   free_irq(hwirq, irq);
+   kfree(irq->name);
+   eventfd_ctx_put(irq->trigger);
+   irq->trigger = NULL;
+   }
+
+   if (fd < 0) /* Disable only */
+   return 0;
+
+   irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)",
+   hwirq, dev_name(>mc_dev->dev));
+   if (!irq->name)
+   return -ENOMEM;
+
+   trigger = eventfd_ctx_fdget(fd);
+   if (IS_ERR(trigger)) {
+   kfree(irq->name);
+   return PTR_ERR(trigger);
+   }
+
+   irq->trigger = trigger;
+
+   ret = request_irq(hwirq, vfio_fsl_mc_irq_handler, 0,
+ irq->name, irq);
+   if (ret

[PATCH v4 10/10] vfio/fsl-mc: Add support for device reset

[Diana Craciun]

Currently only resetting the DPRC container is supported which
will reset all the objects inside it. Resetting individual
objects is possible from the userspace by issueing commands
towards MC firmware.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 27713aa86878..d17c5b3148ad 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -310,7 +310,20 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_RESET:
{
-   return -ENOTTY;
+   int ret = 0;
+
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
+   /* reset is supported only for the DPRC */
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -ENOTTY;
+
+   ret = dprc_reset_container(mc_dev->mc_io, 0,
+  mc_dev->mc_handle,
+  mc_dev->obj_desc.id,
+  DPRC_RESET_OPTION_NON_RECURSIVE);
+   return ret;
+
}
default:
return -ENOTTY;
-- 
2.17.1

[PATCH v4 06/13] bus/fsl-mc: Add dprc-reset-container support

[Diana Craciun]

From: Bharat Bhushan 

DPRC reset is required by VFIO-mc in order to stop a device
to further generate DMA transactions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 71 +
 drivers/bus/fsl-mc/fsl-mc-private.h |  7 +++
 include/linux/fsl/mc.h  |  7 +++
 3 files changed, 85 insertions(+)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index e76f2c76f4c8..2448a723eb28 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -80,6 +80,77 @@ int dprc_close(struct fsl_mc_io *mc_io,
 }
 EXPORT_SYMBOL_GPL(dprc_close);
 
+/**
+ * dprc_reset_container - Reset child container.
+ * @mc_io: Pointer to MC portal's I/O object
+ * @cmd_flags: Command flags; one or more of 'MC_CMD_FLAG_'
+ * @token: Token of DPRC object
+ * @child_container_id:ID of the container to reset
+ * @options: 32 bit options:
+ *   - 0 (no bits set) - all the objects inside the container are
+ * reset. The child containers are entered recursively and the
+ * objects reset. All the objects (including the child containers)
+ * are closed.
+ *   - bit 0 set - all the objects inside the container are reset.
+ * However the child containers are not entered recursively.
+ * This option is supported for API versions >= 6.5
+ * In case a software context crashes or becomes non-responsive, the parent
+ * may wish to reset its resources container before the software context is
+ * restarted.
+ *
+ * This routine informs all objects assigned to the child container that the
+ * container is being reset, so they may perform any cleanup operations that 
are
+ * needed. All objects handles that were owned by the child container shall be
+ * closed.
+ *
+ * Note that such request may be submitted even if the child software context
+ * has not crashed, but the resulting object cleanup operations will not be
+ * aware of that.
+ *
+ * Return: '0' on Success; Error code otherwise.
+ */
+int dprc_reset_container(struct fsl_mc_io *mc_io,
+u32 cmd_flags,
+u16 token,
+int child_container_id,
+u32 options)
+{
+   struct fsl_mc_command cmd = { 0 };
+   struct dprc_cmd_reset_container *cmd_params;
+   u32 cmdid = DPRC_CMDID_RESET_CONT;
+   int err;
+
+   /*
+* If the DPRC object version was not yet cached, cache it now.
+* Otherwise use the already cached value.
+*/
+   if (!dprc_major_ver && !dprc_minor_ver) {
+   err = dprc_get_api_version(mc_io, 0,
+   _major_ver,
+   _minor_ver);
+   if (err)
+   return err;
+   }
+
+   /*
+* MC API 6.5 introduced a new field in the command used to pass
+* some flags.
+* Bit 0 indicates that the child containers are not recursively reset.
+*/
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 5))
+   cmdid = DPRC_CMDID_RESET_CONT_V2;
+
+   /* prepare command */
+   cmd.header = mc_encode_cmd_header(cmdid, cmd_flags, token);
+   cmd_params = (struct dprc_cmd_reset_container *)cmd.params;
+   cmd_params->child_container_id = cpu_to_le32(child_container_id);
+   cmd_params->options = cpu_to_le32(options);
+
+   /* send command to mc*/
+   return mc_send_command(mc_io, );
+}
+EXPORT_SYMBOL_GPL(dprc_reset_container);
+
 /**
  * dprc_set_irq() - Set IRQ information for the DPRC to trigger an interrupt.
  * @mc_io: Pointer to MC portal's I/O object
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 9f200731b274..5f7e762d517c 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -91,6 +91,8 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 #define DPRC_CMDID_GET_API_VERSION  DPRC_CMD(0xa05)
 
 #define DPRC_CMDID_GET_ATTR DPRC_CMD(0x004)
+#define DPRC_CMDID_RESET_CONT   DPRC_CMD(0x005)
+#define DPRC_CMDID_RESET_CONT_V2DPRC_CMD_V2(0x005)
 
 #define DPRC_CMDID_SET_IRQ  DPRC_CMD(0x010)
 #define DPRC_CMDID_SET_IRQ_ENABLE   DPRC_CMD(0x012)
@@ -111,6 +113,11 @@ struct dprc_cmd_open {
__le32 container_id;
 };
 
+struct dprc_cmd_reset_container {
+   __le32 child_container_id;
+   __le32 options;
+};
+
 struct dprc_cmd_set_irq {
/* cmd word 0 */
__le32 irq_val;
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 03a5d16dde73..1d8800acf21f 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -524,6 +524,13 @@ static inline bool is_fsl_mc_bus_dpdmai(const struct 
fsl_mc_device *mc_dev)
return mc_dev->dev.type == 

[PATCH v4 09/13] bus/fsl-mc: Add a container setup function

[Diana Craciun]

Both DPRC driver and VFIO driver use the same initialization code
for the DPRC. Introduced a new function which groups this
initialization code. The function is exported and may be
used by VFIO as well.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 65 ++--
 include/linux/fsl/mc.h   |  2 +
 2 files changed, 48 insertions(+), 19 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 6d7b9412fb77..471386e9b46c 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -588,16 +588,15 @@ static int dprc_setup_irq(struct fsl_mc_device *mc_dev)
 }
 
 /**
- * dprc_probe - callback invoked when a DPRC is being bound to this driver
+ * dprc_setup - opens and creates a mc_io for DPRC
  *
  * @mc_dev: Pointer to fsl-mc device representing a DPRC
  *
  * It opens the physical DPRC in the MC.
- * It scans the DPRC to discover the MC objects contained in it.
- * It creates the interrupt pool for the MC bus associated with the DPRC.
- * It configures the interrupts for the DPRC device itself.
+ * It configures the DPRC portal used to communicate with MC
  */
-static int dprc_probe(struct fsl_mc_device *mc_dev)
+
+int dprc_setup(struct fsl_mc_device *mc_dev)
 {
int error;
size_t region_size;
@@ -681,35 +680,63 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
goto error_cleanup_open;
}
 
+   return 0;
+
+error_cleanup_open:
+   (void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
+
+error_cleanup_msi_domain:
+   if (msi_domain_set)
+   dev_set_msi_domain(_dev->dev, NULL);
+
+   if (mc_io_created) {
+   fsl_destroy_mc_io(mc_dev->mc_io);
+   mc_dev->mc_io = NULL;
+   }
+
+   return error;
+}
+EXPORT_SYMBOL_GPL(dprc_setup);
+
+/**
+ * dprc_probe - callback invoked when a DPRC is being bound to this driver
+ *
+ * @mc_dev: Pointer to fsl-mc device representing a DPRC
+ *
+ * It opens the physical DPRC in the MC.
+ * It scans the DPRC to discover the MC objects contained in it.
+ * It creates the interrupt pool for the MC bus associated with the DPRC.
+ * It configures the interrupts for the DPRC device itself.
+ */
+static int dprc_probe(struct fsl_mc_device *mc_dev)
+{
+   int error;
+
+   error = dprc_setup(mc_dev);
+   if (error < 0)
+   return error;
+
/*
 * Discover MC objects in DPRC object:
 */
error = dprc_scan_container(mc_dev, true);
if (error < 0)
-   goto error_cleanup_open;
+   goto dprc_cleanup;
 
/*
 * Configure interrupt for the DPRC object associated with this MC bus:
 */
error = dprc_setup_irq(mc_dev);
if (error < 0)
-   goto error_cleanup_open;
+   goto scan_cleanup;
 
dev_info(_dev->dev, "DPRC device bound to driver");
return 0;
 
-error_cleanup_open:
-   (void)dprc_close(mc_dev->mc_io, 0, mc_dev->mc_handle);
-
-error_cleanup_msi_domain:
-   if (msi_domain_set)
-   dev_set_msi_domain(_dev->dev, NULL);
-
-   if (mc_io_created) {
-   fsl_destroy_mc_io(mc_dev->mc_io);
-   mc_dev->mc_io = NULL;
-   }
-
+scan_cleanup:
+   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
+dprc_cleanup:
+   dprc_cleanup(mc_dev);
return error;
 }
 
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 5519a510b8c9..e99d181ee4cd 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -540,6 +540,8 @@ void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
 
 int dprc_cleanup(struct fsl_mc_device *mc_dev);
 
+int dprc_setup(struct fsl_mc_device *mc_dev);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v4 10/13] bus/fsl_mc: Do not rely on caller to provide non NULL mc_io

[Diana Craciun]

Before destroying the mc_io, check first that it was
allocated.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/mc-io.c | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/bus/fsl-mc/mc-io.c b/drivers/bus/fsl-mc/mc-io.c
index a30b53f1d87d..305015486b91 100644
--- a/drivers/bus/fsl-mc/mc-io.c
+++ b/drivers/bus/fsl-mc/mc-io.c
@@ -129,7 +129,12 @@ int __must_check fsl_create_mc_io(struct device *dev,
  */
 void fsl_destroy_mc_io(struct fsl_mc_io *mc_io)
 {
-   struct fsl_mc_device *dpmcp_dev = mc_io->dpmcp_dev;
+   struct fsl_mc_device *dpmcp_dev;
+
+   if (!mc_io)
+   return;
+
+   dpmcp_dev = mc_io->dpmcp_dev;
 
if (dpmcp_dev)
fsl_mc_io_unset_dpmcp(mc_io);
-- 
2.17.1

[PATCH v4 07/13] bus/fsl-mc: Export dprc_scan/dprc_remove functions to be used by multiple entities

[Diana Craciun]

Currently the DPRC scan function is used only by the bus driver.
But the same functionality will be needed by the VFIO driver.
To support this, the dprc scan function was exported and a little
bit adjusted to fit both scenarios. Also the scan mutex initialization
is done when the bus object is created, not in dprc_probe in order
to be used by both VFIO and bus driver.
Similarily dprc_remove_devices is exported to be used by VFIO.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 26 +++---
 drivers/bus/fsl-mc/fsl-mc-bus.c  |  1 +
 include/linux/fsl/mc.h   |  7 +++
 3 files changed, 19 insertions(+), 15 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 3d850515e3e5..dde8fe965e30 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -81,9 +81,9 @@ static int __fsl_mc_device_remove(struct device *dev, void 
*data)
  * the MC by removing devices that represent MC objects that have
  * been dynamically removed in the physical DPRC.
  */
-static void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
-   struct fsl_mc_obj_desc *obj_desc_array,
-   int num_child_objects_in_mc)
+void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
+struct fsl_mc_obj_desc *obj_desc_array,
+int num_child_objects_in_mc)
 {
if (num_child_objects_in_mc != 0) {
/*
@@ -105,6 +105,7 @@ static void dprc_remove_devices(struct fsl_mc_device 
*mc_bus_dev,
  __fsl_mc_device_remove);
}
 }
+EXPORT_SYMBOL_GPL(dprc_remove_devices);
 
 static int __fsl_mc_device_match(struct device *dev, void *data)
 {
@@ -354,9 +355,10 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
  * bus driver with the actual state of the MC by adding and removing
  * devices as appropriate.
  */
-static int dprc_scan_container(struct fsl_mc_device *mc_bus_dev)
+int dprc_scan_container(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts)
 {
-   int error;
+   int error = 0;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
 
fsl_mc_init_all_resource_pools(mc_bus_dev);
@@ -365,16 +367,12 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev, true);
+   error = dprc_scan_objects(mc_bus_dev, alloc_interrupts);
mutex_unlock(_bus->scan_mutex);
-   if (error < 0) {
-   fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
-   return error;
-   }
 
-   return 0;
+   return error;
 }
-
+EXPORT_SYMBOL_GPL(dprc_scan_container);
 /**
  * dprc_irq0_handler - Regular ISR for DPRC interrupt 0
  *
@@ -683,12 +681,10 @@ static int dprc_probe(struct fsl_mc_device *mc_dev)
goto error_cleanup_open;
}
 
-   mutex_init(_bus->scan_mutex);
-
/*
 * Discover MC objects in DPRC object:
 */
-   error = dprc_scan_container(mc_dev);
+   error = dprc_scan_container(mc_dev, true);
if (error < 0)
goto error_cleanup_open;
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 1bb70a92167e..32f194814b08 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -678,6 +678,7 @@ int fsl_mc_device_add(struct fsl_mc_obj_desc *obj_desc,
if (!mc_bus)
return -ENOMEM;
 
+   mutex_init(_bus->scan_mutex);
mc_dev = _bus->mc_dev;
} else {
/*
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 1d8800acf21f..da11171bc38f 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -531,6 +531,13 @@ int dprc_reset_container(struct fsl_mc_io *mc_io,
 int child_container_id,
 u32 options);
 
+int dprc_scan_container(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts);
+
+void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
+struct fsl_mc_obj_desc *obj_desc_array,
+int num_child_objects_in_mc);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v4 13/13] bus/fsl-mc: Add a new version for dprc_get_obj_region command

[Diana Craciun]

The region size reported by the firmware for mc and software
portals was less than allocated by the hardware. This may be
problematic when mmapping the region in user space because the
region size is less than page size. However the size as reserved
by the hardware is 64K.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 38 ++---
 drivers/bus/fsl-mc/fsl-mc-private.h |  3 +++
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 650808799a29..57b097caf255 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -536,20 +536,30 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
return err;
}
 
-   /**
-* MC API version 6.3 introduced a new field to the region
-* descriptor: base_address. If the older API is in use then the base
-* address is set to zero to indicate it needs to be obtained elsewhere
-* (typically the device tree).
-*/
-   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
-cmd_flags, token);
-   else
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
-cmd_flags, token);
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 6)) 
{
+   /*
+* MC API version 6.6 changed the size of the MC portals and 
software
+* portals to 64K (as implemented by hardware). If older API is 
in use the
+* size reported is less (64 bytes for mc portals and 4K for 
software
+* portals).
+*/
+
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V3,
+ cmd_flags, token);
+
+   } else if (dprc_major_ver == 6 && dprc_minor_ver >= 3) {
+   /*
+* MC API version 6.3 introduced a new field to the region
+* descriptor: base_address. If the older API is in use then 
the base
+* address is set to zero to indicate it needs to be obtained 
elsewhere
+* (typically the device tree).
+*/
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
+ cmd_flags, token);
+   } else {
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
+ cmd_flags, token);
+   }
 
cmd_params = (struct dprc_cmd_get_obj_region *)cmd.params;
cmd_params->obj_id = cpu_to_le32(obj_id);
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 3e9f9c778631..85ca5fdee581 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -80,10 +80,12 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 /* DPRC command versioning */
 #define DPRC_CMD_BASE_VERSION  1
 #define DPRC_CMD_2ND_VERSION   2
+#define DPRC_CMD_3RD_VERSION   3
 #define DPRC_CMD_ID_OFFSET 4
 
 #define DPRC_CMD(id)   (((id) << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)
 #define DPRC_CMD_V2(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_2ND_VERSION)
+#define DPRC_CMD_V3(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_3RD_VERSION)
 
 /* DPRC command IDs */
 #define DPRC_CMDID_CLOSEDPRC_CMD(0x800)
@@ -105,6 +107,7 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 #define DPRC_CMDID_GET_OBJ  DPRC_CMD(0x15A)
 #define DPRC_CMDID_GET_OBJ_REG  DPRC_CMD(0x15E)
 #define DPRC_CMDID_GET_OBJ_REG_V2   DPRC_CMD_V2(0x15E)
+#define DPRC_CMDID_GET_OBJ_REG_V3   DPRC_CMD_V3(0x15E)
 #define DPRC_CMDID_SET_OBJ_IRQ  DPRC_CMD(0x15F)
 
 #define DPRC_CMDID_GET_CONNECTION   DPRC_CMD(0x16C)
-- 
2.17.1

[PATCH v4 08/13] bus/fsl-mc: Export a cleanup function for DPRC

[Diana Craciun]

Create and export a cleanup function for DPRC. The function
is used by the DPRC driver, but it will be used by the VFIO
driver as well.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 52 
 include/linux/fsl/mc.h   |  2 ++
 2 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index dde8fe965e30..6d7b9412fb77 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -728,33 +728,25 @@ static void dprc_teardown_irq(struct fsl_mc_device 
*mc_dev)
 }
 
 /**
- * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ * dprc_cleanup - function that cleanups a DPRC
  *
  * @mc_dev: Pointer to fsl-mc device representing the DPRC
  *
- * It removes the DPRC's child objects from Linux (not from the MC) and
- * closes the DPRC device in the MC.
- * It tears down the interrupts that were configured for the DPRC device.
+ * It closes the DPRC device in the MC.
  * It destroys the interrupt pool associated with this MC bus.
  */
-static int dprc_remove(struct fsl_mc_device *mc_dev)
+
+int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
-   if (!mc_dev->mc_io)
-   return -EINVAL;
 
-   if (!mc_bus->irq_resources)
+   if (!mc_dev->mc_io)
return -EINVAL;
 
-   if (dev_get_msi_domain(_dev->dev))
-   dprc_teardown_irq(mc_dev);
-
-   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
-
if (dev_get_msi_domain(_dev->dev)) {
fsl_mc_cleanup_irq_pool(mc_bus);
dev_set_msi_domain(_dev->dev, NULL);
@@ -771,6 +763,40 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
mc_dev->mc_io = NULL;
}
 
+   return 0;
+}
+EXPORT_SYMBOL_GPL(dprc_cleanup);
+
+/**
+ * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ *
+ * @mc_dev: Pointer to fsl-mc device representing the DPRC
+ *
+ * It removes the DPRC's child objects from Linux (not from the MC) and
+ * closes the DPRC device in the MC.
+ * It tears down the interrupts that were configured for the DPRC device.
+ * It destroys the interrupt pool associated with this MC bus.
+ */
+static int dprc_remove(struct fsl_mc_device *mc_dev)
+{
+   int error;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -EINVAL;
+
+   if (!mc_bus->irq_resources)
+   return -EINVAL;
+
+   if (dev_get_msi_domain(_dev->dev))
+   dprc_teardown_irq(mc_dev);
+
+   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
+
+   error = dprc_cleanup(mc_dev);
+   if (error < 0)
+   dev_err(_dev->dev, "dprc_close() failed: %d\n", error);
+
dev_info(_dev->dev, "DPRC device unbound from driver");
return 0;
 }
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index da11171bc38f..5519a510b8c9 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -538,6 +538,8 @@ void dprc_remove_devices(struct fsl_mc_device *mc_bus_dev,
 struct fsl_mc_obj_desc *obj_desc_array,
 int num_child_objects_in_mc);
 
+int dprc_cleanup(struct fsl_mc_device *mc_dev);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v4 00/13] bus/fsl-mc: Extend mc-bus driver functionalities in preparation for mc-bus VFIO support

[Diana Craciun]

The vfio-mc bus driver needs some additional services to be exported by the
mc-bus driver like:
- a way to reset the DPRC container
- driver_override support
- functions to setup/tear down a DPRC
- functions for allocating the pool of interrupts. In case of VFIO the
interrupts are not configured at probe time, but later by userspace
request

v3 -> v4
- Rebased on the latest kernel.
- Exported a dprc_remove function

v2 -> v3
- Add a new version for dprc_get_obj_region
- Export the cacheability bus specific bits defines

v1 -> v2
- Remove driver_override propagation through various functions
- Cache the DPRC API version

The patches are related with "vfio/fsl-mc: VFIO support for FSL-MC
devices" patches, but the series were split because they are targeting
different subsystems. However, the mc-bus patches may suffer changes
when addressing the VFIO review comments.

The patches do not address the comment regarding moving driver_override
in the core code. I prefer not to tie these patches on that change and
address that separately.

Bharat Bhushan (3):
  bus/fsl-mc: add support for 'driver_override' in the mc-bus
  bus/fsl-mc: Add dprc-reset-container support
  bus/fsl-mc: Extend ICID size from 16bit to 32bit

Diana Craciun (10):
  bus/fsl-mc: Do no longer export the total number of irqs outside
dprc_scan_objects
  bus/fsl-mc: Add a new parameter to dprc_scan_objects function
  bus/fsl-mc: Set the QMAN/BMAN region flags
  bus/fsl-mc: Cache the DPRC API version
  bus/fsl-mc: Export dprc_scan/dprc_remove functions to be used by
multiple entities
  bus/fsl-mc: Export a cleanup function for DPRC
  bus/fsl-mc: Add a container setup function
  bus/fsl_mc: Do not rely on caller to provide non NULL mc_io
  bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO
  bus/fsl-mc: Add a new version for dprc_get_obj_region command

 drivers/bus/fsl-mc/dprc-driver.c  | 175 --
 drivers/bus/fsl-mc/dprc.c | 141 +
 drivers/bus/fsl-mc/fsl-mc-allocator.c |  12 +-
 drivers/bus/fsl-mc/fsl-mc-bus.c   |  64 +-
 drivers/bus/fsl-mc/fsl-mc-private.h   |  31 ++---
 drivers/bus/fsl-mc/mc-io.c|   7 +-
 include/linux/fsl/mc.h|  41 +-
 7 files changed, 349 insertions(+), 122 deletions(-)

-- 
2.17.1

[PATCH v4 12/13] bus/fsl-mc: Extend ICID size from 16bit to 32bit

[Diana Craciun]

From: Bharat Bhushan 

In virtual machines the device-id range is defined
between 0x1-0x2. The reason for using such a
large range is to avoid overlapping with the PCI range.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 2 +-
 drivers/bus/fsl-mc/fsl-mc-bus.c | 2 +-
 drivers/bus/fsl-mc/fsl-mc-private.h | 5 ++---
 include/linux/fsl/mc.h  | 2 +-
 4 files changed, 5 insertions(+), 6 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 2448a723eb28..650808799a29 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -360,7 +360,7 @@ int dprc_get_attributes(struct fsl_mc_io *mc_io,
/* retrieve response parameters */
rsp_params = (struct dprc_rsp_get_attributes *)cmd.params;
attr->container_id = le32_to_cpu(rsp_params->container_id);
-   attr->icid = le16_to_cpu(rsp_params->icid);
+   attr->icid = le32_to_cpu(rsp_params->icid);
attr->options = le32_to_cpu(rsp_params->options);
attr->portal_id = le32_to_cpu(rsp_params->portal_id);
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 32f194814b08..d375bb585749 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -503,7 +503,7 @@ static int get_dprc_attr(struct fsl_mc_io *mc_io,
 }
 
 static int get_dprc_icid(struct fsl_mc_io *mc_io,
-int container_id, u16 *icid)
+int container_id, u32 *icid)
 {
struct dprc_attributes attr;
int error;
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index ffe709a3f0f8..3e9f9c778631 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -159,8 +159,7 @@ struct dprc_cmd_clear_irq_status {
 struct dprc_rsp_get_attributes {
/* response word 0 */
__le32 container_id;
-   __le16 icid;
-   __le16 pad;
+   __le32 icid;
/* response word 1 */
__le32 options;
__le32 portal_id;
@@ -337,7 +336,7 @@ int dprc_clear_irq_status(struct fsl_mc_io *mc_io,
  */
 struct dprc_attributes {
int container_id;
-   u16 icid;
+   u32 icid;
int portal_id;
u64 options;
 };
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index f791fe38c251..db244874e834 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -195,7 +195,7 @@ struct fsl_mc_device {
struct device dev;
u64 dma_mask;
u16 flags;
-   u16 icid;
+   u32 icid;
u16 mc_handle;
struct fsl_mc_io *mc_io;
struct fsl_mc_obj_desc obj_desc;
-- 
2.17.1

[PATCH v4 03/13] bus/fsl-mc: add support for 'driver_override' in the mc-bus

[Diana Craciun]

From: Bharat Bhushan 

This patch is required for vfio-fsl-mc meta driver to successfully bind
layerscape container devices for device passthrough. This patch adds
a mechanism to allow a layerscape device to specify a driver rather than
a layerscape driver provide a device match.

Example to allow a device (dprc.1) to specifically bind
with driver (vfio-fsl-mc):-
 - echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.1/driver_override
 - echo dprc.1 > /sys/bus/fsl-mc/drivers/fsl_mc_dprc/unbind
 - echo dprc.1 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

Signed-off-by: Bharat Bhushan 
Signed-off-by: Laurentiu Tudor 
Signed-off-by: Diana Craciun 
---

The patche do not address the comment regarding moving driver_override
in the core code. I prefer not to tie these patches on that change and
address that separately.

 drivers/bus/fsl-mc/fsl-mc-bus.c | 54 +
 include/linux/fsl/mc.h  |  2 ++
 2 files changed, 56 insertions(+)

diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index b69794e7364d..9a884936e53e 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -3,6 +3,7 @@
  * Freescale Management Complex (MC) bus driver
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -78,6 +79,12 @@ static int fsl_mc_bus_match(struct device *dev, struct 
device_driver *drv)
struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(drv);
bool found = false;
 
+   /* When driver_override is set, only bind to the matching driver */
+   if (mc_dev->driver_override) {
+   found = !strcmp(mc_dev->driver_override, mc_drv->driver.name);
+   goto out;
+   }
+
if (!mc_drv->match_id_table)
goto out;
 
@@ -147,8 +154,52 @@ static ssize_t modalias_show(struct device *dev, struct 
device_attribute *attr,
 }
 static DEVICE_ATTR_RO(modalias);
 
+static ssize_t driver_override_store(struct device *dev,
+struct device_attribute *attr,
+const char *buf, size_t count)
+{
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   char *driver_override, *old = mc_dev->driver_override;
+   char *cp;
+
+   if (WARN_ON(dev->bus != _mc_bus_type))
+   return -EINVAL;
+
+   if (count >= (PAGE_SIZE - 1))
+   return -EINVAL;
+
+   driver_override = kstrndup(buf, count, GFP_KERNEL);
+   if (!driver_override)
+   return -ENOMEM;
+
+   cp = strchr(driver_override, '\n');
+   if (cp)
+   *cp = '\0';
+
+   if (strlen(driver_override)) {
+   mc_dev->driver_override = driver_override;
+   } else {
+   kfree(driver_override);
+   mc_dev->driver_override = NULL;
+   }
+
+   kfree(old);
+
+   return count;
+}
+
+static ssize_t driver_override_show(struct device *dev,
+   struct device_attribute *attr, char *buf)
+{
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+
+   return snprintf(buf, PAGE_SIZE, "%s\n", mc_dev->driver_override);
+}
+static DEVICE_ATTR_RW(driver_override);
+
 static struct attribute *fsl_mc_dev_attrs[] = {
_attr_modalias.attr,
+   _attr_driver_override.attr,
NULL,
 };
 
@@ -748,6 +799,9 @@ EXPORT_SYMBOL_GPL(fsl_mc_device_add);
  */
 void fsl_mc_device_remove(struct fsl_mc_device *mc_dev)
 {
+   kfree(mc_dev->driver_override);
+   mc_dev->driver_override = NULL;
+
/*
 * The device-specific remove callback will get invoked by device_del()
 */
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index a428c61ead6e..3b5f0c98636d 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -161,6 +161,7 @@ struct fsl_mc_obj_desc {
  * @regions: pointer to array of MMIO region entries
  * @irqs: pointer to array of pointers to interrupts allocated to this device
  * @resource: generic resource associated with this MC object device, if any.
+ * @driver_override: driver name to force a match
  *
  * Generic device object for MC object devices that are "attached" to a
  * MC bus.
@@ -194,6 +195,7 @@ struct fsl_mc_device {
struct fsl_mc_device_irq **irqs;
struct fsl_mc_resource *resource;
struct device_link *consumer_link;
+   char   *driver_override;
 };
 
 #define to_fsl_mc_device(_dev) \
-- 
2.17.1

[PATCH v4 11/13] bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO

[Diana Craciun]

The IRQ pool handling functions can be used by both DPRC
driver and VFIO. Adapt and export those functions.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c  |  7 +++
 drivers/bus/fsl-mc/fsl-mc-allocator.c | 12 
 drivers/bus/fsl-mc/fsl-mc-private.h   | 10 --
 include/linux/fsl/mc.h| 11 +++
 4 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 471386e9b46c..3521515a4ac2 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -327,8 +327,8 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
}
 
if (alloc_interrupts && !mc_bus->irq_resources) {
-   error = fsl_mc_populate_irq_pool(mc_bus,
-   FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+   error = fsl_mc_populate_irq_pool(mc_bus_dev,
+FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
if (error < 0)
return error;
}
@@ -766,7 +766,6 @@ static void dprc_teardown_irq(struct fsl_mc_device *mc_dev)
 int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
-   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
@@ -775,7 +774,7 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev)
return -EINVAL;
 
if (dev_get_msi_domain(_dev->dev)) {
-   fsl_mc_cleanup_irq_pool(mc_bus);
+   fsl_mc_cleanup_irq_pool(mc_dev);
dev_set_msi_domain(_dev->dev, NULL);
}
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-allocator.c 
b/drivers/bus/fsl-mc/fsl-mc-allocator.c
index cc7bb900f524..e71a6f52ea0c 100644
--- a/drivers/bus/fsl-mc/fsl-mc-allocator.c
+++ b/drivers/bus/fsl-mc/fsl-mc-allocator.c
@@ -344,7 +344,7 @@ EXPORT_SYMBOL_GPL(fsl_mc_object_free);
  * Initialize the interrupt pool associated with an fsl-mc bus.
  * It allocates a block of IRQs from the GIC-ITS.
  */
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+int fsl_mc_populate_irq_pool(struct fsl_mc_device *mc_bus_dev,
 unsigned int irq_count)
 {
unsigned int i;
@@ -352,10 +352,14 @@ int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
struct fsl_mc_device_irq *irq_resources;
struct fsl_mc_device_irq *mc_dev_irq;
int error;
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
+   /* do nothing if the IRQ pool is already populated */
+   if (mc_bus->irq_resources)
+   return 0;
+
if (irq_count == 0 ||
irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS)
return -EINVAL;
@@ -407,9 +411,9 @@ EXPORT_SYMBOL_GPL(fsl_mc_populate_irq_pool);
  * Teardown the interrupt pool associated with an fsl-mc bus.
  * It frees the IRQs that were allocated to the pool, back to the GIC-ITS.
  */
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus)
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_device *mc_bus_dev)
 {
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 5f7e762d517c..ffe709a3f0f8 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -519,11 +519,6 @@ struct dpcon_cmd_set_notification {
__le64 user_ctx;
 };
 
-/**
- * Maximum number of total IRQs that can be pre-allocated for an MC bus'
- * IRQ pool
- */
-#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS 256
 
 /**
  * struct fsl_mc_resource_pool - Pool of MC resources of a given
@@ -598,11 +593,6 @@ void fsl_mc_msi_domain_free_irqs(struct device *dev);
 
 struct irq_domain *fsl_mc_find_msi_domain(struct device *dev);
 
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
-unsigned int irq_count);
-
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
-
 int __must_check fsl_create_mc_io(struct device *dev,
  phys_addr_t mc_portal_phys_addr,
  u32 mc_portal_size,
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index e99d181ee4cd..f791fe38c251 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -542,6 +542,17 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev);
 
 int dprc_setup(struct fsl_mc_device *mc_dev);
 
+/**
+ * Maximum number of total IRQs that can be pre-allocated for

[PATCH v4 04/13] bus/fsl-mc: Set the QMAN/BMAN region flags

[Diana Craciun]

The QMAN region is memory mapped, so it should be of type
IORESOURCE_MEM. The region flags bits were wrongly used to
pass additional information. Use the bus specific bits for
this purpose.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/fsl-mc-bus.c | 7 ++-
 drivers/bus/fsl-mc/fsl-mc-private.h | 6 --
 include/linux/fsl/mc.h  | 8 
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c
index 9a884936e53e..1bb70a92167e 100644
--- a/drivers/bus/fsl-mc/fsl-mc-bus.c
+++ b/drivers/bus/fsl-mc/fsl-mc-bus.c
@@ -615,11 +615,8 @@ static int fsl_mc_device_get_mmio_regions(struct 
fsl_mc_device *mc_dev,
 
regions[i].end = regions[i].start + region_desc.size - 1;
regions[i].name = "fsl-mc object MMIO region";
-   regions[i].flags = IORESOURCE_IO;
-   if (region_desc.flags & DPRC_REGION_CACHEABLE)
-   regions[i].flags |= IORESOURCE_CACHEABLE;
-   if (region_desc.flags & DPRC_REGION_SHAREABLE)
-   regions[i].flags |= IORESOURCE_MEM;
+   regions[i].flags = region_desc.flags & IORESOURCE_BITS;
+   regions[i].flags |= IORESOURCE_MEM;
}
 
mc_dev->regions = regions;
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index 7a46a12eb747..9f200731b274 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -358,12 +358,6 @@ int dprc_set_obj_irq(struct fsl_mc_io *mc_io,
 int obj_id,
 u8 irq_index,
 struct dprc_irq_cfg *irq_cfg);
-
-/* Region flags */
-/* Cacheable - Indicates that region should be mapped as cacheable */
-#define DPRC_REGION_CACHEABLE  0x0001
-#define DPRC_REGION_SHAREABLE  0x0002
-
 /**
  * enum dprc_region_type - Region type
  * @DPRC_REGION_TYPE_MC_PORTAL: MC portal region
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 3b5f0c98636d..03a5d16dde73 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -3,6 +3,7 @@
  * Freescale Management Complex (MC) bus public interface
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -148,6 +149,13 @@ struct fsl_mc_obj_desc {
  */
 #define FSL_MC_IS_DPRC 0x0001
 
+/* Region flags */
+/* Indicates that region can be mapped as cacheable */
+#define FSL_MC_REGION_CACHEABLE0x0001
+
+/* Indicates that region can be mapped as shareable */
+#define FSL_MC_REGION_SHAREABLE0x0002
+
 /**
  * struct fsl_mc_device - MC object device object
  * @dev: Linux driver model device object
-- 
2.17.1

[PATCH v4 05/13] bus/fsl-mc: Cache the DPRC API version

[Diana Craciun]

There are already firmware API commands that have multiple
versions. For each multiple version command, another command
to retrieve the API version is issued. This may introduce an important
overhead. The version does not change while the system is running,
so the DPRC API version can be safely cached.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c | 30 +-
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 602f030d84eb..e76f2c76f4c8 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 /*
  * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2020 NXP
  *
  */
 #include 
@@ -8,6 +9,13 @@
 
 #include "fsl-mc-private.h"
 
+/*
+ * cache the DPRC version to reduce the number of commands
+ * towards the mc firmware
+ */
+static u16 dprc_major_ver;
+static u16 dprc_minor_ver;
+
 /**
  * dprc_open() - Open DPRC object for use
  * @mc_io: Pointer to MC portal's I/O object
@@ -443,15 +451,19 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
struct fsl_mc_command cmd = { 0 };
struct dprc_cmd_get_obj_region *cmd_params;
struct dprc_rsp_get_obj_region *rsp_params;
-   u16 major_ver, minor_ver;
int err;
 
-   /* prepare command */
-   err = dprc_get_api_version(mc_io, 0,
-_ver,
-_ver);
-   if (err)
-   return err;
+/*
+ * If the DPRC object version was not yet cached, cache it now.
+ * Otherwise use the already cached value.
+ */
+   if (!dprc_major_ver && !dprc_minor_ver) {
+   err = dprc_get_api_version(mc_io, 0,
+ _major_ver,
+ _minor_ver);
+   if (err)
+   return err;
+   }
 
/**
 * MC API version 6.3 introduced a new field to the region
@@ -459,7 +471,7 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
 * address is set to zero to indicate it needs to be obtained elsewhere
 * (typically the device tree).
 */
-   if (major_ver > 6 || (major_ver == 6 && minor_ver >= 3))
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
cmd.header =
mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
 cmd_flags, token);
@@ -483,7 +495,7 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
rsp_params = (struct dprc_rsp_get_obj_region *)cmd.params;
region_desc->base_offset = le64_to_cpu(rsp_params->base_offset);
region_desc->size = le32_to_cpu(rsp_params->size);
-   if (major_ver > 6 || (major_ver == 6 && minor_ver >= 3))
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
region_desc->base_address = le64_to_cpu(rsp_params->base_addr);
else
region_desc->base_address = 0;
-- 
2.17.1

[PATCH v4 02/13] bus/fsl-mc: Add a new parameter to dprc_scan_objects function

[Diana Craciun]

Prepare the dprc_scan_objects function to be used by
the VFIO mc driver code. The function is used to scan the mc
objects by the bus driver. The same functionality is
needed by the VFIO mc driver, but in this case the
interrupt configuration is delayed until the userspace
configures the interrupts. In order to use the same function
in both drivers add a new parameter.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 54c576d68122..3d850515e3e5 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -221,6 +221,8 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
+ * @alloc_interrupts: if true the function allocates the interrupt pool,
+ * otherwise the interrupt allocation is delayed
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -234,7 +236,8 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
+static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
+   bool alloc_interrupts)
 {
int num_child_objects;
int dprc_get_obj_failures;
@@ -322,7 +325,7 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev)
 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
}
 
-   if (!mc_bus->irq_resources) {
+   if (alloc_interrupts && !mc_bus->irq_resources) {
error = fsl_mc_populate_irq_pool(mc_bus,
FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
if (error < 0)
@@ -362,7 +365,7 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev);
+   error = dprc_scan_objects(mc_bus_dev, true);
mutex_unlock(_bus->scan_mutex);
if (error < 0) {
fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
@@ -432,7 +435,7 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
  DPRC_IRQ_EVENT_OBJ_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_CREATED)) {
 
-   error = dprc_scan_objects(mc_dev);
+   error = dprc_scan_objects(mc_dev, true);
if (error < 0) {
/*
 * If the error is -ENXIO, we ignore it, as it indicates
-- 
2.17.1

[PATCH v4 01/13] bus/fsl-mc: Do no longer export the total number of irqs outside dprc_scan_objects

[Diana Craciun]

The total number of interrupts is only used for some checks
outside the dprc_scan_objects function. Furthermore, in some
situations the check is made twice. Move the bounds check inside
the function for all situations.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 30 ++
 1 file changed, 10 insertions(+), 20 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 2a473c09bc33..54c576d68122 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -3,6 +3,7 @@
  * Freescale data path resource container (DPRC) driver
  *
  * Copyright (C) 2014-2016 Freescale Semiconductor, Inc.
+ * Copyright 2019-2020 NXP
  * Author: German Rivera 
  *
  */
@@ -220,8 +221,6 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * dprc_scan_objects - Discover objects in a DPRC
  *
  * @mc_bus_dev: pointer to the fsl-mc device that represents a DPRC object
- * @total_irq_count: If argument is provided the function populates the
- * total number of IRQs created by objects in the DPRC.
  *
  * Detects objects added and removed from a DPRC and synchronizes the
  * state of the Linux bus driver, MC by adding and removing
@@ -235,8 +234,7 @@ static void dprc_add_new_devices(struct fsl_mc_device 
*mc_bus_dev,
  * populated before they can get allocation requests from probe callbacks
  * of the device drivers for the non-allocatable devices.
  */
-static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev,
-unsigned int *total_irq_count)
+static int dprc_scan_objects(struct fsl_mc_device *mc_bus_dev)
 {
int num_child_objects;
int dprc_get_obj_failures;
@@ -317,22 +315,21 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
 * Allocate IRQ's before binding the scanned devices with their
 * respective drivers.
 */
-   if (dev_get_msi_domain(_bus_dev->dev) && !mc_bus->irq_resources) {
+   if (dev_get_msi_domain(_bus_dev->dev)) {
if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
dev_warn(_bus_dev->dev,
 "IRQs needed (%u) exceed IRQs preallocated 
(%u)\n",
 irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
}
 
-   error = fsl_mc_populate_irq_pool(mc_bus,
+   if (!mc_bus->irq_resources) {
+   error = fsl_mc_populate_irq_pool(mc_bus,
FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
-   if (error < 0)
-   return error;
+   if (error < 0)
+   return error;
+   }
}
 
-   if (total_irq_count)
-   *total_irq_count = irq_count;
-
dprc_remove_devices(mc_bus_dev, child_obj_desc_array,
num_child_objects);
 
@@ -365,7 +362,7 @@ static int dprc_scan_container(struct fsl_mc_device 
*mc_bus_dev)
 * Discover objects in the DPRC:
 */
mutex_lock(_bus->scan_mutex);
-   error = dprc_scan_objects(mc_bus_dev, NULL);
+   error = dprc_scan_objects(mc_bus_dev);
mutex_unlock(_bus->scan_mutex);
if (error < 0) {
fsl_mc_cleanup_all_resource_pools(mc_bus_dev);
@@ -434,9 +431,8 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
  DPRC_IRQ_EVENT_CONTAINER_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_DESTROYED |
  DPRC_IRQ_EVENT_OBJ_CREATED)) {
-   unsigned int irq_count;
 
-   error = dprc_scan_objects(mc_dev, _count);
+   error = dprc_scan_objects(mc_dev);
if (error < 0) {
/*
 * If the error is -ENXIO, we ignore it, as it indicates
@@ -451,12 +447,6 @@ static irqreturn_t dprc_irq0_handler_thread(int irq_num, 
void *arg)
 
goto out;
}
-
-   if (irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS) {
-   dev_warn(dev,
-"IRQs needed (%u) exceed IRQs preallocated 
(%u)\n",
-irq_count, FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
-   }
}
 
 out:
-- 
2.17.1

Re: [PATCH v3 03/13] bus/fsl-mc: add support for 'driver_override' in the mc-bus

[Diana Craciun OSS]

Hi Andrew,

I apologize, I somehow missed you previous comment.

OK, I see that there are a number of drivers using the same code. In 
case we share the code, probably we need to store the driver_override 
value in the struct device (not in the particular driver device 
structure as it is done now).


Diana


On 7/6/2020 4:57 PM, Andrew Lunn wrote:

Hi Diana

On Mon, Jul 06, 2020 at 03:42:33PM +0300, Diana Craciun wrote:

From: Bharat Bhushan 

This patch is required for vfio-fsl-mc meta driver to successfully bind
layerscape container devices for device passthrough. This patch adds
a mechanism to allow a layerscape device to specify a driver rather than
a layerscape driver provide a device match.

Example to allow a device (dprc.1) to specifically bind
with driver (vfio-fsl-mc):-
  - echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.1/driver_override
  - echo dprc.1 > /sys/bus/fsl-mc/drivers/fsl_mc_dprc/unbind
  - echo dprc.1 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

Something i asked before, why is this buried in a driver, when it
could be put in the driver core. This is not the only driver doing
this, so it does make sense to share the code and bugs.

   Andrew

Re: [PATCH v3 13/13] bus/fsl-mc: Add a new version for dprc_get_obj_region command

[Diana Craciun OSS]

On 7/6/2020 4:14 PM, Laurentiu Tudor wrote:


On 7/6/2020 3:42 PM, Diana Craciun wrote:

From: Diana Craciun 

The region size reported by the firmware for mc and software
portals was less than allocated by the hardware. This may be
problematic when mmapping the region in user space because the
region size is less than page size. However the size as reserved
by the hardware is 64K.

Should we also mention in the commit msg that this shows up when
compiling the kernel with 64K page size support, or it's obvious enough?


It's not that obvious. In case of 4K worked by chance because the 
reported region size was 4K. Maybe it worth mentioning.





Signed-off-by: Diana Craciun 
---
  drivers/bus/fsl-mc/dprc.c   | 38 ++---
  drivers/bus/fsl-mc/fsl-mc-private.h |  3 +++
  2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 3f08752c2c19..ba292c56fe19 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -536,20 +536,30 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
return err;
}
  
-	/**

-* MC API version 6.3 introduced a new field to the region
-* descriptor: base_address. If the older API is in use then the base
-* address is set to zero to indicate it needs to be obtained elsewhere
-* (typically the device tree).
-*/
-   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
-cmd_flags, token);
-   else
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
-cmd_flags, token);
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 6)) 
{
+   /**
+* MC API version 6.6 changed the size of the MC portals and 
software
+* portals to 64K (as implemented by hardware). If older API is 
in use the
+* size reported is less (64 bytes for mc portals and 4K for 
software
+* portals).
+*/

Here and below, there's no need to use kernel-doc style comments. And a
nit: there's an extra blank line here.

---
Best Regards, Laurentiu


+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V3,
+ cmd_flags, token);
+
+   } else if (dprc_major_ver == 6 && dprc_minor_ver >= 3) {
+   /**
+* MC API version 6.3 introduced a new field to the region
+* descriptor: base_address. If the older API is in use then 
the base
+* address is set to zero to indicate it needs to be obtained 
elsewhere
+* (typically the device tree).
+*/
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
+ cmd_flags, token);
+   } else {
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
+ cmd_flags, token);
+   }
  
  	cmd_params = (struct dprc_cmd_get_obj_region *)cmd.params;

cmd_params->obj_id = cpu_to_le32(obj_id);
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index e6fcff12c68d..8d65273a78d7 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -80,10 +80,12 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
  /* DPRC command versioning */
  #define DPRC_CMD_BASE_VERSION 1
  #define DPRC_CMD_2ND_VERSION  2
+#define DPRC_CMD_3RD_VERSION   3
  #define DPRC_CMD_ID_OFFSET4
  
  #define DPRC_CMD(id)	(((id) << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)

  #define DPRC_CMD_V2(id)   (((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_2ND_VERSION)
+#define DPRC_CMD_V3(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_3RD_VERSION)
  
  /* DPRC command IDs */

  #define DPRC_CMDID_CLOSEDPRC_CMD(0x800)
@@ -105,6 +107,7 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
  #define DPRC_CMDID_GET_OBJ  DPRC_CMD(0x15A)
  #define DPRC_CMDID_GET_OBJ_REG  DPRC_CMD(0x15E)
  #define DPRC_CMDID_GET_OBJ_REG_V2   DPRC_CMD_V2(0x15E)
+#define DPRC_CMDID_GET_OBJ_REG_V3   DPRC_CMD_V3(0x15E)
  #define DPRC_CMDID_SET_OBJ_IRQ  DPRC_CMD(0x15F)
  
  #define DPRC_CMDID_GET_CONNECTION   DPRC_CMD(0x16C)

[PATCH v3 4/9] vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call

[Diana Craciun]

Expose to userspace information about the memory regions.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 77 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h | 19 ++
 2 files changed, 95 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 937b6eddc71a..10bd9f78b8de 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,16 +17,72 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int count = mc_dev->obj_desc.region_count;
+   int i;
+
+   vdev->regions = kcalloc(count, sizeof(struct vfio_fsl_mc_region),
+   GFP_KERNEL);
+   if (!vdev->regions)
+   return -ENOMEM;
+
+   for (i = 0; i < count; i++) {
+   struct resource *res = _dev->regions[i];
+
+   vdev->regions[i].addr = res->start;
+   vdev->regions[i].size = resource_size(res);
+   vdev->regions[i].flags = 0;
+   }
+
+   vdev->num_regions = mc_dev->obj_desc.region_count;
+   return 0;
+}
+
+static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
+{
+   vdev->num_regions = 0;
+   kfree(vdev->regions);
+}
+
 static int vfio_fsl_mc_open(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
+
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
+   mutex_lock(>driver_lock);
+   if (!vdev->refcnt) {
+   ret = vfio_fsl_mc_regions_init(vdev);
+   if (ret)
+   goto err_reg_init;
+   }
+   vdev->refcnt++;
+
+   mutex_unlock(>driver_lock);
+
return 0;
+
+err_reg_init:
+   mutex_unlock(>driver_lock);
+   module_put(THIS_MODULE);
+   return ret;
 }
 
 static void vfio_fsl_mc_release(void *device_data)
 {
+   struct vfio_fsl_mc_device *vdev = device_data;
+
+   mutex_lock(>driver_lock);
+
+   if (!(--vdev->refcnt))
+   vfio_fsl_mc_regions_cleanup(vdev);
+
+   mutex_unlock(>driver_lock);
+
module_put(THIS_MODULE);
 }
 
@@ -59,7 +115,25 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-   return -ENOTTY;
+   struct vfio_region_info info;
+
+   minsz = offsetofend(struct vfio_region_info, offset);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= vdev->num_regions)
+   return -EINVAL;
+
+   /* map offset to the physical address  */
+   info.offset = VFIO_FSL_MC_INDEX_TO_OFFSET(info.index);
+   info.size = vdev->regions[info.index].size;
+   info.flags = vdev->regions[info.index].flags;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
@@ -201,6 +275,7 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
vfio_iommu_group_put(group, dev);
return ret;
}
+   mutex_init(>driver_lock);
 
return ret;
 }
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
index 37d61eaa58c8..818dfd3df4db 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_private.h
@@ -7,9 +7,28 @@
 #ifndef VFIO_FSL_MC_PRIVATE_H
 #define VFIO_FSL_MC_PRIVATE_H
 
+#define VFIO_FSL_MC_OFFSET_SHIFT40
+#define VFIO_FSL_MC_OFFSET_MASK (((u64)(1) << VFIO_FSL_MC_OFFSET_SHIFT) - 1)
+
+#define VFIO_FSL_MC_OFFSET_TO_INDEX(off) ((off) >> VFIO_FSL_MC_OFFSET_SHIFT)
+
+#define VFIO_FSL_MC_INDEX_TO_OFFSET(index) \
+   ((u64)(index) << VFIO_FSL_MC_OFFSET_SHIFT)
+
+struct vfio_fsl_mc_region {
+   u32 flags;
+   u32 type;
+   u64 addr;
+   resource_size_t size;
+};
+
 struct vfio_fsl_mc_device {
struct fsl_mc_device*mc_dev;
struct notifier_blocknb;
+   int refcnt;
+   u32 num_regions;
+   struct vfio_fsl_mc_region   *regions;
+   struct mutex driver_lock;
 };
 
 #endif /* VFIO_FSL_MC_PRIVATE_H */
-- 
2.17.1

[PATCH v3 8/9] vfio/fsl-mc: trigger an interrupt via eventfd

[Diana Craciun]

This patch allows to set an eventfd for fsl-mc device interrupts
and also to trigger the interrupt eventfd from userspace for testing.

All fsl-mc device interrupts are MSIs. The MSIs are allocated from
the MSI domain only once per DPRC and used by all the DPAA2 objects.
The interrupts are managed by the DPRC in a pool of interrupts. Each
device requests interrupts from this pool. The pool is allocated
when the first virtual device is setting the interrupts.
The pool of interrupts is protected by a lock.

The DPRC has an interrupt of its own which indicates if the DPRC
contents have changed. However, currently, the contents of a DPRC
assigned to the guest cannot be changed at runtime, so this interrupt
is not configured.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c |  18 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 160 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  11 +-
 3 files changed, 186 insertions(+), 3 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 2ce17d297145..0c2d4f11dc43 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -146,12 +146,28 @@ static int vfio_fsl_mc_open(void *device_data)
 static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
+   int ret;
 
mutex_lock(>reflck->lock);
 
-   if (!(--vdev->refcnt))
+   if (!(--vdev->refcnt)) {
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct device *cont_dev = fsl_mc_cont_dev(_dev->dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(cont_dev);
+
vfio_fsl_mc_regions_cleanup(vdev);
 
+   /* reset the device before cleaning up the interrupts */
+   ret = dprc_reset_container(mc_cont->mc_io, 0,
+ mc_cont->mc_handle,
+ mc_cont->obj_desc.id,
+ DPRC_RESET_OPTION_NON_RECURSIVE);
+
+   vfio_fsl_mc_irqs_cleanup(vdev);
+
+   fsl_mc_cleanup_irq_pool(mc_cont);
+   }
+
mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
index 058aa97aa54a..409f3507fcf3 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -29,12 +29,149 @@ static int vfio_fsl_mc_irq_unmask(struct 
vfio_fsl_mc_device *vdev,
return -EINVAL;
 }
 
+int vfio_fsl_mc_irqs_allocate(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   struct vfio_fsl_mc_irq *mc_irq;
+   int irq_count;
+   int ret, i;
+
+/* Device does not support any interrupt */
+   if (mc_dev->obj_desc.irq_count == 0)
+   return 0;
+
+   /* interrupts were already allocated for this device */
+   if (vdev->mc_irqs)
+   return 0;
+
+   irq_count = mc_dev->obj_desc.irq_count;
+
+   mc_irq = kcalloc(irq_count, sizeof(*mc_irq), GFP_KERNEL);
+   if (!mc_irq)
+   return -ENOMEM;
+
+   /* Allocate IRQs */
+   ret = fsl_mc_allocate_irqs(mc_dev);
+   if (ret) {
+   kfree(mc_irq);
+   return ret;
+   }
+
+   for (i = 0; i < irq_count; i++) {
+   mc_irq[i].count = 1;
+   mc_irq[i].flags = VFIO_IRQ_INFO_EVENTFD;
+   }
+
+   vdev->mc_irqs = mc_irq;
+
+   return 0;
+}
+
+static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg)
+{
+   struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg;
+
+   eventfd_signal(mc_irq->trigger, 1);
+   return IRQ_HANDLED;
+}
+
+static int vfio_set_trigger(struct vfio_fsl_mc_device *vdev,
+  int index, int fd)
+{
+   struct vfio_fsl_mc_irq *irq = >mc_irqs[index];
+   struct eventfd_ctx *trigger;
+   int hwirq;
+   int ret;
+
+   hwirq = vdev->mc_dev->irqs[index]->msi_desc->irq;
+   if (irq->trigger) {
+   free_irq(hwirq, irq);
+   kfree(irq->name);
+   eventfd_ctx_put(irq->trigger);
+   irq->trigger = NULL;
+   }
+
+   if (fd < 0) /* Disable only */
+   return 0;
+
+   irq->name = kasprintf(GFP_KERNEL, "vfio-irq[%d](%s)",
+   hwirq, dev_name(>mc_dev->dev));
+   if (!irq->name)
+   return -ENOMEM;
+
+   trigger = eventfd_ctx_fdget(fd);
+   if (IS_ERR(trigger)) {
+   kfree(irq->name);
+   return PTR_ERR(trigger);
+   }
+
+   irq->trigger = trigger;
+
+   ret = request_irq(hwirq, vfio_fsl_mc_irq_handler, 0,
+ irq->name, irq);
+   if (ret

[PATCH v3 7/9] vfio/fsl-mc: Add irq infrastructure for fsl-mc devices

[Diana Craciun]

This patch adds the skeleton for interrupt support
for fsl-mc devices. The interrupts are not yet functional,
the functionality will be added by subsequent patches.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/Makefile  |  2 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 71 ++-
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 63 
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  5 ++
 4 files changed, 138 insertions(+), 3 deletions(-)
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c

diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
index 0c6e5d2ddaae..cad6dbf0b735 100644
--- a/drivers/vfio/fsl-mc/Makefile
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
 
-vfio-fsl-mc-y := vfio_fsl_mc.o
+vfio-fsl-mc-y := vfio_fsl_mc.o vfio_fsl_mc_intr.o
 obj-$(CONFIG_VFIO_FSL_MC) += vfio-fsl-mc.o
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 275c4283e1bc..2ce17d297145 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -208,11 +208,75 @@ static long vfio_fsl_mc_ioctl(void *device_data, unsigned 
int cmd,
}
case VFIO_DEVICE_GET_IRQ_INFO:
{
-   return -ENOTTY;
+   struct vfio_irq_info info;
+
+   minsz = offsetofend(struct vfio_irq_info, count);
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   if (info.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   info.flags = VFIO_IRQ_INFO_EVENTFD;
+   info.count = 1;
+
+   return copy_to_user((void __user *)arg, , minsz);
}
case VFIO_DEVICE_SET_IRQS:
{
-   return -ENOTTY;
+   struct vfio_irq_set hdr;
+   u8 *data = NULL;
+   int ret = 0;
+
+   minsz = offsetofend(struct vfio_irq_set, count);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (hdr.argsz < minsz)
+   return -EINVAL;
+
+   if (hdr.index >= mc_dev->obj_desc.irq_count)
+   return -EINVAL;
+
+   if (hdr.start != 0 || hdr.count > 1)
+   return -EINVAL;
+
+   if (hdr.count == 0 &&
+   (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE) ||
+   !(hdr.flags & VFIO_IRQ_SET_ACTION_TRIGGER)))
+   return -EINVAL;
+
+   if (hdr.flags & ~(VFIO_IRQ_SET_DATA_TYPE_MASK |
+ VFIO_IRQ_SET_ACTION_TYPE_MASK))
+   return -EINVAL;
+
+   if (!(hdr.flags & VFIO_IRQ_SET_DATA_NONE)) {
+   size_t size;
+
+   if (hdr.flags & VFIO_IRQ_SET_DATA_BOOL)
+   size = sizeof(uint8_t);
+   else if (hdr.flags & VFIO_IRQ_SET_DATA_EVENTFD)
+   size = sizeof(int32_t);
+   else
+   return -EINVAL;
+
+   if (hdr.argsz - minsz < hdr.count * size)
+   return -EINVAL;
+
+   data = memdup_user((void __user *)(arg + minsz),
+  hdr.count * size);
+   if (IS_ERR(data))
+   return PTR_ERR(data);
+   }
+
+   ret = vfio_fsl_mc_set_irqs_ioctl(vdev, hdr.flags,
+hdr.index, hdr.start,
+hdr.count, data);
+   return ret;
}
case VFIO_DEVICE_RESET:
{
@@ -336,6 +400,9 @@ static int vfio_fsl_mc_init_device(struct 
vfio_fsl_mc_device *vdev)
struct fsl_mc_device *mc_dev = vdev->mc_dev;
int ret;
 
+   /* innherit the msi domain from parent */
+   dev_set_msi_domain(_dev->dev, 
dev_get_msi_domain(mc_dev->dev.parent));
+
/* Non-dprc devices share mc_io from parent */
if (!is_fsl_mc_bus_dprc(mc_dev)) {
struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
new file mode 100644
index ..058aa97aa54a
--- /dev/null
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
@@ -0,0 +1,63 @@
+// SPDX-License-Identifier: (GPL-2.0+ OR BSD-3-Clause)
+/*
+ * Copyright 2013-2016 Freescale Semiconductor Inc.
+ * Copyright 2019 NXP
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#incl

[PATCH v3 6/9] vfio/fsl-mc: Added lock support in preparation for interrupt handling

[Diana Craciun]

From: Diana Craciun 

Only the DPRC object allocates interrupts from the MSI
interrupt domain. The interrupts are managed by the DPRC in
a pool of interrupts. The access to this pool of interrupts
has to be protected with a lock.
This patch extends the current lock implementation to have a
lock per DPRC.

Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 90 +--
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  8 +-
 2 files changed, 91 insertions(+), 7 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 37ccc7d2acbc..275c4283e1bc 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -17,6 +17,76 @@
 
 static struct fsl_mc_driver vfio_fsl_mc_driver;
 
+static DEFINE_MUTEX(reflck_lock);
+
+static void vfio_fsl_mc_reflck_get(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_get(>kref);
+}
+
+static void vfio_fsl_mc_reflck_release(struct kref *kref)
+{
+   struct vfio_fsl_mc_reflck *reflck = container_of(kref,
+ struct vfio_fsl_mc_reflck,
+ kref);
+
+   kfree(reflck);
+   mutex_unlock(_lock);
+}
+
+static void vfio_fsl_mc_reflck_put(struct vfio_fsl_mc_reflck *reflck)
+{
+   kref_put_mutex(>kref, vfio_fsl_mc_reflck_release, _lock);
+}
+
+static struct vfio_fsl_mc_reflck *vfio_fsl_mc_reflck_alloc(void)
+{
+   struct vfio_fsl_mc_reflck *reflck;
+
+   reflck = kzalloc(sizeof(*reflck), GFP_KERNEL);
+   if (!reflck)
+   return ERR_PTR(-ENOMEM);
+
+   kref_init(>kref);
+   mutex_init(>lock);
+
+   return reflck;
+}
+
+static int vfio_fsl_mc_reflck_attach(struct vfio_fsl_mc_device *vdev)
+{
+   int ret = 0;
+
+   mutex_lock(_lock);
+   if (is_fsl_mc_bus_dprc(vdev->mc_dev)) {
+   vdev->reflck = vfio_fsl_mc_reflck_alloc();
+   } else {
+   struct device *mc_cont_dev = vdev->mc_dev->dev.parent;
+   struct vfio_device *device;
+   struct vfio_fsl_mc_device *cont_vdev;
+
+   device = vfio_device_get_from_dev(mc_cont_dev);
+   if (!device) {
+   ret = -ENODEV;
+   goto unlock;
+   }
+
+   cont_vdev = vfio_device_data(device);
+   if (!cont_vdev->reflck) {
+   vfio_device_put(device);
+   ret = -ENODEV;
+   goto unlock;
+   }
+   vfio_fsl_mc_reflck_get(cont_vdev->reflck);
+   vdev->reflck = cont_vdev->reflck;
+   vfio_device_put(device);
+   }
+
+unlock:
+   mutex_unlock(_lock);
+   return ret;
+}
+
 static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device *vdev)
 {
struct fsl_mc_device *mc_dev = vdev->mc_dev;
@@ -55,7 +125,7 @@ static int vfio_fsl_mc_open(void *device_data)
if (!try_module_get(THIS_MODULE))
return -ENODEV;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
if (!vdev->refcnt) {
ret = vfio_fsl_mc_regions_init(vdev);
if (ret)
@@ -63,12 +133,12 @@ static int vfio_fsl_mc_open(void *device_data)
}
vdev->refcnt++;
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
return 0;
 
 err_reg_init:
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
module_put(THIS_MODULE);
return ret;
 }
@@ -77,12 +147,12 @@ static void vfio_fsl_mc_release(void *device_data)
 {
struct vfio_fsl_mc_device *vdev = device_data;
 
-   mutex_lock(>driver_lock);
+   mutex_lock(>reflck->lock);
 
if (!(--vdev->refcnt))
vfio_fsl_mc_regions_cleanup(vdev);
 
-   mutex_unlock(>driver_lock);
+   mutex_unlock(>reflck->lock);
 
module_put(THIS_MODULE);
 }
@@ -325,12 +395,18 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
 
+   ret = vfio_fsl_mc_reflck_attach(vdev);
+   if (ret) {
+   vfio_iommu_group_put(group, dev);
+   return ret;
+   }
+
ret = vfio_fsl_mc_init_device(vdev);
if (ret < 0) {
+   vfio_fsl_mc_reflck_put(vdev->reflck);
vfio_iommu_group_put(group, dev);
return ret;
}
-   mutex_init(>driver_lock);
 
return ret;
 }
@@ -374,6 +450,8 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (vdev->nb.notifier_call)
bus_unregister_notifier(_mc_bus_type, >nb);
 
+   vfio_fsl_mc_reflck_put(vdev->reflck);
+
if (is_fsl_mc_bus_dprc(mc_dev))
vfio_fsl_mc_cleanup_dprc(vdev->mc_dev);
 
diff --git a/drivers/vfio

[PATCH v3 2/9] vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind

[Diana Craciun]

The DPRC (Data Path Resource Container) device is a bus device and has
child devices attached to it. When the vfio-fsl-mc driver is probed
the DPRC is scanned and the child devices discovered and initialized.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 106 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
 2 files changed, 107 insertions(+)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 8b53c2a25b32..ad8d06cceb71 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -15,6 +15,8 @@
 
 #include "vfio_fsl_mc_private.h"
 
+static struct fsl_mc_driver vfio_fsl_mc_driver;
+
 static int vfio_fsl_mc_open(void *device_data)
 {
if (!try_module_get(THIS_MODULE))
@@ -84,6 +86,69 @@ static const struct vfio_device_ops vfio_fsl_mc_ops = {
.mmap   = vfio_fsl_mc_mmap,
 };
 
+static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb,
+   unsigned long action, void *data)
+{
+   struct vfio_fsl_mc_device *vdev = container_of(nb,
+   struct vfio_fsl_mc_device, nb);
+   struct device *dev = data;
+   struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev);
+   struct fsl_mc_device *mc_cont = to_fsl_mc_device(mc_dev->dev.parent);
+
+   if (action == BUS_NOTIFY_ADD_DEVICE &&
+   vdev->mc_dev == mc_cont) {
+   mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s",
+   vfio_fsl_mc_ops.name);
+   dev_info(dev, "Setting driver override for device in dprc %s\n",
+dev_name(_cont->dev));
+   } else if (action == BUS_NOTIFY_BOUND_DRIVER &&
+   vdev->mc_dev == mc_cont) {
+   struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(dev->driver);
+
+   if (mc_drv && mc_drv != _fsl_mc_driver)
+   dev_warn(dev, "Object %s bound to driver %s while DPRC 
bound to vfio-fsl-mc\n",
+dev_name(dev), mc_drv->driver.name);
+   }
+
+   return 0;
+}
+
+static int vfio_fsl_mc_init_device(struct vfio_fsl_mc_device *vdev)
+{
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int ret;
+
+   /* Non-dprc devices share mc_io from parent */
+   if (!is_fsl_mc_bus_dprc(mc_dev)) {
+   struct fsl_mc_device *mc_cont = 
to_fsl_mc_device(mc_dev->dev.parent);
+
+   mc_dev->mc_io = mc_cont->mc_io;
+   return 0;
+   }
+
+   vdev->nb.notifier_call = vfio_fsl_mc_bus_notifier;
+   ret = bus_register_notifier(_mc_bus_type, >nb);
+   if (ret)
+   return ret;
+
+   /* open DPRC, allocate a MC portal */
+   ret = dprc_setup(mc_dev);
+   if (ret < 0) {
+   dev_err(_dev->dev, "Failed to setup DPRC (error = %d)\n", 
ret);
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   return ret;
+   }
+
+   ret = dprc_scan_container(mc_dev, false);
+   if (ret < 0) {
+   dev_err(_dev->dev, "Container scanning failed: %d\n", ret);
+   bus_unregister_notifier(_mc_bus_type, >nb);
+   dprc_cleanup(mc_dev);
+   }
+
+   return ret;
+}
+
 static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
 {
struct iommu_group *group;
@@ -112,9 +177,42 @@ static int vfio_fsl_mc_probe(struct fsl_mc_device *mc_dev)
return ret;
}
 
+   ret = vfio_fsl_mc_init_device(vdev);
+   if (ret < 0) {
+   vfio_iommu_group_put(group, dev);
+   return ret;
+   }
+
return ret;
 }
 
+static int vfio_fsl_mc_device_remove(struct device *dev, void *data)
+{
+   struct fsl_mc_device *mc_dev;
+
+   WARN_ON(!dev);
+   mc_dev = to_fsl_mc_device(dev);
+   if (WARN_ON(!mc_dev))
+   return -ENODEV;
+
+   kfree(mc_dev->driver_override);
+   mc_dev->driver_override = NULL;
+
+   /*
+* The device-specific remove callback will get invoked by device_del()
+*/
+   device_del(_dev->dev);
+   put_device(_dev->dev);
+
+   return 0;
+}
+
+static void vfio_fsl_mc_cleanup_dprc(struct fsl_mc_device *mc_dev)
+{
+   device_for_each_child(_dev->dev, NULL, vfio_fsl_mc_device_remove);
+   dprc_cleanup(mc_dev);
+}
+
 static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
 {
struct vfio_fsl_mc_device *vdev;
@@ -124,6 +222,14 @@ static int vfio_fsl_mc_remove(struct fsl_mc_device *mc_dev)
if (!vdev)
return -EINVAL;
 
+   if (vdev->nb.notifier_call)
+   bus_unregister_notifier(_mc_bus_type, >nb);
+
+   if (is_fsl_mc_

[PATCH v3 9/9] vfio/fsl-mc: Add read/write support for fsl-mc devices

[Diana Craciun]

The software uses a memory-mapped I/O command interface (MC portals) to
communicate with the MC hardware. This command interface is used to
discover, enumerate, configure and remove DPAA2 objects. The DPAA2
objects use MSIs, so the command interface needs to be emulated
such that the correct MSI is configured in the hardware (the guest
has the virtual MSIs).

This patch is adding read/write support for fsl-mc devices. The mc
commands are emulated by the userspace. The host is just passing
the correct command to the hardware.

Also the current patch limits userspace to write complete
64byte command once and read 64byte response by one ioctl.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 115 +-
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |   1 +
 2 files changed, 114 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 0c2d4f11dc43..104e3bb78bf8 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -12,6 +12,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "vfio_fsl_mc_private.h"
 
@@ -105,6 +106,9 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
vdev->regions[i].size = resource_size(res);
vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_READ;
+   if (!(mc_dev->regions[i].flags & IORESOURCE_READONLY))
+   vdev->regions[i].flags |= VFIO_REGION_INFO_FLAG_WRITE;
}
 
vdev->num_regions = mc_dev->obj_desc.region_count;
@@ -113,6 +117,11 @@ static int vfio_fsl_mc_regions_init(struct 
vfio_fsl_mc_device *vdev)
 
 static void vfio_fsl_mc_regions_cleanup(struct vfio_fsl_mc_device *vdev)
 {
+   int i;
+
+   for (i = 0; i < vdev->num_regions; i++)
+   iounmap(vdev->regions[i].ioaddr);
+
vdev->num_regions = 0;
kfree(vdev->regions);
 }
@@ -306,13 +315,115 @@ static long vfio_fsl_mc_ioctl(void *device_data, 
unsigned int cmd,
 static ssize_t vfio_fsl_mc_read(void *device_data, char __user *buf,
size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   unsigned int index = VFIO_FSL_MC_OFFSET_TO_INDEX(*ppos);
+   loff_t off = *ppos & VFIO_FSL_MC_OFFSET_MASK;
+   struct vfio_fsl_mc_region *region;
+   u64 data[8];
+   int i;
+
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   region = >regions[index];
+
+   if (!(region->flags & VFIO_REGION_INFO_FLAG_READ))
+   return -EINVAL;
+
+   if (!region->ioaddr) {
+   region->ioaddr = ioremap(region->addr, region->size);
+   if (!region->ioaddr)
+   return -ENOMEM;
+   }
+
+   if (count != 64 || off != 0)
+   return -EINVAL;
+
+   for (i = 7; i >= 0; i--)
+   data[i] = readq(region->ioaddr + i * sizeof(uint64_t));
+
+   if (copy_to_user(buf, data, 64))
+   return -EFAULT;
+
+   return count;
+}
+
+#define MC_CMD_COMPLETION_TIMEOUT_MS5000
+#define MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS500
+
+static int vfio_fsl_mc_send_command(void __iomem *ioaddr, uint64_t *cmd_data)
+{
+   int i;
+   enum mc_cmd_status status;
+   unsigned long timeout_usecs = MC_CMD_COMPLETION_TIMEOUT_MS * 1000;
+
+   /* Write at command parameter into portal */
+   for (i = 7; i >= 1; i--)
+   writeq_relaxed(cmd_data[i], ioaddr + i * sizeof(uint64_t));
+
+   /* Write command header in the end */
+   writeq(cmd_data[0], ioaddr);
+
+   /* Wait for response before returning to user-space
+* This can be optimized in future to even prepare response
+* before returning to user-space and avoid read ioctl.
+*/
+   for (;;) {
+   u64 header;
+   struct mc_cmd_header *resp_hdr;
+
+   header = cpu_to_le64(readq_relaxed(ioaddr));
+
+   resp_hdr = (struct mc_cmd_header *)
+   status = (enum mc_cmd_status)resp_hdr->status;
+   if (status != MC_CMD_STATUS_READY)
+   break;
+
+   udelay(MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS);
+   timeout_usecs -= MC_CMD_COMPLETION_POLLING_MAX_SLEEP_USECS;
+   if (timeout_usecs == 0)
+   return -ETIMEDOUT;
+   }
+
+   return 0;
 }
 
 static ssize_t vfio_fsl_mc_write(void *device_data, const char __user *buf,
 size_t count, loff_t *ppos)
 {
-   return -EINVAL;
+   st

[PATCH v3 1/9] vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

[Diana Craciun]

From: Bharat Bhushan 

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

This patch adds the infrastructure for VFIO support for fsl-mc
devices. Subsequent patches will add support for binding and secure
assigning these devices using VFIO.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 ++
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 160 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  14 ++
 include/uapi/linux/vfio.h |   1 +
 8 files changed, 196 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 1d4aa7f942de..18af3e530f8f 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -18015,6 +18015,12 @@ F: drivers/vfio/
 F: include/linux/vfio.h
 F: include/uapi/linux/vfio.h
 
+VFIO FSL-MC DRIVER
+M: Diana Craciun 
+L: k...@vger.kernel.org
+S: Maintained
+F: drivers/vfio/fsl-mc/
+
 VFIO MEDIATED DEVICE DRIVERS
 M: Kirti Wankhede 
 L: k...@vger.kernel.org
diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
index fd17db9b432f..5533df91b257 100644
--- a/drivers/vfio/Kconfig
+++ b/drivers/vfio/Kconfig
@@ -47,4 +47,5 @@ menuconfig VFIO_NOIOMMU
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
 source "drivers/vfio/mdev/Kconfig"
+source "drivers/vfio/fsl-mc/Kconfig"
 source "virt/lib/Kconfig"
diff --git a/drivers/vfio/Makefile b/drivers/vfio/Makefile
index de67c4725cce..fee73f3d9480 100644
--- a/drivers/vfio/Makefile
+++ b/drivers/vfio/Makefile
@@ -9,3 +9,4 @@ obj-$(CONFIG_VFIO_SPAPR_EEH) += vfio_spapr_eeh.o
 obj-$(CONFIG_VFIO_PCI) += pci/
 obj-$(CONFIG_VFIO_PLATFORM) += platform/
 obj-$(CONFIG_VFIO_MDEV) += mdev/
+obj-$(CONFIG_VFIO_FSL_MC) += fsl-mc/
diff --git a/drivers/vfio/fsl-mc/Kconfig b/drivers/vfio/fsl-mc/Kconfig
new file mode 100644
index ..b1a527d6b6f2
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Kconfig
@@ -0,0 +1,9 @@
+config VFIO_FSL_MC
+   tristate "VFIO support for QorIQ DPAA2 fsl-mc bus devices"
+   depends on VFIO && FSL_MC_BUS && EVENTFD
+   help
+ Driver to enable support for the VFIO QorIQ DPAA2 fsl-mc
+ (Management Complex) devices. This is required to passthrough
+ fsl-mc bus devices using the VFIO framework.
+
+ If you don't know what to do here, say N.
diff --git a/drivers/vfio/fsl-mc/Makefile b/drivers/vfio/fsl-mc/Makefile
new file mode 100644
index ..0c6e5d2ddaae
--- /dev/null
+++ b/drivers/vfio/fsl-mc/Makefile
@@ -0,0 +1,4 @@
+# SPDX-Lic

[PATCH v3 5/9] vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions

[Diana Craciun]

Allow userspace to mmap device regions for direct access of
fsl-mc devices.

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 59 +--
 1 file changed, 57 insertions(+), 2 deletions(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index 10bd9f78b8de..37ccc7d2acbc 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,7 +33,8 @@ static int vfio_fsl_mc_regions_init(struct vfio_fsl_mc_device 
*vdev)
 
vdev->regions[i].addr = res->start;
vdev->regions[i].size = resource_size(res);
-   vdev->regions[i].flags = 0;
+   vdev->regions[i].flags = VFIO_REGION_INFO_FLAG_MMAP;
+   vdev->regions[i].type = mc_dev->regions[i].flags & 
IORESOURCE_BITS;
}
 
vdev->num_regions = mc_dev->obj_desc.region_count;
@@ -164,9 +165,63 @@ static ssize_t vfio_fsl_mc_write(void *device_data, const 
char __user *buf,
return -EINVAL;
 }
 
+static int vfio_fsl_mc_mmap_mmio(struct vfio_fsl_mc_region region,
+struct vm_area_struct *vma)
+{
+   u64 size = vma->vm_end - vma->vm_start;
+   u64 pgoff, base;
+   u8 region_cacheable;
+
+   pgoff = vma->vm_pgoff &
+   ((1U << (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
+   base = pgoff << PAGE_SHIFT;
+
+   if (region.size < PAGE_SIZE || base + size > region.size)
+   return -EINVAL;
+
+   region_cacheable = (region.type & FSL_MC_REGION_CACHEABLE) && 
(region.type & FSL_MC_REGION_SHAREABLE);
+   if (!region_cacheable)
+   vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
+
+   vma->vm_pgoff = (region.addr >> PAGE_SHIFT) + pgoff;
+
+   return remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff,
+  size, vma->vm_page_prot);
+}
+
 static int vfio_fsl_mc_mmap(void *device_data, struct vm_area_struct *vma)
 {
-   return -EINVAL;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+   int index;
+
+   index = vma->vm_pgoff >> (VFIO_FSL_MC_OFFSET_SHIFT - PAGE_SHIFT);
+
+   if (vma->vm_end < vma->vm_start)
+   return -EINVAL;
+   if (vma->vm_start & ~PAGE_MASK)
+   return -EINVAL;
+   if (vma->vm_end & ~PAGE_MASK)
+   return -EINVAL;
+   if (!(vma->vm_flags & VM_SHARED))
+   return -EINVAL;
+   if (index >= vdev->num_regions)
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_MMAP))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_READ)
+   && (vma->vm_flags & VM_READ))
+   return -EINVAL;
+
+   if (!(vdev->regions[index].flags & VFIO_REGION_INFO_FLAG_WRITE)
+   && (vma->vm_flags & VM_WRITE))
+   return -EINVAL;
+
+   vma->vm_private_data = mc_dev;
+
+   return vfio_fsl_mc_mmap_mmio(vdev->regions[index], vma);
 }
 
 static const struct vfio_device_ops vfio_fsl_mc_ops = {
-- 
2.17.1

[PATCH v3 3/9] vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl

[Diana Craciun]

Allow userspace to get fsl-mc device info (number of regions
and irqs).

Signed-off-by: Bharat Bhushan 
Signed-off-by: Diana Craciun 
---
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 21 -
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c 
b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
index ad8d06cceb71..937b6eddc71a 100644
--- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c
+++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c
@@ -33,10 +33,29 @@ static void vfio_fsl_mc_release(void *device_data)
 static long vfio_fsl_mc_ioctl(void *device_data, unsigned int cmd,
  unsigned long arg)
 {
+   unsigned long minsz;
+   struct vfio_fsl_mc_device *vdev = device_data;
+   struct fsl_mc_device *mc_dev = vdev->mc_dev;
+
switch (cmd) {
case VFIO_DEVICE_GET_INFO:
{
-   return -ENOTTY;
+   struct vfio_device_info info;
+
+   minsz = offsetofend(struct vfio_device_info, num_irqs);
+
+   if (copy_from_user(, (void __user *)arg, minsz))
+   return -EFAULT;
+
+   if (info.argsz < minsz)
+   return -EINVAL;
+
+   info.flags = VFIO_DEVICE_FLAGS_FSL_MC;
+   info.num_regions = mc_dev->obj_desc.region_count;
+   info.num_irqs = mc_dev->obj_desc.irq_count;
+
+   return copy_to_user((void __user *)arg, , minsz) ?
+   -EFAULT : 0;
}
case VFIO_DEVICE_GET_REGION_INFO:
{
-- 
2.17.1

[PATCH v3 0/9] vfio/fsl-mc: VFIO support for FSL-MC devices

[Diana Craciun]

DPAA2 (Data Path Acceleration Architecture) consists in
mechanisms for processing Ethernet packets, queue management,
accelerators, etc.

The Management Complex (mc) is a hardware entity that manages the DPAA2
hardware resources. It provides an object-based abstraction for software
drivers to use the DPAA2 hardware. The MC mediates operations such as
create, discover, destroy of DPAA2 objects.
The MC provides memory-mapped I/O command interfaces (MC portals) which
DPAA2 software drivers use to operate on DPAA2 objects.

A DPRC is a container object that holds other types of DPAA2 objects.
Each object in the DPRC is a Linux device and bound to a driver.
The MC-bus driver is a platform driver (different from PCI or platform
bus). The DPRC driver does runtime management of a bus instance. It
performs the initial scan of the DPRC and handles changes in the DPRC
configuration (adding/removing objects).

All objects inside a container share the same hardware isolation
context, meaning that only an entire DPRC can be assigned to
a virtual machine.
When a container is assigned to a virtual machine, all the objects
within that container are assigned to that virtual machine.
The DPRC container assigned to the virtual machine is not allowed
to change contents (add/remove objects) by the guest. The restriction
is set by the host and enforced by the mc hardware.

The DPAA2 objects can be directly assigned to the guest. However
the MC portals (the memory mapped command interface to the MC) need
to be emulated because there are commands that configure the
interrupts and the isolation IDs which are virtual in the guest.

Example:
echo vfio-fsl-mc > /sys/bus/fsl-mc/devices/dprc.2/driver_override
echo dprc.2 > /sys/bus/fsl-mc/drivers/vfio-fsl-mc/bind

The dprc.2 is bound to the VFIO driver and all the objects within
dprc.2 are going to be bound to the VFIO driver.

More details about the DPAA2 objects can be found here:
Documentation/networking/device_drivers/freescale/dpaa2/overview.rst

The patches are dependent on some changes in the mc-bus (bus/fsl-mc)
driver. The changes were needed in order to re-use code and to export
some more functions that are needed by the VFIO driver.
Currenlty the mc-bus patches are under review:
https://www.spinics.net/lists/kernel/msg3578910.html

v2 --> v3
- There is no need to align region size to page size
- read/write implemented for all DPAA2 objects
- review fixes

v1 --> v2
- Fixed the container reset, a new flag added to the firmware command
- Implement a bus notifier for setting driver_override

Bharat Bhushan (1):
  vfio/fsl-mc: Add VFIO framework skeleton for fsl-mc devices

Diana Craciun (8):
  vfio/fsl-mc: Scan DPRC objects on vfio-fsl-mc driver bind
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_INFO ioctl
  vfio/fsl-mc: Implement VFIO_DEVICE_GET_REGION_INFO ioctl call
  vfio/fsl-mc: Allow userspace to MMAP fsl-mc device MMIO regions
  vfio/fsl-mc: Added lock support in preparation for interrupt handling
  vfio/fsl-mc: Add irq infrastructure for fsl-mc devices
  vfio/fsl-mc: trigger an interrupt via eventfd
  vfio/fsl-mc: Add read/write support for fsl-mc devices

 MAINTAINERS   |   6 +
 drivers/vfio/Kconfig  |   1 +
 drivers/vfio/Makefile |   1 +
 drivers/vfio/fsl-mc/Kconfig   |   9 +
 drivers/vfio/fsl-mc/Makefile  |   4 +
 drivers/vfio/fsl-mc/vfio_fsl_mc.c | 687 ++
 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c| 221 +++
 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h |  55 ++
 include/uapi/linux/vfio.h |   1 +
 9 files changed, 985 insertions(+)
 create mode 100644 drivers/vfio/fsl-mc/Kconfig
 create mode 100644 drivers/vfio/fsl-mc/Makefile
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c
 create mode 100644 drivers/vfio/fsl-mc/vfio_fsl_mc_private.h

-- 
2.17.1

[PATCH v3 11/13] bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO

[Diana Craciun]

From: Diana Craciun 

The IRQ pool handling functions can be used by both DPRC
driver and VFIO. Adapt and export those functions.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c  |  7 +++
 drivers/bus/fsl-mc/fsl-mc-allocator.c | 12 
 drivers/bus/fsl-mc/fsl-mc-private.h   | 10 --
 include/linux/fsl/mc.h| 11 +++
 4 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index ec44d025f623..75a53c38bcd8 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -303,8 +303,8 @@ static int dprc_scan_objects(struct fsl_mc_device 
*mc_bus_dev,
}
 
if (alloc_interrupts && !mc_bus->irq_resources) {
-   error = fsl_mc_populate_irq_pool(mc_bus,
-   FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
+   error = fsl_mc_populate_irq_pool(mc_bus_dev,
+FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS);
if (error < 0)
return error;
}
@@ -758,7 +758,6 @@ static void dprc_teardown_irq(struct fsl_mc_device *mc_dev)
 int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
-   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
@@ -767,7 +766,7 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev)
return -EINVAL;
 
if (dev_get_msi_domain(_dev->dev)) {
-   fsl_mc_cleanup_irq_pool(mc_bus);
+   fsl_mc_cleanup_irq_pool(mc_dev);
dev_set_msi_domain(_dev->dev, NULL);
}
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-allocator.c 
b/drivers/bus/fsl-mc/fsl-mc-allocator.c
index cc7bb900f524..e71a6f52ea0c 100644
--- a/drivers/bus/fsl-mc/fsl-mc-allocator.c
+++ b/drivers/bus/fsl-mc/fsl-mc-allocator.c
@@ -344,7 +344,7 @@ EXPORT_SYMBOL_GPL(fsl_mc_object_free);
  * Initialize the interrupt pool associated with an fsl-mc bus.
  * It allocates a block of IRQs from the GIC-ITS.
  */
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
+int fsl_mc_populate_irq_pool(struct fsl_mc_device *mc_bus_dev,
 unsigned int irq_count)
 {
unsigned int i;
@@ -352,10 +352,14 @@ int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
struct fsl_mc_device_irq *irq_resources;
struct fsl_mc_device_irq *mc_dev_irq;
int error;
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
+   /* do nothing if the IRQ pool is already populated */
+   if (mc_bus->irq_resources)
+   return 0;
+
if (irq_count == 0 ||
irq_count > FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS)
return -EINVAL;
@@ -407,9 +411,9 @@ EXPORT_SYMBOL_GPL(fsl_mc_populate_irq_pool);
  * Teardown the interrupt pool associated with an fsl-mc bus.
  * It frees the IRQs that were allocated to the pool, back to the GIC-ITS.
  */
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus)
+void fsl_mc_cleanup_irq_pool(struct fsl_mc_device *mc_bus_dev)
 {
-   struct fsl_mc_device *mc_bus_dev = _bus->mc_dev;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_bus_dev);
struct fsl_mc_resource_pool *res_pool =
_bus->resource_pools[FSL_MC_POOL_IRQ];
 
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index fdd9e9aa6701..48255e8944bd 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -519,11 +519,6 @@ struct dpcon_cmd_set_notification {
__le64 user_ctx;
 };
 
-/**
- * Maximum number of total IRQs that can be pre-allocated for an MC bus'
- * IRQ pool
- */
-#define FSL_MC_IRQ_POOL_MAX_TOTAL_IRQS 256
 
 /**
  * struct fsl_mc_resource_pool - Pool of MC resources of a given
@@ -599,11 +594,6 @@ void fsl_mc_msi_domain_free_irqs(struct device *dev);
 int fsl_mc_find_msi_domain(struct device *mc_platform_dev,
   struct irq_domain **mc_msi_domain);
 
-int fsl_mc_populate_irq_pool(struct fsl_mc_bus *mc_bus,
-unsigned int irq_count);
-
-void fsl_mc_cleanup_irq_pool(struct fsl_mc_bus *mc_bus);
-
 int __must_check fsl_create_mc_io(struct device *dev,
  phys_addr_t mc_portal_phys_addr,
  u32 mc_portal_size,
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 7cfe3b304602..8ee4e919e860 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -508,6 +508,17 @@ int dprc_cleanup(struct fsl_mc_device *mc_dev);
 
 int dprc_setup(struct fsl_mc_device *mc_dev);
 

[PATCH v3 00/13] bus/fsl-mc: Extend mc-bus driver functionalities in preparation for mc-bus VFIO support

[Diana Craciun]

From: Diana Craciun 

The vfio-mc bus driver needs some additional services to be exported by the
mc-bus driver like:
- a way to reset the DPRC container
- driver_override support
- functions to setup/tear down a DPRC
- functions for allocating the pool of interrupts. In case of VFIO the
interrupts are not configured at probe time, but later by userspace
request

v2 -> v3
- Add a new version for dprc_get_obj_region
- Export the cacheability bus specific bits defines

v1 -> v2
- Remove driver_override propagation through various functions
- Cache the DPRC API version

The patches are related with "vfio/fsl-mc: VFIO support for FSL-MC
devices" patches, but the series were split because they are targeting
different subsystems. However, the mc-bus patches may suffer changes
when addressing the VFIO review comments.

Bharat Bhushan (3):
  bus/fsl-mc: add support for 'driver_override' in the mc-bus
  bus/fsl-mc: Add dprc-reset-container support
  bus/fsl-mc: Extend ICID size from 16bit to 32bit

Diana Craciun (10):
  bus/fsl-mc: Do no longer export the total number of irqs outside
dprc_scan_objects
  bus/fsl-mc: Add a new parameter to dprc_scan_objects function
  bus/fsl-mc: Set the QMAN/BMAN region flags
  bus/fsl-mc: Cache the DPRC API version
  bus/fsl-mc: Export a dprc scan function to be used by multiple
entities
  bus/fsl-mc: Export a cleanup function for DPRC
  bus/fsl-mc: Add a container setup function
  bus/fsl_mc: Do not rely on caller to provide non NULL mc_io
  bus/fsl-mc: Export IRQ pool handling functions to be used by VFIO
  bus/fsl-mc: Add a new version for dprc_get_obj_region command

 drivers/bus/fsl-mc/dprc-driver.c  | 181 --
 drivers/bus/fsl-mc/dprc.c | 141 
 drivers/bus/fsl-mc/fsl-mc-allocator.c |  12 +-
 drivers/bus/fsl-mc/fsl-mc-bus.c   |  64 -
 drivers/bus/fsl-mc/fsl-mc-private.h   |  31 ++---
 drivers/bus/fsl-mc/mc-io.c|   7 +-
 include/linux/fsl/mc.h|  37 +-
 7 files changed, 348 insertions(+), 125 deletions(-)

-- 
2.17.1

[PATCH v3 08/13] bus/fsl-mc: Export a cleanup function for DPRC

[Diana Craciun]

From: Diana Craciun 

Create and export a cleanup function for DPRC. The function
is used by the DPRC driver, but it will be used by the VFIO
driver as well.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc-driver.c | 52 
 include/linux/fsl/mc.h   |  2 ++
 2 files changed, 41 insertions(+), 13 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc-driver.c b/drivers/bus/fsl-mc/dprc-driver.c
index 4f7fa5127844..07012894cc9c 100644
--- a/drivers/bus/fsl-mc/dprc-driver.c
+++ b/drivers/bus/fsl-mc/dprc-driver.c
@@ -719,33 +719,25 @@ static void dprc_teardown_irq(struct fsl_mc_device 
*mc_dev)
 }
 
 /**
- * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ * dprc_cleanup - function that cleanups a DPRC
  *
  * @mc_dev: Pointer to fsl-mc device representing the DPRC
  *
- * It removes the DPRC's child objects from Linux (not from the MC) and
- * closes the DPRC device in the MC.
- * It tears down the interrupts that were configured for the DPRC device.
+ * It closes the DPRC device in the MC.
  * It destroys the interrupt pool associated with this MC bus.
  */
-static int dprc_remove(struct fsl_mc_device *mc_dev)
+
+int dprc_cleanup(struct fsl_mc_device *mc_dev)
 {
int error;
struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
 
if (!is_fsl_mc_bus_dprc(mc_dev))
return -EINVAL;
-   if (!mc_dev->mc_io)
-   return -EINVAL;
 
-   if (!mc_bus->irq_resources)
+   if (!mc_dev->mc_io)
return -EINVAL;
 
-   if (dev_get_msi_domain(_dev->dev))
-   dprc_teardown_irq(mc_dev);
-
-   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
-
if (dev_get_msi_domain(_dev->dev)) {
fsl_mc_cleanup_irq_pool(mc_bus);
dev_set_msi_domain(_dev->dev, NULL);
@@ -762,6 +754,40 @@ static int dprc_remove(struct fsl_mc_device *mc_dev)
mc_dev->mc_io = NULL;
}
 
+   return 0;
+}
+EXPORT_SYMBOL_GPL(dprc_cleanup);
+
+/**
+ * dprc_remove - callback invoked when a DPRC is being unbound from this driver
+ *
+ * @mc_dev: Pointer to fsl-mc device representing the DPRC
+ *
+ * It removes the DPRC's child objects from Linux (not from the MC) and
+ * closes the DPRC device in the MC.
+ * It tears down the interrupts that were configured for the DPRC device.
+ * It destroys the interrupt pool associated with this MC bus.
+ */
+static int dprc_remove(struct fsl_mc_device *mc_dev)
+{
+   int error;
+   struct fsl_mc_bus *mc_bus = to_fsl_mc_bus(mc_dev);
+
+   if (!is_fsl_mc_bus_dprc(mc_dev))
+   return -EINVAL;
+
+   if (!mc_bus->irq_resources)
+   return -EINVAL;
+
+   if (dev_get_msi_domain(_dev->dev))
+   dprc_teardown_irq(mc_dev);
+
+   device_for_each_child(_dev->dev, NULL, __fsl_mc_device_remove);
+
+   error = dprc_cleanup(mc_dev);
+   if (error < 0)
+   dev_err(_dev->dev, "dprc_close() failed: %d\n", error);
+
dev_info(_dev->dev, "DPRC device unbound from driver");
return 0;
 }
diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h
index 4bfe18fe8fea..ee5ffae043a3 100644
--- a/include/linux/fsl/mc.h
+++ b/include/linux/fsl/mc.h
@@ -504,6 +504,8 @@ int dprc_reset_container(struct fsl_mc_io *mc_io,
 int dprc_scan_container(struct fsl_mc_device *mc_bus_dev,
bool alloc_interrupts);
 
+int dprc_cleanup(struct fsl_mc_device *mc_dev);
+
 /*
  * Data Path Buffer Pool (DPBP) API
  * Contains initialization APIs and runtime control APIs for DPBP
-- 
2.17.1

[PATCH v3 13/13] bus/fsl-mc: Add a new version for dprc_get_obj_region command

[Diana Craciun]

From: Diana Craciun 

The region size reported by the firmware for mc and software
portals was less than allocated by the hardware. This may be
problematic when mmapping the region in user space because the
region size is less than page size. However the size as reserved
by the hardware is 64K.

Signed-off-by: Diana Craciun 
---
 drivers/bus/fsl-mc/dprc.c   | 38 ++---
 drivers/bus/fsl-mc/fsl-mc-private.h |  3 +++
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/drivers/bus/fsl-mc/dprc.c b/drivers/bus/fsl-mc/dprc.c
index 3f08752c2c19..ba292c56fe19 100644
--- a/drivers/bus/fsl-mc/dprc.c
+++ b/drivers/bus/fsl-mc/dprc.c
@@ -536,20 +536,30 @@ int dprc_get_obj_region(struct fsl_mc_io *mc_io,
return err;
}
 
-   /**
-* MC API version 6.3 introduced a new field to the region
-* descriptor: base_address. If the older API is in use then the base
-* address is set to zero to indicate it needs to be obtained elsewhere
-* (typically the device tree).
-*/
-   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 3))
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
-cmd_flags, token);
-   else
-   cmd.header =
-   mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
-cmd_flags, token);
+   if (dprc_major_ver > 6 || (dprc_major_ver == 6 && dprc_minor_ver >= 6)) 
{
+   /**
+* MC API version 6.6 changed the size of the MC portals and 
software
+* portals to 64K (as implemented by hardware). If older API is 
in use the
+* size reported is less (64 bytes for mc portals and 4K for 
software
+* portals).
+*/
+
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V3,
+ cmd_flags, token);
+
+   } else if (dprc_major_ver == 6 && dprc_minor_ver >= 3) {
+   /**
+* MC API version 6.3 introduced a new field to the region
+* descriptor: base_address. If the older API is in use then 
the base
+* address is set to zero to indicate it needs to be obtained 
elsewhere
+* (typically the device tree).
+*/
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG_V2,
+ cmd_flags, token);
+   } else {
+   cmd.header = mc_encode_cmd_header(DPRC_CMDID_GET_OBJ_REG,
+ cmd_flags, token);
+   }
 
cmd_params = (struct dprc_cmd_get_obj_region *)cmd.params;
cmd_params->obj_id = cpu_to_le32(obj_id);
diff --git a/drivers/bus/fsl-mc/fsl-mc-private.h 
b/drivers/bus/fsl-mc/fsl-mc-private.h
index e6fcff12c68d..8d65273a78d7 100644
--- a/drivers/bus/fsl-mc/fsl-mc-private.h
+++ b/drivers/bus/fsl-mc/fsl-mc-private.h
@@ -80,10 +80,12 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 /* DPRC command versioning */
 #define DPRC_CMD_BASE_VERSION  1
 #define DPRC_CMD_2ND_VERSION   2
+#define DPRC_CMD_3RD_VERSION   3
 #define DPRC_CMD_ID_OFFSET 4
 
 #define DPRC_CMD(id)   (((id) << DPRC_CMD_ID_OFFSET) | DPRC_CMD_BASE_VERSION)
 #define DPRC_CMD_V2(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_2ND_VERSION)
+#define DPRC_CMD_V3(id)(((id) << DPRC_CMD_ID_OFFSET) | 
DPRC_CMD_3RD_VERSION)
 
 /* DPRC command IDs */
 #define DPRC_CMDID_CLOSEDPRC_CMD(0x800)
@@ -105,6 +107,7 @@ int dpmcp_reset(struct fsl_mc_io *mc_io,
 #define DPRC_CMDID_GET_OBJ  DPRC_CMD(0x15A)
 #define DPRC_CMDID_GET_OBJ_REG  DPRC_CMD(0x15E)
 #define DPRC_CMDID_GET_OBJ_REG_V2   DPRC_CMD_V2(0x15E)
+#define DPRC_CMDID_GET_OBJ_REG_V3   DPRC_CMD_V3(0x15E)
 #define DPRC_CMDID_SET_OBJ_IRQ  DPRC_CMD(0x15F)
 
 #define DPRC_CMDID_GET_CONNECTION   DPRC_CMD(0x16C)
-- 
2.17.1