[PATCH] vop: Add missing __iomem annotation in vop_dc_to_vdev()

[Ashutosh Dixit]

Fix the following sparse warnings in drivers/misc/mic/vop//vop_main.c:

551:58: warning: incorrect type in argument 1 (different address spaces)
551:58:expected void const volatile [noderef] __iomem *addr
551:58:got restricted __le64 *
560:49: warning: incorrect type in argument 1 (different address spaces)
560:49:expected struct mic_device_ctrl *dc
560:49:got struct mic_device_ctrl [noderef] __iomem *dc
579:49: warning: incorrect type in argument 1 (different address spaces)
579:49:expected struct mic_device_ctrl *dc
579:49:got struct mic_device_ctrl [noderef] __iomem *dc

Cc: Michael S. Tsirkin 
Cc: Sudeep Dutt 
Cc: Arnd Bergmann 
Cc: Vincent Whitchurch 
Cc: stable 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/vop/vop_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/misc/mic/vop/vop_main.c b/drivers/misc/mic/vop/vop_main.c
index 85942f6717c5..25ed7d731701 100644
--- a/drivers/misc/mic/vop/vop_main.c
+++ b/drivers/misc/mic/vop/vop_main.c
@@ -546,7 +546,7 @@ static int vop_match_desc(struct device *dev, void *data)
return vdev->desc == (void __iomem *)data;
 }
 
-static struct _vop_vdev *vop_dc_to_vdev(struct mic_device_ctrl *dc)
+static struct _vop_vdev *vop_dc_to_vdev(struct mic_device_ctrl __iomem *dc)
 {
return (struct _vop_vdev *)(unsigned long)readq(>vdev);
 }
-- 
2.26.2.108.g048abe1751

Re: [PATCH-next] misc: mic: Use PTR_ERR_OR_ZERO

[Ashutosh Dixit]

On Tue, Jan 23 2018 at 02:55:19 PM, Al Viro  wrote:
> On Tue, Jan 23, 2018 at 03:10:09PM -0500, Christopher Díaz Riveros wrote:
>> Use PTR_ERR_OR_ZERO rather than if(IS_ERR(...)) + PTR_ERR
>> 
>> This issue was detected by using the Coccinelle software.
>
> ... and that's a wonderful demonstration of the reasons why PTR_ERR_OR_ZERO()
> is in bad taste.
>
> Look at the callers of that sucker:
>
> err = scif_anon_inode_getfile(ep);
> if (err)
> goto err_anon_inode;
>
> and
>
> err = scif_anon_inode_getfile(cep);
> if (err)
> goto scif_accept_error_anon_inode;
>
> See the problem?  What really happens here is that we
>   * call anon_inode_getfile() and stick the result into ->anon.
>   * if that was ERR_PTR(), we bugger off
>
> Checking that PTR_ERR_OR_ZERO() is non-zero is a bloody convoluted way of
> spelling
>   if (IS_ERR(...)) {
>   err = PTR_ERR(...);
>   goto sod_off;
>   }
>
> Your change preserves the ugliness of the original calling conventions;
> the mistake here is making it return an int.  Add the fact that
> all (both) callers are in drivers/misc/mic/scif/scif_api.c and take
> a look at this:
> ; git grep -n -w scif_anon_fops
> drivers/misc/mic/scif/scif_api.c:47:const struct file_operations 
> scif_anon_fops = {
> drivers/misc/mic/scif/scif_epd.h:167:   epd->anon = 
> anon_inode_getfile("scif", _anon_fops, NULL, 0);
> drivers/misc/mic/scif/scif_main.h:213:extern const struct file_operations 
> scif_anon_fops;
> ;
>
> So scif_anon_fops is
>   * defined in scif_api.c
>   * declared in scif_main.h
>   * used only in an inline function defined in scif_epd.h, which is used
> only in scif_api.c
>
> Could you spell "way too convoluted"?
>
> Please, do the following: move that sucker scif_api.c, and make it return
> struct file *.  As in return epd->anon = anon_inode_getfile(...);
> And turn the callers into
>   if (IS_ERR(whatever_you_call_it(ep))) {
>   err = PTR_ERR(ep->anon);
>   goto ...;
>   }
> Then make scif_anon_fops static and kill that extern in scif_main.h.
>
>   What the hell?   for callers of scif_poll()> 
>
> Let me see if I got it right - *ALL* callers of that thing pass it
> an array consisting of 1 (one) entry.  So playing with poll_wqueues
> and __pollwait() is utterly pointless in scif_poll().  So is grabbing
> references to that struct file, BTW, regardless of everything else.
> And so is having the bloody ->anon at all - __scif_pollfd() should be
> passed NULL as the first argument and that's it.  All it takes is
> a separate queue callback set via init_poll_funcptr() and to hell
> with all those complication.  Oh, and since we don't really need
> dynmic allocations inside that queue callback, table.error crap
> also goes to hell...
>
> Incidentally, since this is one of the two places outside of fs/select.c
> using poll_{init,free}wait(), we can bloody well unexport those - the
> other caller (in drivers/staging/farce^Wcomedi) also uses it for
> single struct file, pinned by the caller.  And while we are at it,
> that would also make struct poll_wqueues private to fs/select.c...
>
> Could Intel folks describe the planned future uses of scif_poll()?
> If it's not going to be used for large sets, the whole thing could
> be simplified quite nicely...

Cleanups and suggestions to improve the code such as those outlined at
the top of this reply are of course welcome. The SCIF API has both user
and kernel space clients, and the kernel API mirrors the user space
API. Similar to user space poll which supports multiple file
descriptors, the kernel client poll API was designed to allow poll on
multiple SCIF endpoints as well. Since the API has been stable for a
couple of years now, our preference is to preserve the API as far as
possible.

Re: [PATCH-next] misc: mic: Use PTR_ERR_OR_ZERO

[Ashutosh Dixit]

On Tue, Jan 23 2018 at 02:55:19 PM, Al Viro  wrote:
> On Tue, Jan 23, 2018 at 03:10:09PM -0500, Christopher Díaz Riveros wrote:
>> Use PTR_ERR_OR_ZERO rather than if(IS_ERR(...)) + PTR_ERR
>> 
>> This issue was detected by using the Coccinelle software.
>
> ... and that's a wonderful demonstration of the reasons why PTR_ERR_OR_ZERO()
> is in bad taste.
>
> Look at the callers of that sucker:
>
> err = scif_anon_inode_getfile(ep);
> if (err)
> goto err_anon_inode;
>
> and
>
> err = scif_anon_inode_getfile(cep);
> if (err)
> goto scif_accept_error_anon_inode;
>
> See the problem?  What really happens here is that we
>   * call anon_inode_getfile() and stick the result into ->anon.
>   * if that was ERR_PTR(), we bugger off
>
> Checking that PTR_ERR_OR_ZERO() is non-zero is a bloody convoluted way of
> spelling
>   if (IS_ERR(...)) {
>   err = PTR_ERR(...);
>   goto sod_off;
>   }
>
> Your change preserves the ugliness of the original calling conventions;
> the mistake here is making it return an int.  Add the fact that
> all (both) callers are in drivers/misc/mic/scif/scif_api.c and take
> a look at this:
> ; git grep -n -w scif_anon_fops
> drivers/misc/mic/scif/scif_api.c:47:const struct file_operations 
> scif_anon_fops = {
> drivers/misc/mic/scif/scif_epd.h:167:   epd->anon = 
> anon_inode_getfile("scif", _anon_fops, NULL, 0);
> drivers/misc/mic/scif/scif_main.h:213:extern const struct file_operations 
> scif_anon_fops;
> ;
>
> So scif_anon_fops is
>   * defined in scif_api.c
>   * declared in scif_main.h
>   * used only in an inline function defined in scif_epd.h, which is used
> only in scif_api.c
>
> Could you spell "way too convoluted"?
>
> Please, do the following: move that sucker scif_api.c, and make it return
> struct file *.  As in return epd->anon = anon_inode_getfile(...);
> And turn the callers into
>   if (IS_ERR(whatever_you_call_it(ep))) {
>   err = PTR_ERR(ep->anon);
>   goto ...;
>   }
> Then make scif_anon_fops static and kill that extern in scif_main.h.
>
>   What the hell?   for callers of scif_poll()> 
>
> Let me see if I got it right - *ALL* callers of that thing pass it
> an array consisting of 1 (one) entry.  So playing with poll_wqueues
> and __pollwait() is utterly pointless in scif_poll().  So is grabbing
> references to that struct file, BTW, regardless of everything else.
> And so is having the bloody ->anon at all - __scif_pollfd() should be
> passed NULL as the first argument and that's it.  All it takes is
> a separate queue callback set via init_poll_funcptr() and to hell
> with all those complication.  Oh, and since we don't really need
> dynmic allocations inside that queue callback, table.error crap
> also goes to hell...
>
> Incidentally, since this is one of the two places outside of fs/select.c
> using poll_{init,free}wait(), we can bloody well unexport those - the
> other caller (in drivers/staging/farce^Wcomedi) also uses it for
> single struct file, pinned by the caller.  And while we are at it,
> that would also make struct poll_wqueues private to fs/select.c...
>
> Could Intel folks describe the planned future uses of scif_poll()?
> If it's not going to be used for large sets, the whole thing could
> be simplified quite nicely...

Cleanups and suggestions to improve the code such as those outlined at
the top of this reply are of course welcome. The SCIF API has both user
and kernel space clients, and the kernel API mirrors the user space
API. Similar to user space poll which supports multiple file
descriptors, the kernel client poll API was designed to allow poll on
multiple SCIF endpoints as well. Since the API has been stable for a
couple of years now, our preference is to preserve the API as far as
possible.

Re: hfi1 use of PCI internals

[Ashutosh Dixit]

On Thu, Jun 16 2016 at 04:08:17 PM, Bjorn Helgaas  wrote:
>
> That's a good start, but leads to more questions.  For example, it
> doesn't answer the obvious question of why the driver needs to
> enable/disable ASPM from interrupt context.

For power saving reasons we keep ASPM L1 enabled, but implement a
heuristic to "quickly" disable ASPM L1 when we notice PCIe traffic (as
measured by the interrupt rate) starting up. If interrupt activity
ceases ASPM L1 is re-enabled.

> Disabling ASPM should only require writing the device's Link Control
> register.  The PCI core could probably provide an interface to do that
> in interrupt context.
>
> Enabling ASPM is not latency-critical and could probably be done from
> a work queue outside interrupt context, although conceptually there
> shouldn't be much required here either, and possibly the PCI core
> interface could be improved.

That is true, to keep latencies low we need to disable ASPM from
interrupt context, but re-enabling ASPM is not latency critical.

> It's possible the latency problem could be handled by some sort of
> quirk that overrides the acceptable latency.

Correct, this is another issue that needs to be resolved.

Re: hfi1 use of PCI internals

[Ashutosh Dixit]

On Thu, Jun 16 2016 at 04:08:17 PM, Bjorn Helgaas  wrote:
>
> That's a good start, but leads to more questions.  For example, it
> doesn't answer the obvious question of why the driver needs to
> enable/disable ASPM from interrupt context.

For power saving reasons we keep ASPM L1 enabled, but implement a
heuristic to "quickly" disable ASPM L1 when we notice PCIe traffic (as
measured by the interrupt rate) starting up. If interrupt activity
ceases ASPM L1 is re-enabled.

> Disabling ASPM should only require writing the device's Link Control
> register.  The PCI core could probably provide an interface to do that
> in interrupt context.
>
> Enabling ASPM is not latency-critical and could probably be done from
> a work queue outside interrupt context, although conceptually there
> shouldn't be much required here either, and possibly the PCI core
> interface could be improved.

That is true, to keep latencies low we need to disable ASPM from
interrupt context, but re-enabling ASPM is not latency critical.

> It's possible the latency problem could be handled by some sort of
> quirk that overrides the acceptable latency.

Correct, this is another issue that needs to be resolved.

Re: hfi1 use of PCI internals

[Ashutosh Dixit]

On Thu, Jun 16 2016 at 12:20:52 PM, Bjorn Helgaas  wrote:
> I noticed drivers/infiniband/hw/hfi1 got moved from staging to
> drivers/ for v4.7.  It does a bunch of grubbing around in PCIe ASPM
> configuration, e.g., see drivers/infiniband/hw/hfi1/aspm.h.
>
> I know there have been lots of ASPM issues, both hardware problems and
> Linux kernel problems, but it is *supposed* to be manageable by the
> core, without special driver support.  What's the justification for
> having to do this in the hfi1 driver?

The description for commit affa48de84 "staging/rdma/hfi1: Add support
for enabling/disabling PCIe ASPM" anticipates this question and
describes why this was done in the hfi1 driver:

Finally, the kernel ASPM API is not used in this patch. This is
because this patch does several non-standard things as SW
workarounds for HW issues. As mentioned above, it enables ASPM even
when advertised actual latencies are greater than acceptable
latencies. Also, whereas the kernel API only allows drivers to
disable ASPM from driver probe, this patch enables/disables ASPM
directly from interrupt context. Due to these reasons the kernel
ASPM API was not used.

Re: hfi1 use of PCI internals

[Ashutosh Dixit]

On Thu, Jun 16 2016 at 12:20:52 PM, Bjorn Helgaas  wrote:
> I noticed drivers/infiniband/hw/hfi1 got moved from staging to
> drivers/ for v4.7.  It does a bunch of grubbing around in PCIe ASPM
> configuration, e.g., see drivers/infiniband/hw/hfi1/aspm.h.
>
> I know there have been lots of ASPM issues, both hardware problems and
> Linux kernel problems, but it is *supposed* to be manageable by the
> core, without special driver support.  What's the justification for
> having to do this in the hfi1 driver?

The description for commit affa48de84 "staging/rdma/hfi1: Add support
for enabling/disabling PCIe ASPM" anticipates this question and
describes why this was done in the hfi1 driver:

Finally, the kernel ASPM API is not used in this patch. This is
because this patch does several non-standard things as SW
workarounds for HW issues. As mentioned above, it enables ASPM even
when advertised actual latencies are greater than acceptable
latencies. Also, whereas the kernel API only allows drivers to
disable ASPM from driver probe, this patch enables/disables ASPM
directly from interrupt context. Due to these reasons the kernel
ASPM API was not used.

Re: [PATCH] mpssd: fix buffer overflow warning

[Ashutosh Dixit]

On Fri, May 20 2016 at 12:54:51 AM, Mike Danese  wrote:
> The compilation emits a warning in function ‘snprintf’,
> inlined from ‘set_cmdline’ at
> ../Documentation/mic/mpssd/mpssd.c:1541:9:
> /usr/include/x86_64-linux-gnu/bits/stdio2.h:64:10:
> warning: call to __builtin___snprintf_chk will always overflow
> destination buffer
>
> This was introduced in commit f4a66c204482 ("misc: mic: Update MIC host
> daemon with COSM changes") and is fixed by reverting the changes to the
> size argument of these snprintf statements.

Thanks for the patch, the fix is fine.

Re: [PATCH] mpssd: fix buffer overflow warning

[Ashutosh Dixit]

On Fri, May 20 2016 at 12:54:51 AM, Mike Danese  wrote:
> The compilation emits a warning in function ‘snprintf’,
> inlined from ‘set_cmdline’ at
> ../Documentation/mic/mpssd/mpssd.c:1541:9:
> /usr/include/x86_64-linux-gnu/bits/stdio2.h:64:10:
> warning: call to __builtin___snprintf_chk will always overflow
> destination buffer
>
> This was introduced in commit f4a66c204482 ("misc: mic: Update MIC host
> daemon with COSM changes") and is fixed by reverting the changes to the
> size argument of these snprintf statements.

Thanks for the patch, the fix is fine.

[PATCH char-misc-linus] misc: mic: Fix for double fetch security bug in VOP driver

[Ashutosh Dixit]

The MIC VOP driver does two successive reads from user space to read a
variable length data structure. Kernel memory corruption can result if
the data structure changes between the two reads. This patch disallows
the chance of this happening.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=116651
Reported by: Pengfei Wang <wpengfein...@gmail.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/vop/vop_vringh.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/drivers/misc/mic/vop/vop_vringh.c 
b/drivers/misc/mic/vop/vop_vringh.c
index e94c7fb..88e4523 100644
--- a/drivers/misc/mic/vop/vop_vringh.c
+++ b/drivers/misc/mic/vop/vop_vringh.c
@@ -945,6 +945,11 @@ static long vop_ioctl(struct file *f, unsigned int cmd, 
unsigned long arg)
ret = -EFAULT;
goto free_ret;
}
+   /* Ensure desc has not changed between the two reads */
+   if (memcmp(, dd_config, sizeof(dd))) {
+   ret = -EINVAL;
+   goto free_ret;
+   }
mutex_lock(>vdev_mutex);
mutex_lock(>vop_mutex);
ret = vop_virtio_add_device(vdev, dd_config);
-- 
2.0.0.rc3.2.g998f840

[PATCH char-misc-linus] misc: mic: Fix for double fetch security bug in VOP driver

[Ashutosh Dixit]

The MIC VOP driver does two successive reads from user space to read a
variable length data structure. Kernel memory corruption can result if
the data structure changes between the two reads. This patch disallows
the chance of this happening.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=116651
Reported by: Pengfei Wang 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/vop/vop_vringh.c | 5 +
 1 file changed, 5 insertions(+)

diff --git a/drivers/misc/mic/vop/vop_vringh.c 
b/drivers/misc/mic/vop/vop_vringh.c
index e94c7fb..88e4523 100644
--- a/drivers/misc/mic/vop/vop_vringh.c
+++ b/drivers/misc/mic/vop/vop_vringh.c
@@ -945,6 +945,11 @@ static long vop_ioctl(struct file *f, unsigned int cmd, 
unsigned long arg)
ret = -EFAULT;
goto free_ret;
}
+   /* Ensure desc has not changed between the two reads */
+   if (memcmp(, dd_config, sizeof(dd))) {
+   ret = -EINVAL;
+   goto free_ret;
+   }
mutex_lock(>vdev_mutex);
mutex_lock(>vop_mutex);
ret = vop_virtio_add_device(vdev, dd_config);
-- 
2.0.0.rc3.2.g998f840

Re: [PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

On Sun, Jan 03 2016 at 10:35:26 PM, "Koul, Vinod"  wrote:
> On Tue, Dec 22, 2015 at 07:35:23PM -0800, Ashutosh Dixit wrote:
>> This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
>> spin_unlock").
>>
>> The above patch is incorrect. There is nothing wrong with the original
>> code. The spin_lock is acquired in the "prep" functions and released
>> in "submit".
>
> And going by dmaengine sematics, I do not think that is entrely right.
>
> A user may choose to prepare multiple desciptors and then sumbit later,
> looking at code I do not see how that will work.
>
> Please fix that

The mic_x100_dma driver still allows a client to prepare and submit
multiple descriptors and triggers the hardware only when issue_pending
is called (or a threshold is exceeded). Identical coding patterns exist
in the IOAT dma driver, on which the mic_x100_dma driver is based.

Further, mic_x100 dma channels are private and used only by other MIC
drivers such as SCIF (drivers/misc/mic/scif). These drivers obviously
alternate prep and submit calls as required by mic_x100_dma. We do not
envisage a wider use of the mic_x100_dma driver at this point.

A change such as allowing multiple prep's before a submit requires large
scale changes in the driver. In the absence of a real use case, there is
no plan to make such changes at present. At this point we only want the
driver to be restored to its previously functional state for the 4.4
kernel. The patch in question results in system lockups so it is a
serious v4.4 regression.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

On Sun, Jan 03 2016 at 10:35:26 PM, "Koul, Vinod" <vinod.k...@intel.com> wrote:
> On Tue, Dec 22, 2015 at 07:35:23PM -0800, Ashutosh Dixit wrote:
>> This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
>> spin_unlock").
>>
>> The above patch is incorrect. There is nothing wrong with the original
>> code. The spin_lock is acquired in the "prep" functions and released
>> in "submit".
>
> And going by dmaengine sematics, I do not think that is entrely right.
>
> A user may choose to prepare multiple desciptors and then sumbit later,
> looking at code I do not see how that will work.
>
> Please fix that

The mic_x100_dma driver still allows a client to prepare and submit
multiple descriptors and triggers the hardware only when issue_pending
is called (or a threshold is exceeded). Identical coding patterns exist
in the IOAT dma driver, on which the mic_x100_dma driver is based.

Further, mic_x100 dma channels are private and used only by other MIC
drivers such as SCIF (drivers/misc/mic/scif). These drivers obviously
alternate prep and submit calls as required by mic_x100_dma. We do not
envisage a wider use of the mic_x100_dma driver at this point.

A change such as allowing multiple prep's before a submit requires large
scale changes in the driver. In the absence of a real use case, there is
no plan to make such changes at present. At this point we only want the
driver to be restored to its previously functional state for the 4.4
kernel. The patch in question results in system lockups so it is a
serious v4.4 regression.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

On Wed, Dec 23 2015 at 12:45:31 AM, Saurabh Sengar  
wrote:
> On 23 December 2015 at 09:05, Ashutosh Dixit  wrote:
>> This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
>> spin_unlock").
>>
>> The above patch is incorrect. There is nothing wrong with the original
>> code. The spin_lock is acquired in the "prep" functions and released
>> in "submit".
>
> Hi Ashutosh,
>
> If it is need to be released by submit function, we don't require the
> spin_unlock on success case as well.
> am I correct ?

No, you are wrong. In the prep functions, we are not using spin_unlock in
success cases but in failure ones.

>> Signed-off-by: Ashutosh Dixit 
>> ---
>>  drivers/dma/mic_x100_dma.c | 15 +--
>>  1 file changed, 5 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
>> index cddfa8d..068e920 100644
>> --- a/drivers/dma/mic_x100_dma.c
>> +++ b/drivers/dma/mic_x100_dma.c
>> @@ -317,7 +317,6 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
>> dma_dest,
>> struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
>> struct device *dev = mic_dma_ch_to_device(mic_ch);
>> int result;
>> -   struct dma_async_tx_descriptor *tx = NULL;
>>
>> if (!len && !flags)
>> return NULL;
>> @@ -325,13 +324,10 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, 
>> dma_addr_t dma_dest,
>> spin_lock(_ch->prep_lock);
>> result = mic_dma_do_dma(mic_ch, flags, dma_src, dma_dest, len);
>> if (result >= 0)
>> -   tx = allocate_tx(mic_ch);
>> -
>> -   if (!tx)
>> -   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
>> -
>> +   return allocate_tx(mic_ch);
>> +   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
>> spin_unlock(_ch->prep_lock);
>
> This spin_unlock shouldn't be required as explained it is getting
> released by submit function

This is a failure case. Submit will not be called in this case so
spin_unlock needs to be called here.

>> -   return tx;
>> +   return NULL;
>>  }
>>
>>  static struct dma_async_tx_descriptor *
>> @@ -339,14 +335,13 @@ mic_dma_prep_interrupt_lock(struct dma_chan *ch, 
>> unsigned long flags)
>>  {
>> struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
>> int ret;
>> -   struct dma_async_tx_descriptor *tx = NULL;
>>
>> spin_lock(_ch->prep_lock);
>> ret = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
>> if (!ret)
>> -   tx = allocate_tx(mic_ch);
>> +   return allocate_tx(mic_ch);
>> spin_unlock(_ch->prep_lock);
>
> and this too ?

Yes, this too.

>> -   return tx;
>> +   return NULL;
>>  }
>>
>>  /* Return the status of the transaction */
>> --
>> 2.0.0.rc3.2.g998f840
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
spin_unlock").

The above patch is incorrect. There is nothing wrong with the original
code. The spin_lock is acquired in the "prep" functions and released
in "submit".

Signed-off-by: Ashutosh Dixit 
---
 drivers/dma/mic_x100_dma.c | 15 +--
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
index cddfa8d..068e920 100644
--- a/drivers/dma/mic_x100_dma.c
+++ b/drivers/dma/mic_x100_dma.c
@@ -317,7 +317,6 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
dma_dest,
struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
struct device *dev = mic_dma_ch_to_device(mic_ch);
int result;
-   struct dma_async_tx_descriptor *tx = NULL;
 
if (!len && !flags)
return NULL;
@@ -325,13 +324,10 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
dma_dest,
spin_lock(_ch->prep_lock);
result = mic_dma_do_dma(mic_ch, flags, dma_src, dma_dest, len);
if (result >= 0)
-   tx = allocate_tx(mic_ch);
-
-   if (!tx)
-   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
-
+   return allocate_tx(mic_ch);
+   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
spin_unlock(_ch->prep_lock);
-   return tx;
+   return NULL;
 }
 
 static struct dma_async_tx_descriptor *
@@ -339,14 +335,13 @@ mic_dma_prep_interrupt_lock(struct dma_chan *ch, unsigned 
long flags)
 {
struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
int ret;
-   struct dma_async_tx_descriptor *tx = NULL;
 
spin_lock(_ch->prep_lock);
ret = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
if (!ret)
-   tx = allocate_tx(mic_ch);
+   return allocate_tx(mic_ch);
spin_unlock(_ch->prep_lock);
-   return tx;
+   return NULL;
 }
 
 /* Return the status of the transaction */
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
spin_unlock").

The above patch is incorrect. There is nothing wrong with the original
code. The spin_lock is acquired in the "prep" functions and released
in "submit".

Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/dma/mic_x100_dma.c | 15 +--
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
index cddfa8d..068e920 100644
--- a/drivers/dma/mic_x100_dma.c
+++ b/drivers/dma/mic_x100_dma.c
@@ -317,7 +317,6 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
dma_dest,
struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
struct device *dev = mic_dma_ch_to_device(mic_ch);
int result;
-   struct dma_async_tx_descriptor *tx = NULL;
 
if (!len && !flags)
return NULL;
@@ -325,13 +324,10 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
dma_dest,
spin_lock(_ch->prep_lock);
result = mic_dma_do_dma(mic_ch, flags, dma_src, dma_dest, len);
if (result >= 0)
-   tx = allocate_tx(mic_ch);
-
-   if (!tx)
-   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
-
+   return allocate_tx(mic_ch);
+   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
spin_unlock(_ch->prep_lock);
-   return tx;
+   return NULL;
 }
 
 static struct dma_async_tx_descriptor *
@@ -339,14 +335,13 @@ mic_dma_prep_interrupt_lock(struct dma_chan *ch, unsigned 
long flags)
 {
struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
int ret;
-   struct dma_async_tx_descriptor *tx = NULL;
 
spin_lock(_ch->prep_lock);
ret = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
if (!ret)
-   tx = allocate_tx(mic_ch);
+   return allocate_tx(mic_ch);
spin_unlock(_ch->prep_lock);
-   return tx;
+   return NULL;
 }
 
 /* Return the status of the transaction */
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] dma: Revert "dmaengine: mic_x100: add missing spin_unlock"

[Ashutosh Dixit]

On Wed, Dec 23 2015 at 12:45:31 AM, Saurabh Sengar <saurabh.tr...@gmail.com> 
wrote:
> On 23 December 2015 at 09:05, Ashutosh Dixit <ashutosh.di...@intel.com> wrote:
>> This reverts commit e958e079e254 ("dmaengine: mic_x100: add missing
>> spin_unlock").
>>
>> The above patch is incorrect. There is nothing wrong with the original
>> code. The spin_lock is acquired in the "prep" functions and released
>> in "submit".
>
> Hi Ashutosh,
>
> If it is need to be released by submit function, we don't require the
> spin_unlock on success case as well.
> am I correct ?

No, you are wrong. In the prep functions, we are not using spin_unlock in
success cases but in failure ones.

>> Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
>> ---
>>  drivers/dma/mic_x100_dma.c | 15 +--
>>  1 file changed, 5 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
>> index cddfa8d..068e920 100644
>> --- a/drivers/dma/mic_x100_dma.c
>> +++ b/drivers/dma/mic_x100_dma.c
>> @@ -317,7 +317,6 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
>> dma_dest,
>> struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
>> struct device *dev = mic_dma_ch_to_device(mic_ch);
>> int result;
>> -   struct dma_async_tx_descriptor *tx = NULL;
>>
>> if (!len && !flags)
>> return NULL;
>> @@ -325,13 +324,10 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, 
>> dma_addr_t dma_dest,
>> spin_lock(_ch->prep_lock);
>> result = mic_dma_do_dma(mic_ch, flags, dma_src, dma_dest, len);
>> if (result >= 0)
>> -   tx = allocate_tx(mic_ch);
>> -
>> -   if (!tx)
>> -   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
>> -
>> +   return allocate_tx(mic_ch);
>> +   dev_err(dev, "Error enqueueing dma, error=%d\n", result);
>> spin_unlock(_ch->prep_lock);
>
> This spin_unlock shouldn't be required as explained it is getting
> released by submit function

This is a failure case. Submit will not be called in this case so
spin_unlock needs to be called here.

>> -   return tx;
>> +   return NULL;
>>  }
>>
>>  static struct dma_async_tx_descriptor *
>> @@ -339,14 +335,13 @@ mic_dma_prep_interrupt_lock(struct dma_chan *ch, 
>> unsigned long flags)
>>  {
>> struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
>> int ret;
>> -   struct dma_async_tx_descriptor *tx = NULL;
>>
>> spin_lock(_ch->prep_lock);
>> ret = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
>> if (!ret)
>> -   tx = allocate_tx(mic_ch);
>> +   return allocate_tx(mic_ch);
>> spin_unlock(_ch->prep_lock);
>
> and this too ?

Yes, this too.

>> -   return tx;
>> +   return NULL;
>>  }
>>
>>  /* Return the status of the transaction */
>> --
>> 2.0.0.rc3.2.g998f840
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH RESEND] misc: mic: Fix crash when MIC reset is invoked in RESET_FAILED state

[Ashutosh Dixit]

This patch fixes the following crash seen when MIC reset is invoked in
RESET_FAILED state due to device_del being called a second time on an
already deleted device:

[] device_del+0x45/0x1d0
[] device_unregister+0x1e/0x60
[] scif_unregister_device+0x12/0x20 [scif_bus]
[] cosm_stop+0xaa/0xe0 [mic_cosm]
[] cosm_reset_trigger_work+0x14/0x20 [mic_cosm]

The fix consists in realizing that because cosm_reset changes the
state to MIC_RESETTING, cosm_stop needs the previous state, before it
changed to MIC_RESETTING, to decide whether a hw_ops->stop had
previously been issued. This is now provided in a new cosm_device
member cdev->prev_state.

Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/bus/cosm_bus.h   |  2 ++
 drivers/misc/mic/cosm/cosm_main.c | 13 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/misc/mic/bus/cosm_bus.h b/drivers/misc/mic/bus/cosm_bus.h
index f7c57f2..8b63418 100644
--- a/drivers/misc/mic/bus/cosm_bus.h
+++ b/drivers/misc/mic/bus/cosm_bus.h
@@ -30,6 +30,7 @@
  * @attr_group: Pointer to list of sysfs attribute groups.
  * @sdev: Device for sysfs entries.
  * @state: MIC state.
+ * @prev_state: MIC state previous to MIC_RESETTING
  * @shutdown_status: MIC status reported by card for shutdown/crashes.
  * @shutdown_status_int: Internal shutdown status maintained by the driver
  * @cosm_mutex: Mutex for synchronizing access to data structures.
@@ -55,6 +56,7 @@ struct cosm_device {
const struct attribute_group **attr_group;
struct device *sdev;
u8 state;
+   u8 prev_state;
u8 shutdown_status;
u8 shutdown_status_int;
struct mutex cosm_mutex;
diff --git a/drivers/misc/mic/cosm/cosm_main.c 
b/drivers/misc/mic/cosm/cosm_main.c
index 4b4b356..7005cb1 100644
--- a/drivers/misc/mic/cosm/cosm_main.c
+++ b/drivers/misc/mic/cosm/cosm_main.c
@@ -153,8 +153,10 @@ void cosm_stop(struct cosm_device *cdev, bool force)
 * stop(..) calls device_unregister and will crash the system if
 * called multiple times.
 */
-   bool call_hw_ops = cdev->state != MIC_RESET_FAILED &&
-   cdev->state != MIC_READY;
+   u8 state = cdev->state == MIC_RESETTING ?
+   cdev->prev_state : cdev->state;
+   bool call_hw_ops = state != MIC_RESET_FAILED &&
+   state != MIC_READY;
 
if (cdev->state != MIC_RESETTING)
cosm_set_state(cdev, MIC_RESETTING);
@@ -195,8 +197,11 @@ int cosm_reset(struct cosm_device *cdev)
 
mutex_lock(>cosm_mutex);
if (cdev->state != MIC_READY) {
-   cosm_set_state(cdev, MIC_RESETTING);
-   schedule_work(>reset_trigger_work);
+   if (cdev->state != MIC_RESETTING) {
+   cdev->prev_state = cdev->state;
+   cosm_set_state(cdev, MIC_RESETTING);
+   schedule_work(>reset_trigger_work);
+   }
} else {
dev_err(>dev, "%s %d MIC is READY\n", __func__, __LINE__);
rc = -EINVAL;
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH RESEND] misc: mic: Fix crash when MIC reset is invoked in RESET_FAILED state

[Ashutosh Dixit]

This patch fixes the following crash seen when MIC reset is invoked in
RESET_FAILED state due to device_del being called a second time on an
already deleted device:

[] device_del+0x45/0x1d0
[] device_unregister+0x1e/0x60
[] scif_unregister_device+0x12/0x20 [scif_bus]
[] cosm_stop+0xaa/0xe0 [mic_cosm]
[] cosm_reset_trigger_work+0x14/0x20 [mic_cosm]

The fix consists in realizing that because cosm_reset changes the
state to MIC_RESETTING, cosm_stop needs the previous state, before it
changed to MIC_RESETTING, to decide whether a hw_ops->stop had
previously been issued. This is now provided in a new cosm_device
member cdev->prev_state.

Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/bus/cosm_bus.h   |  2 ++
 drivers/misc/mic/cosm/cosm_main.c | 13 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/misc/mic/bus/cosm_bus.h b/drivers/misc/mic/bus/cosm_bus.h
index f7c57f2..8b63418 100644
--- a/drivers/misc/mic/bus/cosm_bus.h
+++ b/drivers/misc/mic/bus/cosm_bus.h
@@ -30,6 +30,7 @@
  * @attr_group: Pointer to list of sysfs attribute groups.
  * @sdev: Device for sysfs entries.
  * @state: MIC state.
+ * @prev_state: MIC state previous to MIC_RESETTING
  * @shutdown_status: MIC status reported by card for shutdown/crashes.
  * @shutdown_status_int: Internal shutdown status maintained by the driver
  * @cosm_mutex: Mutex for synchronizing access to data structures.
@@ -55,6 +56,7 @@ struct cosm_device {
const struct attribute_group **attr_group;
struct device *sdev;
u8 state;
+   u8 prev_state;
u8 shutdown_status;
u8 shutdown_status_int;
struct mutex cosm_mutex;
diff --git a/drivers/misc/mic/cosm/cosm_main.c 
b/drivers/misc/mic/cosm/cosm_main.c
index 4b4b356..7005cb1 100644
--- a/drivers/misc/mic/cosm/cosm_main.c
+++ b/drivers/misc/mic/cosm/cosm_main.c
@@ -153,8 +153,10 @@ void cosm_stop(struct cosm_device *cdev, bool force)
 * stop(..) calls device_unregister and will crash the system if
 * called multiple times.
 */
-   bool call_hw_ops = cdev->state != MIC_RESET_FAILED &&
-   cdev->state != MIC_READY;
+   u8 state = cdev->state == MIC_RESETTING ?
+   cdev->prev_state : cdev->state;
+   bool call_hw_ops = state != MIC_RESET_FAILED &&
+   state != MIC_READY;
 
if (cdev->state != MIC_RESETTING)
cosm_set_state(cdev, MIC_RESETTING);
@@ -195,8 +197,11 @@ int cosm_reset(struct cosm_device *cdev)
 
mutex_lock(>cosm_mutex);
if (cdev->state != MIC_READY) {
-   cosm_set_state(cdev, MIC_RESETTING);
-   schedule_work(>reset_trigger_work);
+   if (cdev->state != MIC_RESETTING) {
+   cdev->prev_state = cdev->state;
+   cosm_set_state(cdev, MIC_RESETTING);
+   schedule_work(>reset_trigger_work);
+   }
} else {
dev_err(>dev, "%s %d MIC is READY\n", __func__, __LINE__);
rc = -EINVAL;
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] misc: mic: Fix crash when MIC reset is invoked in RESET_FAILED state

[Ashutosh Dixit]

This patch fixes the following crash seen when MIC reset is invoked in
RESET_FAILED state due to device_del being called a second time on an
already deleted device:

[] device_del+0x45/0x1d0
[] device_unregister+0x1e/0x60
[] scif_unregister_device+0x12/0x20 [scif_bus]
[] cosm_stop+0xaa/0xe0 [mic_cosm]
[] cosm_reset_trigger_work+0x14/0x20 [mic_cosm]

The fix consists in realizing that because cosm_reset changes the
state to MIC_RESETTING, cosm_stop needs the previous state, before it
changed to MIC_RESETTING, to decide whether a hw_ops->stop had
previously been issued. This is now provided in a new cosm_device
member cdev->prev_state.

Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/bus/cosm_bus.h   |  2 ++
 drivers/misc/mic/cosm/cosm_main.c | 13 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/misc/mic/bus/cosm_bus.h b/drivers/misc/mic/bus/cosm_bus.h
index f7c57f2..8b63418 100644
--- a/drivers/misc/mic/bus/cosm_bus.h
+++ b/drivers/misc/mic/bus/cosm_bus.h
@@ -30,6 +30,7 @@
  * @attr_group: Pointer to list of sysfs attribute groups.
  * @sdev: Device for sysfs entries.
  * @state: MIC state.
+ * @prev_state: MIC state previous to MIC_RESETTING
  * @shutdown_status: MIC status reported by card for shutdown/crashes.
  * @shutdown_status_int: Internal shutdown status maintained by the driver
  * @cosm_mutex: Mutex for synchronizing access to data structures.
@@ -55,6 +56,7 @@ struct cosm_device {
const struct attribute_group **attr_group;
struct device *sdev;
u8 state;
+   u8 prev_state;
u8 shutdown_status;
u8 shutdown_status_int;
struct mutex cosm_mutex;
diff --git a/drivers/misc/mic/cosm/cosm_main.c 
b/drivers/misc/mic/cosm/cosm_main.c
index 4b4b356..7005cb1 100644
--- a/drivers/misc/mic/cosm/cosm_main.c
+++ b/drivers/misc/mic/cosm/cosm_main.c
@@ -153,8 +153,10 @@ void cosm_stop(struct cosm_device *cdev, bool force)
 * stop(..) calls device_unregister and will crash the system if
 * called multiple times.
 */
-   bool call_hw_ops = cdev->state != MIC_RESET_FAILED &&
-   cdev->state != MIC_READY;
+   u8 state = cdev->state == MIC_RESETTING ?
+   cdev->prev_state : cdev->state;
+   bool call_hw_ops = state != MIC_RESET_FAILED &&
+   state != MIC_READY;
 
if (cdev->state != MIC_RESETTING)
cosm_set_state(cdev, MIC_RESETTING);
@@ -195,8 +197,11 @@ int cosm_reset(struct cosm_device *cdev)
 
mutex_lock(>cosm_mutex);
if (cdev->state != MIC_READY) {
-   cosm_set_state(cdev, MIC_RESETTING);
-   schedule_work(>reset_trigger_work);
+   if (cdev->state != MIC_RESETTING) {
+   cdev->prev_state = cdev->state;
+   cosm_set_state(cdev, MIC_RESETTING);
+   schedule_work(>reset_trigger_work);
+   }
} else {
dev_err(>dev, "%s %d MIC is READY\n", __func__, __LINE__);
rc = -EINVAL;
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] misc: mic: Fix crash when MIC reset is invoked in RESET_FAILED state

[Ashutosh Dixit]

This patch fixes the following crash seen when MIC reset is invoked in
RESET_FAILED state due to device_del being called a second time on an
already deleted device:

[] device_del+0x45/0x1d0
[] device_unregister+0x1e/0x60
[] scif_unregister_device+0x12/0x20 [scif_bus]
[] cosm_stop+0xaa/0xe0 [mic_cosm]
[] cosm_reset_trigger_work+0x14/0x20 [mic_cosm]

The fix consists in realizing that because cosm_reset changes the
state to MIC_RESETTING, cosm_stop needs the previous state, before it
changed to MIC_RESETTING, to decide whether a hw_ops->stop had
previously been issued. This is now provided in a new cosm_device
member cdev->prev_state.

Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/bus/cosm_bus.h   |  2 ++
 drivers/misc/mic/cosm/cosm_main.c | 13 +
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/misc/mic/bus/cosm_bus.h b/drivers/misc/mic/bus/cosm_bus.h
index f7c57f2..8b63418 100644
--- a/drivers/misc/mic/bus/cosm_bus.h
+++ b/drivers/misc/mic/bus/cosm_bus.h
@@ -30,6 +30,7 @@
  * @attr_group: Pointer to list of sysfs attribute groups.
  * @sdev: Device for sysfs entries.
  * @state: MIC state.
+ * @prev_state: MIC state previous to MIC_RESETTING
  * @shutdown_status: MIC status reported by card for shutdown/crashes.
  * @shutdown_status_int: Internal shutdown status maintained by the driver
  * @cosm_mutex: Mutex for synchronizing access to data structures.
@@ -55,6 +56,7 @@ struct cosm_device {
const struct attribute_group **attr_group;
struct device *sdev;
u8 state;
+   u8 prev_state;
u8 shutdown_status;
u8 shutdown_status_int;
struct mutex cosm_mutex;
diff --git a/drivers/misc/mic/cosm/cosm_main.c 
b/drivers/misc/mic/cosm/cosm_main.c
index 4b4b356..7005cb1 100644
--- a/drivers/misc/mic/cosm/cosm_main.c
+++ b/drivers/misc/mic/cosm/cosm_main.c
@@ -153,8 +153,10 @@ void cosm_stop(struct cosm_device *cdev, bool force)
 * stop(..) calls device_unregister and will crash the system if
 * called multiple times.
 */
-   bool call_hw_ops = cdev->state != MIC_RESET_FAILED &&
-   cdev->state != MIC_READY;
+   u8 state = cdev->state == MIC_RESETTING ?
+   cdev->prev_state : cdev->state;
+   bool call_hw_ops = state != MIC_RESET_FAILED &&
+   state != MIC_READY;
 
if (cdev->state != MIC_RESETTING)
cosm_set_state(cdev, MIC_RESETTING);
@@ -195,8 +197,11 @@ int cosm_reset(struct cosm_device *cdev)
 
mutex_lock(>cosm_mutex);
if (cdev->state != MIC_READY) {
-   cosm_set_state(cdev, MIC_RESETTING);
-   schedule_work(>reset_trigger_work);
+   if (cdev->state != MIC_RESETTING) {
+   cdev->prev_state = cdev->state;
+   cosm_set_state(cdev, MIC_RESETTING);
+   schedule_work(>reset_trigger_work);
+   }
} else {
dev_err(>dev, "%s %d MIC is READY\n", __func__, __LINE__);
rc = -EINVAL;
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v2] misc: mic: replace kfree with put_device

[Ashutosh Dixit]

On Sun, Oct 04 2015 at 05:33:47 PM, Geliang Tang  wrote:
> Handle a failed device_register(), replace kfree() with put_device(),
> which will call mbus_release_dev() or scif_release_dev().

Could you add the same change to cosm_bus.c too in the same path and
resubmit the patch. Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH v2] misc: mic: replace kfree with put_device

[Ashutosh Dixit]

On Sun, Oct 04 2015 at 05:33:47 PM, Geliang Tang  wrote:
> Handle a failed device_register(), replace kfree() with put_device(),
> which will call mbus_release_dev() or scif_release_dev().

Could you add the same change to cosm_bus.c too in the same path and
resubmit the patch. Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] misc: mic: use kstrdup() in cosm_sysfs

[Ashutosh Dixit]

On Sat, Oct 10 2015 at 12:14:30 PM, Joe Perches  wrote:
> On Sat, 2015-10-10 at 04:46 -0700, Geliang Tang wrote:
>> Use kstrdup instead of kmalloc and strncpy.
>
>> diff --git a/drivers/misc/mic/cosm/cosm_sysfs.c 
>> b/drivers/misc/mic/cosm/cosm_sysfs.c
> []
>> @@ -211,18 +211,14 @@ cmdline_store(struct device *dev, struct 
>> device_attribute *attr,
>>  mutex_lock(>cosm_mutex);
>>  kfree(cdev->cmdline);
>>
>> -cdev->cmdline = kmalloc(count + 1, GFP_KERNEL);
>> +cdev->cmdline = kstrdup(buf, GFP_KERNEL);
>>
> As count is a function argument, this isn't the same code.
>
Correct, is this code safe for unterminated strings from user land?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] misc: mic: use kstrdup() in cosm_sysfs

[Ashutosh Dixit]

On Sat, Oct 10 2015 at 12:14:30 PM, Joe Perches  wrote:
> On Sat, 2015-10-10 at 04:46 -0700, Geliang Tang wrote:
>> Use kstrdup instead of kmalloc and strncpy.
>
>> diff --git a/drivers/misc/mic/cosm/cosm_sysfs.c 
>> b/drivers/misc/mic/cosm/cosm_sysfs.c
> []
>> @@ -211,18 +211,14 @@ cmdline_store(struct device *dev, struct 
>> device_attribute *attr,
>>  mutex_lock(>cosm_mutex);
>>  kfree(cdev->cmdline);
>>
>> -cdev->cmdline = kmalloc(count + 1, GFP_KERNEL);
>> +cdev->cmdline = kstrdup(buf, GFP_KERNEL);
>>
> As count is a function argument, this isn't the same code.
>
Correct, is this code safe for unterminated strings from user land?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next] misc: mic: Fix randconfig build error

[Ashutosh Dixit]

Fixes randconfig build error reported at
http://www.spinics.net/lists/kernel/msg2092346.html

Reported-by: Jim Davis 
Reviewed-by: Dasaratharaman Chandramouli 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm/cosm_debugfs.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/misc/mic/cosm/cosm_debugfs.c 
b/drivers/misc/mic/cosm/cosm_debugfs.c
index 30d953d..216cb3c 100644
--- a/drivers/misc/mic/cosm/cosm_debugfs.c
+++ b/drivers/misc/mic/cosm/cosm_debugfs.c
@@ -21,6 +21,7 @@
 
 #include 
 #include 
+#include 
 #include "cosm_main.h"
 
 /* Debugfs parent dir */
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next] misc: mic: Fix randconfig build error

[Ashutosh Dixit]

Fixes randconfig build error reported at
http://www.spinics.net/lists/kernel/msg2092346.html

Reported-by: Jim Davis <jim.ep...@gmail.com>
Reviewed-by: Dasaratharaman Chandramouli <dasaratharaman.chandramo...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/cosm/cosm_debugfs.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/misc/mic/cosm/cosm_debugfs.c 
b/drivers/misc/mic/cosm/cosm_debugfs.c
index 30d953d..216cb3c 100644
--- a/drivers/misc/mic/cosm/cosm_debugfs.c
+++ b/drivers/misc/mic/cosm/cosm_debugfs.c
@@ -21,6 +21,7 @@
 
 #include 
 #include 
+#include 
 #include "cosm_main.h"
 
 /* Debugfs parent dir */
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 19/22] misc: mic: SCIF remote memory map/unmap interface

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the SCIF mmap/munmap interface. A similar
capability is provided to kernel clients via the
scif_get_pages()/scif_put_pages() APIs. The SCIF mmap interface
queries to check if a window is valid and then remaps the local
virtual address to the remote physical pages. These mappings are
subsequently destroyed upon receipt of the VMA close operation or
scif_get_pages().  This functionality allows SCIF users to directly
access remote memory without any driver interaction once the mappings
are created thereby providing bare-metal PCIe latency. These mappings
are zapped to avoid RMA accesses from user space, if a Coprocessor is
reset.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_mmap.c | 699 ++
 1 file changed, 699 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_mmap.c

diff --git a/drivers/misc/mic/scif/scif_mmap.c 
b/drivers/misc/mic/scif/scif_mmap.c
new file mode 100644
index 000..49cb8f7
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_mmap.c
@@ -0,0 +1,699 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+
+/*
+ * struct scif_vma_info - Information about a remote memory mapping
+ *   created via scif_mmap(..)
+ * @vma: VM area struct
+ * @list: link to list of active vmas
+ */
+struct scif_vma_info {
+   struct vm_area_struct *vma;
+   struct list_head list;
+};
+
+void scif_recv_munmap(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_rma_req req;
+   struct scif_window *window = NULL;
+   struct scif_window *recv_window =
+   (struct scif_window *)msg->payload[0];
+   struct scif_endpt *ep;
+
+   ep = (struct scif_endpt *)recv_window->ep;
+   req.out_window = 
+   req.offset = recv_window->offset;
+   req.prot = recv_window->prot;
+   req.nr_bytes = recv_window->nr_pages << PAGE_SHIFT;
+   req.type = SCIF_WINDOW_FULL;
+   req.head = >rma_info.reg_list;
+   msg->payload[0] = ep->remote_ep;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Does a valid window exist? */
+   if (scif_query_window()) {
+   dev_err(>sdev->dev,
+   "%s %d -ENXIO\n", __func__, __LINE__);
+   msg->uop = SCIF_UNREGISTER_ACK;
+   goto error;
+   }
+
+   scif_put_window(window, window->nr_pages);
+
+   if (!window->ref_count) {
+   atomic_inc(>rma_info.tw_refcount);
+   ep->rma_info.async_list_del = 1;
+   list_del_init(>list);
+   scif_free_window_offset(ep, window, window->offset);
+   }
+error:
+   mutex_unlock(>rma_info.rma_lock);
+   if (window && !window->ref_count)
+   scif_queue_for_cleanup(window, _info.rma);
+}
+
+/*
+ * Remove valid remote memory mappings created via scif_mmap(..) from the
+ * process address space since the remote node is lost
+ */
+static void __scif_zap_mmaps(struct scif_endpt *ep)
+{
+   struct list_head *item;
+   struct scif_vma_info *info;
+   struct vm_area_struct *vma;
+   unsigned long size;
+
+   spin_lock(>lock);
+   list_for_each(item, >rma_info.vma_list) {
+   info = list_entry(item, struct scif_vma_info, list);
+   vma = info->vma;
+   size = vma->vm_end - vma->vm_start;
+   zap_vma_ptes(vma, vma->vm_start, size);
+   dev_dbg(scif_info.mdev.this_device,
+   "%s ep %p zap vma %p size 0x%lx\n",
+   __func__, ep, info->vma, size);
+   }
+   spin_unlock(>lock);
+}
+
+/*
+ * Traverse the list of endpoints for a particular remote node and
+ * zap valid remote memory mappings since the remote node is lost
+ */
+static void _scif_zap_mmaps(int node, struct list_head *head)
+{
+   struct scif_endpt *ep;
+   struct list_head *item;
+
+   mutex_lock(_info.connlock);
+   list_for_each(item, head) {
+   ep = list_entry(item, struct scif_endpt, list);
+   if (ep->remote_dev->node == node)
+   __scif_zap_mmaps(ep);
+   }
+   mutex_unlock(_info.connlock);
+}
+
+/*
+ * Wrapper for removing remote memory ma

[PATCH char-misc-next v2 22/22] misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the SCIF kernel node QP control messages required to
enable SCIF RMAs. Examples of such node QP control messages include
registration, unregistration, remote memory allocation requests,
remote memory unmap and SCIF remote fence requests.

The patch also updates the SCIF driver with minor changes required to
enable SCIF RMAs by adding the new files to the build, initializing
RMA specific information during SCIF endpoint creation, reserving SCIF
DMA channels, initializing SCIF RMA specific global data structures,
adding the IOCTL hooks required for SCIF RMAs and updating RMA
specific debugfs hooks.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/Kconfig  |   1 +
 drivers/misc/mic/scif/Makefile|   5 +
 drivers/misc/mic/scif/scif_api.c  |  33 +--
 drivers/misc/mic/scif/scif_debugfs.c  |  85 -
 drivers/misc/mic/scif/scif_epd.c  |  26 +++---
 drivers/misc/mic/scif/scif_epd.h  |  28 ++
 drivers/misc/mic/scif/scif_fd.c   | 169 +-
 drivers/misc/mic/scif/scif_main.c |  23 -
 drivers/misc/mic/scif/scif_main.h |  30 +-
 drivers/misc/mic/scif/scif_map.h  |  25 -
 drivers/misc/mic/scif/scif_nm.c   |  10 +-
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +++--
 drivers/misc/mic/scif/scif_nodeqp.h   |  42 -
 drivers/misc/mic/scif/scif_peer_bus.c |  14 +++
 14 files changed, 516 insertions(+), 40 deletions(-)

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index 1488232..60376fb 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -75,6 +75,7 @@ comment "SCIF Driver"
 config SCIF
tristate "SCIF Driver"
depends on 64BIT && PCI && X86 && SCIF_BUS
+   select IOMMU_IOVA
help
  This enables SCIF Driver support for the Intel Many Integrated
  Core (MIC) family of PCIe form factor coprocessor devices that
diff --git a/drivers/misc/mic/scif/Makefile b/drivers/misc/mic/scif/Makefile
index bf10bb7..29cfc3e 100644
--- a/drivers/misc/mic/scif/Makefile
+++ b/drivers/misc/mic/scif/Makefile
@@ -13,3 +13,8 @@ scif-objs += scif_epd.o
 scif-objs += scif_rb.o
 scif-objs += scif_nodeqp.o
 scif-objs += scif_nm.o
+scif-objs += scif_dma.o
+scif-objs += scif_fence.o
+scif-objs += scif_mmap.o
+scif-objs += scif_rma.o
+scif-objs += scif_rma_list.o
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index b47d56d..ddc9e4b 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -70,6 +70,7 @@ scif_epd_t scif_open(void)
mutex_init(>sendlock);
mutex_init(>recvlock);
 
+   scif_rma_ep_init(ep);
ep->state = SCIFEP_UNBOUND;
dev_dbg(scif_info.mdev.this_device,
"SCIFAPI open: ep %p success\n", ep);
@@ -184,8 +185,11 @@ int scif_close(scif_epd_t epd)
 
switch (oldstate) {
case SCIFEP_ZOMBIE:
+   dev_err(scif_info.mdev.this_device,
+   "SCIFAPI close: zombie state unexpected\n");
case SCIFEP_DISCONNECTED:
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
/* Remove from the disconnected list */
mutex_lock(_info.connlock);
list_for_each_safe(pos, tmpq, _info.disconnected) {
@@ -207,6 +211,7 @@ int scif_close(scif_epd_t epd)
case SCIFEP_CLOSING:
{
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
scif_disconnect_ep(ep);
break;
}
@@ -218,7 +223,7 @@ int scif_close(scif_epd_t epd)
struct scif_endpt *aep;
 
spin_unlock(>lock);
-   spin_lock(_info.eplock);
+   mutex_lock(_info.eplock);
 
/* remove from listen list */
list_for_each_safe(pos, tmpq, _info.listen) {
@@ -240,7 +245,7 @@ int scif_close(scif_epd_t epd)
break;
}
}
-   spin_unlock(_info.eplock);
+   mutex_unlock(_info.eplock);
mutex_lock(_info.connlock);
list_for_each_safe(pos, tmpq, _info.connected) {
tmpep = list_entry(pos,
@@ -260,13 +265,13 @@ int scif_close(scif_epd_t epd)
}
mutex_unlock(_info.connlock);
scif_teardown_ep(aep);
-   spin_lock(_info.eplock);
+   mutex_lock(_info.eplock);
scif_add_epd_to_zombie_list(aep, SCIF_EPLOCK_HELD);
ep->acceptcnt--;
}
 
spin_lock(>loc

[PATCH char-misc-next v2 18/22] misc: mic: SCIF RMA list operations

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the implementation for operations performed on the
list of SCIF windows. Examples of such operations includes adding the
windows to the list of registered (or cached) windows, querying the
list of self or remote windows and unregistering windows. The query
operation is used by SCIF APIs which initiate DMAs, CPU copies or
fences to ensure that a window remains valid during a transfer.

Reviewed-by: Ashutosh Dixit 
Signed-off-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma_list.c | 291 ++
 drivers/misc/mic/scif/scif_rma_list.h |  57 +++
 2 files changed, 348 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.c
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.h

diff --git a/drivers/misc/mic/scif/scif_rma_list.c 
b/drivers/misc/mic/scif/scif_rma_list.c
new file mode 100644
index 000..e1ef8da
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma_list.c
@@ -0,0 +1,291 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include 
+#include 
+
+/*
+ * scif_insert_tcw:
+ *
+ * Insert a temp window to the temp registration list sorted by va_for_temp.
+ * RMA lock must be held.
+ */
+void scif_insert_tcw(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL;
+   struct scif_window *prev = list_entry(head, struct scif_window, list);
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   /* Compare with tail and if the entry is new tail add it to the end */
+   if (!list_empty(head)) {
+   curr = list_entry(head->prev, struct scif_window, list);
+   if (curr->va_for_temp < window->va_for_temp) {
+   list_add_tail(>list, head);
+   return;
+   }
+   }
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->va_for_temp > window->va_for_temp)
+   break;
+   prev = curr;
+   }
+   list_add(>list, >list);
+}
+
+/*
+ * scif_insert_window:
+ *
+ * Insert a window to the self registration list sorted by offset.
+ * RMA lock must be held.
+ */
+void scif_insert_window(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL, *prev = NULL;
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->offset > window->offset)
+   break;
+   prev = curr;
+   }
+   if (!prev)
+   list_add(>list, head);
+   else
+   list_add(>list, >list);
+   scif_set_window_ref(window, window->nr_pages);
+}
+
+/*
+ * scif_query_tcw:
+ *
+ * Query the temp cached registration list of ep for an overlapping window
+ * in case of permission mismatch, destroy the previous window. if permissions
+ * match and overlap is partial, destroy the window but return the new range
+ * RMA lock must be held.
+ */
+int scif_query_tcw(struct scif_endpt *ep, struct scif_rma_req *req)
+{
+   struct list_head *item, *temp, *head = req->head;
+   struct scif_window *window;
+   u64 start_va_window, start_va_req = req->va_for_temp;
+   u64 end_va_window, end_va_req = start_va_req + req->nr_bytes;
+
+   if (!req->nr_bytes)
+   return -EINVAL;
+   /*
+* Avoid traversing the entire list to find out that there
+* is no entry that matches
+*/
+   if (!list_empty(head)) {
+   window = list_last_entry(head, struct scif_window, list);
+   end_va_window = window->va_for_temp +
+   (window->nr_pages << PAGE_SHIFT);
+   if (start_va_req > end_va_window)
+   return -ENXIO;
+   }
+   list_for_each_safe(item, temp, head) {
+   window = list_entry(item, struct scif_window, list);
+   start_va_window = window->va_for_temp;
+   end_va_window = window->va_for_temp +
+   (window->nr_pages << PAGE_SHIFT);
+   if (start_va_req < start_va_window &&
+   end_va_req < start_va

[PATCH char-misc-next v2 20/22] misc: mic: SCIF DMA and CPU copy interface

[Ashutosh Dixit]

From: Sudeep Dutt 

SCIF allows users to read from or write to registered remote memory
via CPU copies or DMA. The API verifies that both local and remote
windows are valid before initiating the CPU or DMA transfers. SCIF has
optimized algorithms for handling byte aligned as well as cache line
aligned DMA engines. A registration cache is maintained to avoid the
overhead of pinning pages repeatedly if buffers are reused. The
registration cache is invalidated upon receipt of MMU notifier
callbacks.  SCIF windows are destroyed and the pages are unpinned only
once all prior DMAs initiated using that window are drained. Users can
request synchronous DMA operations as well as tail byte ordering if
required. CPU copies are always performed synchronously.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_dma.c | 1979 ++
 1 file changed, 1979 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_dma.c

diff --git a/drivers/misc/mic/scif/scif_dma.c b/drivers/misc/mic/scif/scif_dma.c
new file mode 100644
index 000..95a13c6
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_dma.c
@@ -0,0 +1,1979 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include "scif_map.h"
+
+/*
+ * struct scif_dma_comp_cb - SCIF DMA completion callback
+ *
+ * @dma_completion_func: DMA completion callback
+ * @cb_cookie: DMA completion callback cookie
+ * @temp_buf: Temporary buffer
+ * @temp_buf_to_free: Temporary buffer to be freed
+ * @is_cache: Is a kmem_cache allocated buffer
+ * @dst_offset: Destination registration offset
+ * @dst_window: Destination registration window
+ * @len: Length of the temp buffer
+ * @temp_phys: DMA address of the temp buffer
+ * @sdev: The SCIF device
+ * @header_padding: padding for cache line alignment
+ */
+struct scif_dma_comp_cb {
+   void (*dma_completion_func)(void *cookie);
+   void *cb_cookie;
+   u8 *temp_buf;
+   u8 *temp_buf_to_free;
+   bool is_cache;
+   s64 dst_offset;
+   struct scif_window *dst_window;
+   size_t len;
+   dma_addr_t temp_phys;
+   struct scif_dev *sdev;
+   int header_padding;
+};
+
+/**
+ * struct scif_copy_work - Work for DMA copy
+ *
+ * @src_offset: Starting source offset
+ * @dst_offset: Starting destination offset
+ * @src_window: Starting src registered window
+ * @dst_window: Starting dst registered window
+ * @loopback: true if this is a loopback DMA transfer
+ * @len: Length of the transfer
+ * @comp_cb: DMA copy completion callback
+ * @remote_dev: The remote SCIF peer device
+ * @fence_type: polling or interrupt based
+ * @ordered: is this a tail byte ordered DMA transfer
+ */
+struct scif_copy_work {
+   s64 src_offset;
+   s64 dst_offset;
+   struct scif_window *src_window;
+   struct scif_window *dst_window;
+   int loopback;
+   size_t len;
+   struct scif_dma_comp_cb   *comp_cb;
+   struct scif_dev *remote_dev;
+   int fence_type;
+   bool ordered;
+};
+
+#ifndef list_entry_next
+#define list_entry_next(pos, member) \
+   list_entry(pos->member.next, typeof(*pos), member)
+#endif
+
+/**
+ * scif_reserve_dma_chan:
+ * @ep: Endpoint Descriptor.
+ *
+ * This routine reserves a DMA channel for a particular
+ * endpoint. All DMA transfers for an endpoint are always
+ * programmed on the same DMA channel.
+ */
+int scif_reserve_dma_chan(struct scif_endpt *ep)
+{
+   int err = 0;
+   struct scif_dev *scifdev;
+   struct scif_hw_dev *sdev;
+   struct dma_chan *chan;
+
+   /* Loopback DMAs are not supported on the management node */
+   if (!scif_info.nodeid && scifdev_self(ep->remote_dev))
+   return 0;
+   if (scif_info.nodeid)
+   scifdev = _dev[0];
+   else
+   scifdev = ep->remote_dev;
+   sdev = scifdev->sdev;
+   if (!sdev->num_dma_ch)
+   return -ENODEV;
+   chan = sdev->dma_ch[scifdev->dma_ch_idx];
+   scifdev->dma_ch_idx = (scifdev->dma_ch_idx + 1) % sdev->num_dma_ch;
+   mutex_lock(>rma_info.rma_lock);
+   ep->rma_info.dma_chan = chan;
+   mutex_unlock(>rma_info.rma_lock);
+   return err;
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+/**
+ * scif_rma_destroy_tcw:
+ *
+ * This routine destroys temporary cached windows
+

[PATCH char-misc-next v2 21/22] misc: mic: SCIF fence

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the fence APIs required to synchronize
DMAs. SCIF provides an interface to return a "mark" for all DMAs
programmed at the instant the API was called. Users can then "wait" on
the mark provided previously by blocking inside the kernel. Upon
receipt of a DMA completion interrupt the waiting thread is woken
up. There is also an interface to signal DMA completion by polling for
a location to be updated via a "signal" cookie to avoid the interrupt
overhead in the mark/wait interface. SCIF allows programming fences on
both the local and the remote node for both the mark/wait or the fence
signal APIs.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Jacek Lawrynowicz 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_fence.c | 771 +
 1 file changed, 771 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_fence.c

diff --git a/drivers/misc/mic/scif/scif_fence.c 
b/drivers/misc/mic/scif/scif_fence.c
new file mode 100644
index 000..7f2c96f
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_fence.c
@@ -0,0 +1,771 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+
+#include "scif_main.h"
+
+/**
+ * scif_recv_mark: Handle SCIF_MARK request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested a mark.
+ */
+void scif_recv_mark(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   int mark, err;
+
+   err = _scif_fence_mark(ep, );
+   if (err)
+   msg->uop = SCIF_MARK_NACK;
+   else
+   msg->uop = SCIF_MARK_ACK;
+   msg->payload[0] = ep->remote_ep;
+   msg->payload[2] = mark;
+   scif_nodeqp_send(ep->remote_dev, msg);
+}
+
+/**
+ * scif_recv_mark_resp: Handle SCIF_MARK_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_MARK message.
+ */
+void scif_recv_mark_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if (msg->uop == SCIF_MARK_ACK) {
+   fence_req->state = OP_COMPLETED;
+   fence_req->dma_mark = (int)msg->payload[2];
+   } else {
+   fence_req->state = OP_FAILED;
+   }
+   mutex_unlock(>rma_info.rma_lock);
+   complete(_req->comp);
+}
+
+/**
+ * scif_recv_wait: Handle SCIF_WAIT request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested waiting on a fence.
+ */
+void scif_recv_wait(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_remote_fence_info *fence;
+
+   /*
+* Allocate structure for remote fence information and
+* send a NACK if the allocation failed. The peer will
+* return ENOMEM upon receiving a NACK.
+*/
+   fence = kmalloc(sizeof(*fence), GFP_KERNEL);
+   if (!fence) {
+   msg->payload[0] = ep->remote_ep;
+   msg->uop = SCIF_WAIT_NACK;
+   scif_nodeqp_send(ep->remote_dev, msg);
+   return;
+   }
+
+   /* Prepare the fence request */
+   memcpy(>msg, msg, sizeof(struct scifmsg));
+   INIT_LIST_HEAD(>list);
+
+   /* Insert to the global remote fence request list */
+   mutex_lock(_info.fencelock);
+   atomic_inc(>rma_info.fence_refcount);
+   list_add_tail(>list, _info.fence);
+   mutex_unlock(_info.fencelock);
+
+   schedule_work(_info.misc_work);
+}
+
+/**
+ * scif_recv_wait_resp: Handle SCIF_WAIT_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_WAIT message.
+ */
+void scif_recv_wait_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if (msg->uop == SCIF_WAIT_ACK)
+   fence_req->state = OP_COMPLETED;
+   else
+   fence_req->state = OP_FAILED;
+   m

[PATCH char-misc-next v2 13/22] misc: mic: Remove COSM functionality from the MIC card driver

[Ashutosh Dixit]

Since card side COSM functionality, to trigger MIC device shutdowns
and communicate shutdown status to the host, is now moved into a
separate COSM client driver, this patch removes this functionality
from the base MIC card driver. The mic_bus driver is also updated to
use the device index provided by COSM rather than maintain its own
device index.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/bus/mic_bus.c | 22 ++
 drivers/misc/mic/card/mic_device.c | 88 ++
 drivers/misc/mic/card/mic_x100.c   |  2 +-
 drivers/misc/mic/common/mic_dev.h  | 13 ++
 include/linux/mic_bus.h|  3 +-
 5 files changed, 23 insertions(+), 105 deletions(-)

diff --git a/drivers/misc/mic/bus/mic_bus.c b/drivers/misc/mic/bus/mic_bus.c
index 961ae90..c64955d 100644
--- a/drivers/misc/mic/bus/mic_bus.c
+++ b/drivers/misc/mic/bus/mic_bus.c
@@ -25,9 +25,6 @@
 #include 
 #include 
 
-/* Unique numbering for mbus devices. */
-static DEFINE_IDA(mbus_index_ida);
-
 static ssize_t device_show(struct device *d,
   struct device_attribute *attr, char *buf)
 {
@@ -147,7 +144,8 @@ static void mbus_release_dev(struct device *d)
 
 struct mbus_device *
 mbus_register_device(struct device *pdev, int id, struct dma_map_ops *dma_ops,
-struct mbus_hw_ops *hw_ops, void __iomem *mmio_va)
+struct mbus_hw_ops *hw_ops, int index,
+void __iomem *mmio_va)
 {
int ret;
struct mbus_device *mbdev;
@@ -166,13 +164,7 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
mbdev->dev.release = mbus_release_dev;
mbdev->hw_ops = hw_ops;
mbdev->dev.bus = _bus;
-
-   /* Assign a unique device index and hence name. */
-   ret = ida_simple_get(_index_ida, 0, 0, GFP_KERNEL);
-   if (ret < 0)
-   goto free_mbdev;
-
-   mbdev->index = ret;
+   mbdev->index = index;
dev_set_name(>dev, "mbus-dev%u", mbdev->index);
/*
 * device_register() causes the bus infrastructure to look for a
@@ -180,10 +172,8 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
 */
ret = device_register(>dev);
if (ret)
-   goto ida_remove;
+   goto free_mbdev;
return mbdev;
-ida_remove:
-   ida_simple_remove(_index_ida, mbdev->index);
 free_mbdev:
kfree(mbdev);
return ERR_PTR(ret);
@@ -192,10 +182,7 @@ EXPORT_SYMBOL_GPL(mbus_register_device);
 
 void mbus_unregister_device(struct mbus_device *mbdev)
 {
-   int index = mbdev->index; /* save for after device release */
-
device_unregister(>dev);
-   ida_simple_remove(_index_ida, index);
 }
 EXPORT_SYMBOL_GPL(mbus_unregister_device);
 
@@ -207,7 +194,6 @@ static int __init mbus_init(void)
 static void __exit mbus_exit(void)
 {
bus_unregister(_bus);
-   ida_destroy(_index_ida);
 }
 
 core_initcall(mbus_init);
diff --git a/drivers/misc/mic/card/mic_device.c 
b/drivers/misc/mic/card/mic_device.c
index 6338908..d0edaf7 100644
--- a/drivers/misc/mic/card/mic_device.c
+++ b/drivers/misc/mic/card/mic_device.c
@@ -37,71 +37,6 @@
 #include "mic_virtio.h"
 
 static struct mic_driver *g_drv;
-static struct mic_irq *shutdown_cookie;
-
-static void mic_notify_host(u8 state)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(state, >shutdown_status);
-   dev_dbg(mdrv->dev, "%s %d system_state %d\n",
-   __func__, __LINE__, state);
-   mic_send_intr(>mdev, ioread8(>c2h_shutdown_db));
-}
-
-static int mic_panic_event(struct notifier_block *this, unsigned long event,
-   void *ptr)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(-1, >h2c_config_db);
-   iowrite8(-1, >h2c_shutdown_db);
-   mic_notify_host(MIC_CRASHED);
-   return NOTIFY_DONE;
-}
-
-static struct notifier_block mic_panic = {
-   .notifier_call  = mic_panic_event,
-};
-
-static irqreturn_t mic_shutdown_isr(int irq, void *data)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   mic_ack_interrupt(_drv->mdev);
-   if (ioread8(>shutdown_card))
-   orderly_poweroff(true);
-   return IRQ_HANDLED;
-}
-
-static int mic_shutdown_init(void)
-{
-   int rc = 0;
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-   int shutdown_db;
-
-   shutdown_db = mic_next_card_db();
-   shutdown_cookie = mic_request_card_irq(mic_shutdown_isr, NULL,
-  "Shutdown", mdrv

[PATCH char-misc-next v2 15/22] misc: mic: SCIF RMA header file and IOCTL changes

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch updates the SCIF header file and IOCTL interface with the
changes required to support RMAs.  APIs added include the ability to
pin pages and register those pages with SCIF. SCIF kernel clients can
also add references to remote registered pages and access them via the
CPU. The user space IOCTL interface has been updated to enable SCIF
registration, RDMA/CPU copies and fence APIs for RDMA synchronization.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 include/linux/scif.h| 234 ++--
 include/uapi/linux/scif_ioctl.h |  85 +++
 2 files changed, 309 insertions(+), 10 deletions(-)

diff --git a/include/linux/scif.h b/include/linux/scif.h
index fd62c05..49a35d6 100644
--- a/include/linux/scif.h
+++ b/include/linux/scif.h
@@ -93,6 +93,27 @@ enum {
 #define SCIF_PORT_RSVD 1088
 
 typedef struct scif_endpt *scif_epd_t;
+typedef struct scif_pinned_pages *scif_pinned_pages_t;
+
+/**
+ * struct scif_range - SCIF registered range used in kernel mode
+ * @cookie: cookie used internally by SCIF
+ * @nr_pages: number of pages of PAGE_SIZE
+ * @prot_flags: R/W protection
+ * @phys_addr: Array of bus addresses
+ * @va: Array of kernel virtual addresses backed by the pages in the phys_addr
+ * array. The va is populated only when called on the host for a remote
+ * SCIF connection on MIC. This is required to support the use case of DMA
+ * between MIC and another device which is not a SCIF node e.g., an IB or
+ * ethernet NIC.
+ */
+struct scif_range {
+   void *cookie;
+   int nr_pages;
+   int prot_flags;
+   dma_addr_t *phys_addr;
+   void __iomem **va;
+};
 
 /**
  * struct scif_pollepd - SCIF endpoint to be monitored via scif_poll
@@ -389,7 +410,6 @@ int scif_close(scif_epd_t epd);
  * Errors:
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -442,7 +462,6 @@ int scif_send(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The destination node is returning from a low power state
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -505,9 +524,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * SCIF_PROT_READ - allow read operations from the window
  * SCIF_PROT_WRITE - allow write operations to the window
  *
- * The map_flags argument can be set to SCIF_MAP_FIXED which interprets a
- * fixed offset.
- *
  * Return:
  * Upon successful completion, scif_register() returns the offset at which the
  * mapping was placed (po); otherwise in user mode SCIF_REGISTER_FAILED (that
@@ -520,7 +536,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The mapping could not be performed due to lack of resources
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - map_flags is invalid, or prot_flags is invalid, or SCIF_MAP_FIXED 
is
  * set in flags, and offset is not a multiple of the page size, or addr is not 
a
  * multiple of the page size, or len is not a multiple of the page size, or is
@@ -803,7 +818,6 @@ int scif_writeto(scif_epd_t epd, off_t loffset, size_t len, 
off_t
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -884,7 +898,6 @@ int scif_vreadfrom(scif_epd_t epd, void *addr, size_t len, 
off_t roffset,
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -1028,11 +1041,212 @@ int scif_fence_signal(scif_epd_t epd, off_t loff, u64 
lval, off_t roff,
  * online nodes in the SCIF network including 'self'; otherwise in user mode
  * -1 is returned and errno is set to indicate the error; in kernel mode no
  * errors are returned.
+ */
+int scif_get_node_ids(u16 *nodes, int len, u16

[PATCH char-misc-next v2 17/22] misc: mic: SCIF memory registration and unregistration

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the SCIF APIs required to pin and unpin
pages. SCIF registration locks down the pages. It then sends a remote
window allocation request to the peer. Once the peer has allocated
memory, the local SCIF endpoint copies the pinned page information to
the peer and notifies the peer once the copy has complete. The peer
upon receipt of the registration notification adds the new remote
window to its list. At this point the window page information is
available on both self and remote nodes so that they can start
performing SCIF DMAs, CPU copies and fences. The unregistration API
tears down the registration at both self and remote nodes.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma.c | 1770 ++
 1 file changed, 1770 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.c

diff --git a/drivers/misc/mic/scif/scif_rma.c b/drivers/misc/mic/scif/scif_rma.c
new file mode 100644
index 000..e288996
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.c
@@ -0,0 +1,1770 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include 
+#include 
+#include "scif_main.h"
+#include "scif_map.h"
+
+/* Used to skip ulimit checks for registrations with SCIF_MAP_KERNEL flag */
+#define SCIF_MAP_ULIMIT 0x40
+
+bool scif_ulimit_check = 1;
+
+/**
+ * scif_rma_ep_init:
+ * @ep: end point
+ *
+ * Initialize RMA per EP data structures.
+ */
+void scif_rma_ep_init(struct scif_endpt *ep)
+{
+   struct scif_endpt_rma_info *rma = >rma_info;
+
+   mutex_init(>rma_lock);
+   init_iova_domain(>iovad, PAGE_SIZE, SCIF_IOVA_START_PFN,
+SCIF_DMA_64BIT_PFN);
+   spin_lock_init(>tc_lock);
+   mutex_init(>mmn_lock);
+   INIT_LIST_HEAD(>reg_list);
+   INIT_LIST_HEAD(>remote_reg_list);
+   atomic_set(>tw_refcount, 0);
+   atomic_set(>tcw_refcount, 0);
+   atomic_set(>tcw_total_pages, 0);
+   atomic_set(>fence_refcount, 0);
+
+   rma->async_list_del = 0;
+   rma->dma_chan = NULL;
+   INIT_LIST_HEAD(>mmn_list);
+   INIT_LIST_HEAD(>vma_list);
+   init_waitqueue_head(>markwq);
+}
+
+/**
+ * scif_rma_ep_can_uninit:
+ * @ep: end point
+ *
+ * Returns 1 if an endpoint can be uninitialized and 0 otherwise.
+ */
+int scif_rma_ep_can_uninit(struct scif_endpt *ep)
+{
+   int ret = 0;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Destroy RMA Info only if both lists are empty */
+   if (list_empty(>rma_info.reg_list) &&
+   list_empty(>rma_info.remote_reg_list) &&
+   list_empty(>rma_info.mmn_list) &&
+   !atomic_read(>rma_info.tw_refcount) &&
+   !atomic_read(>rma_info.tcw_refcount) &&
+   !atomic_read(>rma_info.fence_refcount))
+   ret = 1;
+   mutex_unlock(>rma_info.rma_lock);
+   return ret;
+}
+
+/**
+ * scif_create_pinned_pages:
+ * @nr_pages: number of pages in window
+ * @prot: read/write protection
+ *
+ * Allocate and prepare a set of pinned pages.
+ */
+static struct scif_pinned_pages *
+scif_create_pinned_pages(int nr_pages, int prot)
+{
+   struct scif_pinned_pages *pin;
+
+   might_sleep();
+   pin = scif_zalloc(sizeof(*pin));
+   if (!pin)
+   goto error;
+
+   pin->pages = scif_zalloc(nr_pages * sizeof(*pin->pages));
+   if (!pin->pages)
+   goto error_free_pinned_pages;
+
+   pin->prot = prot;
+   pin->magic = SCIFEP_MAGIC;
+   return pin;
+
+error_free_pinned_pages:
+   scif_free(pin, sizeof(*pin));
+error:
+   return NULL;
+}
+
+/**
+ * scif_destroy_pinned_pages:
+ * @pin: A set of pinned pages.
+ *
+ * Deallocate resources for pinned pages.
+ */
+static int scif_destroy_pinned_pages(struct scif_pinned_pages *pin)
+{
+   int j;
+   int writeable = pin->prot & SCIF_PROT_WRITE;
+   int kernel = SCIF_MAP_KERNEL & pin->map_flags;
+
+   for (j = 0; j < pin->nr_pages; j++) {
+   if (pin->pages[j] && !kernel) {
+   if (writeable)
+   SetPageDirty(pin->pages[j]);
+   put_page(pin->pages[j]);
+   }
+   }
+
+   scif_free(pin->pages,
+ p

[PATCH char-misc-next v2 14/22] misc: mic: Update MIC host daemon with COSM changes

[Ashutosh Dixit]

This patch updates the MIC host daemon to work with corresponding
changes in COSM. Other MIC daemon fixes, cleanups and enhancements as
are also rolled into this patch. Changes to MIC sysfs ABI which go
into effect with this patch are also documented.

Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 Documentation/ABI/testing/sysfs-class-mic.txt |  29 ++-
 Documentation/mic/mic_overview.txt|  31 ++-
 Documentation/mic/mpssd/mpss  |   4 +-
 Documentation/mic/mpssd/mpssd.c   | 362 --
 Documentation/mic/mpssd/mpssd.h   |   1 +
 drivers/misc/mic/Makefile |   4 +
 drivers/misc/mic/bus/scif_bus.c   |   7 +-
 drivers/misc/mic/bus/scif_bus.h   |   6 +-
 include/uapi/linux/mic_common.h   |  16 +-
 9 files changed, 285 insertions(+), 175 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-class-mic.txt 
b/Documentation/ABI/testing/sysfs-class-mic.txt
index 13f48af..d45eed2 100644
--- a/Documentation/ABI/testing/sysfs-class-mic.txt
+++ b/Documentation/ABI/testing/sysfs-class-mic.txt
@@ -41,18 +41,15 @@ Description:
When read, this entry provides the current state of an Intel
MIC device in the context of the card OS. Possible values that
will be read are:
-   "offline" - The MIC device is ready to boot the card OS. On
+   "ready" - The MIC device is ready to boot the card OS. On
reading this entry after an OSPM resume, a "boot" has to be
written to this entry if the card was previously shutdown
during OSPM suspend.
-   "online" - The MIC device has initiated booting a card OS.
+   "booting" - The MIC device has initiated booting a card OS.
+   "online" - The MIC device has completed boot and is online
"shutting_down" - The card OS is shutting down.
+   "resetting" - A reset has been initiated for the MIC device
"reset_failed" - The MIC device has failed to reset.
-   "suspending" - The MIC device is currently being prepared for
-   suspend. On reading this entry, a "suspend" has to be written
-   to the state sysfs entry to ensure the card is shutdown during
-   OSPM suspend.
-   "suspended" - The MIC device has been suspended.
 
When written, this sysfs entry triggers different state change
operations depending upon the current state of the card OS.
@@ -62,8 +59,6 @@ Description:
sysfs entries.
"reset" - Initiates device reset.
"shutdown" - Initiates card OS shutdown.
-   "suspend" - Initiates card OS shutdown and also marks the card
-   as suspended.
 
 What:  /sys/class/mic/mic(x)/shutdown_status
 Date:  October 2013
@@ -126,7 +121,7 @@ Description:
the card. This sysfs entry can be written with the following
valid strings:
a) linux - Boot a Linux image.
-   b) elf - Boot an elf image for flash updates.
+   b) flash - Boot an image for flash updates.
 
 What:  /sys/class/mic/mic(x)/log_buf_addr
 Date:  October 2013
@@ -155,3 +150,17 @@ Description:
daemon to set the log buffer length address. The correct log
buffer length address to be written can be found in the
System.map file of the card OS.
+
+What:  /sys/class/mic/mic(x)/heartbeat_enable
+Date:  March 2015
+KernelVersion: 3.20
+Contact:   Ashutosh Dixit 
+Description:
+   The MIC drivers detect and inform user space about card crashes
+   via a heartbeat mechanism (see the description of
+   shutdown_status above). User space can turn off this
+   notification by setting heartbeat_enable to 0 and enable it by
+   setting this entry to 1. If this notification is disabled it is
+   the responsibility of user space to detect card crashes via
+   alternative means such as a network ping. This setting is
+   enabled by default.
diff --git a/Documentation/mic/mic_overview.txt 
b/Documentation/mic/mic_overview.txt
index 1a2f2c8..73f44fc 100644
--- a/Documentation/mic/mic_overview.txt
+++ b/Documentation/mic/mic_overview.txt
@@ -28,6 +28,10 @@ The Symmetric Communication Interface (SCIF (pronounced as 
skiff)) is a
 low level communications API across PCIe currently implemented for MIC.
 More details are available at scif_overview.txt.
 
+The Coprocessor State Management (COSM) driver on 

[PATCH char-misc-next v2 12/22] misc: mic: Remove COSM functionality from the MIC host driver

[Ashutosh Dixit]

Since COSM functionality is now moved into a separate COSM driver
drivers, this patch removes this functionality from the base MIC host
driver. The MIC host driver now implements cosm_hw_ops and registers a
COSM device which allows the COSM driver to trigger
boot/shutdown/reset of the MIC devices via the cosm_hw_ops.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/Makefile   |   2 -
 drivers/misc/mic/host/Makefile  |   1 -
 drivers/misc/mic/host/mic_boot.c| 317 +++--
 drivers/misc/mic/host/mic_debugfs.c | 114 +
 drivers/misc/mic/host/mic_device.h  |  88 +--
 drivers/misc/mic/host/mic_fops.c|   4 +-
 drivers/misc/mic/host/mic_intr.c|  46 ++--
 drivers/misc/mic/host/mic_main.c| 223 +++---
 drivers/misc/mic/host/mic_smpt.c|  30 ++-
 drivers/misc/mic/host/mic_sysfs.c   | 459 
 drivers/misc/mic/host/mic_virtio.c  |  17 +-
 drivers/misc/mic/host/mic_virtio.h  |   2 +-
 drivers/misc/mic/host/mic_x100.c|  46 ++--
 13 files changed, 204 insertions(+), 1145 deletions(-)
 delete mode 100644 drivers/misc/mic/host/mic_sysfs.c

diff --git a/drivers/misc/mic/Makefile b/drivers/misc/mic/Makefile
index a74042c..a2e75f4 100644
--- a/drivers/misc/mic/Makefile
+++ b/drivers/misc/mic/Makefile
@@ -2,7 +2,5 @@
 # Makefile - Intel MIC Linux driver.
 # Copyright(c) 2013, Intel Corporation.
 #
-obj-$(CONFIG_INTEL_MIC_HOST) += host/
-obj-$(CONFIG_INTEL_MIC_CARD) += card/
 obj-y += bus/
 obj-$(CONFIG_SCIF) += scif/
diff --git a/drivers/misc/mic/host/Makefile b/drivers/misc/mic/host/Makefile
index c2197f9..004d3db 100644
--- a/drivers/misc/mic/host/Makefile
+++ b/drivers/misc/mic/host/Makefile
@@ -5,7 +5,6 @@
 obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
 mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
-mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_intr.o
 mic_host-objs += mic_boot.o
diff --git a/drivers/misc/mic/host/mic_boot.c b/drivers/misc/mic/host/mic_boot.c
index e5f6a5e..7845564 100644
--- a/drivers/misc/mic/host/mic_boot.c
+++ b/drivers/misc/mic/host/mic_boot.c
@@ -22,9 +22,9 @@
 #include 
 #include 
 #include 
-
 #include 
 #include 
+#include "../bus/scif_bus.h"
 #include "../common/mic_dev.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
@@ -99,7 +99,7 @@ static int __mic_dma_map_sg(struct device *dev, struct 
scatterlist *sg,
int i, j, ret;
dma_addr_t da;
 
-   ret = dma_map_sg(mdev->sdev->parent, sg, nents, dir);
+   ret = dma_map_sg(>pdev->dev, sg, nents, dir);
if (ret <= 0)
return 0;
 
@@ -115,7 +115,7 @@ err:
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = mic_to_dma_addr(mdev, sg_dma_address(s));
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
return 0;
 }
 
@@ -135,7 +135,7 @@ static void __mic_dma_unmap_sg(struct device *dev,
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = da;
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
 }
 
 static struct dma_map_ops __mic_dma_ops = {
@@ -270,48 +270,13 @@ static struct mbus_hw_ops mbus_hw_ops = {
.ack_interrupt = _mic_ack_interrupt,
 };
 
-/**
- * mic_reset - Reset the MIC device.
- * @mdev: pointer to mic_device instance
- */
-static void mic_reset(struct mic_device *mdev)
-{
-   int i;
-
-#define MIC_RESET_TO (45)
-
-   reinit_completion(>reset_wait);
-   mdev->ops->reset_fw_ready(mdev);
-   mdev->ops->reset(mdev);
-
-   for (i = 0; i < MIC_RESET_TO; i++) {
-   if (mdev->ops->is_fw_ready(mdev))
-   goto done;
-   /*
-* Resets typically take 10s of seconds to complete.
-* Since an MMIO read is required to check if the
-* firmware is ready or not, a 1 second delay works nicely.
-*/
-   msleep(1000);
-   }
-   mic_set_state(mdev, MIC_RESET_FAILED);
-done:
-   complete_all(>reset_wait);
-}
-
 /* Initialize the MIC bootparams */
 void mic_bootparam_init(struct mic_device *mdev)
 {
struct mic_bootparam *bootparam = mdev->dp;
 
bootparam->magic = cpu_to_le32(MIC_MAGIC);
-   bootparam->c2h_shutdown_db = mdev->shutdown_db;
-   bootparam->h2c_shutdown_db = -1;
bootparam->h2c_config_db = -1;
-   bootparam->shutdown_status = 0;
-   bootparam->shutdown_card = 0;
-   /* Total nodes = number of MICs + 1 for self node */
-   bootparam->tot_nodes = atomic_read(_num_mics) + 1;
bootparam-&

[PATCH char-misc-next v2 16/22] misc: mic: SCIF RMA header file

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the internal data structures required to perform SCIF
RMAs. The data structures required to maintain per SCIF endpoint, RMA
information are contained in scif_endpt_rma_info. scif_pinned_pages
describes a set of SCIF pinned pages maintained locally. The
scif_window is a data structure which contains all the fields required
to describe a SCIF registered window on self and remote nodes. It
contains an offset which is used as a key to perform SCIF DMAs and CPU
copies between self and remote registered windows.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma.h | 464 +++
 1 file changed, 464 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.h

diff --git a/drivers/misc/mic/scif/scif_rma.h b/drivers/misc/mic/scif/scif_rma.h
new file mode 100644
index 000..fa67222
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.h
@@ -0,0 +1,464 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ *   notice, this list of conditions and the following disclaimer in
+ *   the documentation and/or other materials provided with the
+ *   distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ *   contributors may be used to endorse or promote products derived
+ *   from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#ifndef SCIF_RMA_H
+#define SCIF_RMA_H
+
+#include 
+#include 
+
+#include "../bus/scif_bus.h"
+
+/* If this bit is set then the mark is a remote fence mark */
+#define SCIF_REMOTE_FENCE_BIT  31
+/* Magic value used to indicate a remote fence request */
+#define SCIF_REMOTE_FENCE BIT_ULL(SCIF_REMOTE_FENCE_BIT)
+
+#define SCIF_MAX_UNALIGNED_BUF_SIZE (1024 * 1024ULL)
+#define SCIF_KMEM_UNALIGNED_BUF_SIZE (SCIF_MAX_UNALIGNED_BUF_SIZE + \
+ (L1_CACHE_BYTES << 1))
+
+#define SCIF_IOVA_START_PFN(1)
+#define SCIF_IOVA_PFN(addr) ((addr) >> PAGE_SHIFT)
+#define SCIF_DMA_64BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(64))
+#define SCIF_DMA_63BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(63))
+
+/*
+ * struct scif_endpt_rma_info - Per Endpoint Remote Memory Access Information
+ *
+ * @reg_list: List of registration windows for self
+ * @remote_reg_list: List of registration windows for peer
+ * @iovad: Offset generator
+ * @rma_lock: Synchronizes access to self/remote list and also protects the
+ *   window from being destroyed while RMAs are in progress.
+ * @tc_lock: Synchronizes access to temporary cached windows list
+ *  for SCIF Registration Caching.
+ * @mmn_lock: Synchronizes access to the list of MMU notifiers registered
+ * @tw_refcount: Keeps track of number of outstanding temporary registered
+ *  windows created by scif_vreadfrom/scif_vwriteto which have
+ *  not been destroyed.
+ * @tcw_refcount: Same as tw_refcount but for temporary cached windows
+ * @tcw_total_pages: Same as tcw_refcount but in terms of

[PATCH char-misc-next v2 11/22] misc: mic: COSM client driver

[Ashutosh Dixit]

The COSM client driver running on the MIC cards is implemented as a
kernel mode SCIF client. It responds to a "shutdown" message from the
host by triggering a card shutdown and also communicates the shutdown
or reboot status back the host. It is also responsible for syncing the
card time to that of the host. Because SCIF messaging cannot be used
in a panic context, the COSM client driver also periodically sends a
heartbeat SCIF message to the host thereby enabling the host to detect
card crashes.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm_client/Makefile   |   7 +
 drivers/misc/mic/cosm_client/cosm_scif_client.c | 275 
 2 files changed, 282 insertions(+)
 create mode 100644 drivers/misc/mic/cosm_client/Makefile
 create mode 100644 drivers/misc/mic/cosm_client/cosm_scif_client.c

diff --git a/drivers/misc/mic/cosm_client/Makefile 
b/drivers/misc/mic/cosm_client/Makefile
new file mode 100644
index 000..6f751a5
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/Makefile
@@ -0,0 +1,7 @@
+#
+# Makefile - Intel MIC COSM Client Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += cosm_client.o
+
+cosm_client-objs += cosm_scif_client.o
diff --git a/drivers/misc/mic/cosm_client/cosm_scif_client.c 
b/drivers/misc/mic/cosm_client/cosm_scif_client.c
new file mode 100644
index 000..03e98bf
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/cosm_scif_client.c
@@ -0,0 +1,275 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Client Driver
+ *
+ */
+#include 
+#include 
+#include 
+#include 
+#include "../cosm/cosm_main.h"
+
+#define COSM_SCIF_MAX_RETRIES 10
+#define COSM_HEARTBEAT_SEND_MSEC (COSM_HEARTBEAT_SEND_SEC * MSEC_PER_SEC)
+
+static struct task_struct *client_thread;
+static scif_epd_t client_epd;
+static struct scif_peer_dev *client_spdev;
+
+/*
+ * Reboot notifier: receives shutdown status from the OS and communicates it
+ * back to the COSM process on the host
+ */
+static int cosm_reboot_event(struct notifier_block *this, unsigned long event,
+void *ptr)
+{
+   struct cosm_msg msg = { .id = COSM_MSG_SHUTDOWN_STATUS };
+   int rc;
+
+   event = (event == SYS_RESTART) ? SYSTEM_RESTART : event;
+   dev_info(_spdev->dev, "%s %d received event %ld\n",
+__func__, __LINE__, event);
+
+   msg.shutdown_status = event;
+   rc = scif_send(client_epd, , sizeof(msg), SCIF_SEND_BLOCK);
+   if (rc < 0)
+   dev_err(_spdev->dev, "%s %d scif_send rc %d\n",
+   __func__, __LINE__, rc);
+
+   return NOTIFY_DONE;
+}
+
+static struct notifier_block cosm_reboot = {
+   .notifier_call  = cosm_reboot_event,
+};
+
+/* Set system time from timespec value received from the host */
+static void cosm_set_time(struct cosm_msg *msg)
+{
+   int rc = do_settimeofday64(>timespec);
+
+   if (rc)
+   dev_err(_spdev->dev, "%s: %d settimeofday rc %d\n",
+   __func__, __LINE__, rc);
+}
+
+/* COSM client receive message processing */
+static void cosm_client_recv(void)
+{
+   struct cosm_msg msg;
+   int rc;
+
+   while (1) {
+   rc = scif_recv(client_epd, , sizeof(msg), 0);
+   if (!rc) {
+   return;
+   } else if (rc < 0) {
+   dev_err(_spdev->dev, "%s: %d rc %d\n",
+   __func__, __LINE__, rc);
+   return;
+   }
+
+   dev_dbg(_spdev->dev, "%s: %d rc %d id 0x%llx\n",
+   __func__, __LINE__, rc, msg.id);
+
+   switch (msg.id) {
+   case COSM_MSG_SYNC_TIME:
+   cosm_set_time();
+   break;
+   case COSM_MSG_SHUTDOWN:
+   orderly_poweroff(true);
+   break;
+   default:
+   dev_err(_spdev->dev, "%s: %d unknown id %lld\n",
+   __func__, __LINE__, msg.id);
+   break;
+   }
+   }
+}
+
+/* Initiate connection to the COSM server on the host */
+static in

[PATCH char-misc-next v2 10/22] misc: mic: COSM SCIF server

[Ashutosh Dixit]

The COSM driver communicates with the MIC cards over SCIF. A SCIF
"server" listens for incoming connections from "client" MIC cards as
they boot. After the connection is accepted a separate work item is
scheduled for each MIC card. This work item normally stays blocked in
scif_poll but wakes up to process messages from the card.

The SCIF connection between the host and card COSM components is used
to (a) send the command to shut down the card (b) receive shutdown
status back from the card upon completion of shutdown (c) receive
periodic heartbeat messages to detect card crashes (d) send host time
to the card to enable the card to sync its time to the host.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm/cosm_scif_server.c | 405 +++
 1 file changed, 405 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/cosm_scif_server.c

diff --git a/drivers/misc/mic/cosm/cosm_scif_server.c 
b/drivers/misc/mic/cosm/cosm_scif_server.c
new file mode 100644
index 000..5696df4
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_scif_server.c
@@ -0,0 +1,405 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+#include 
+#include "cosm_main.h"
+
+/*
+ * The COSM driver uses SCIF to communicate between the management node and the
+ * MIC cards. SCIF is used to (a) Send a shutdown command to the card (b)
+ * receive a shutdown status back from the card upon completion of shutdown and
+ * (c) receive periodic heartbeat messages from the card used to deduce if the
+ * card has crashed.
+ *
+ * A COSM server consisting of a SCIF listening endpoint waits for incoming
+ * connections from the card. Upon acceptance of the connection, a separate
+ * work-item is scheduled to handle SCIF message processing for that card. The
+ * life-time of this work-item is therefore the time from which the connection
+ * from a card is accepted to the time at which the connection is closed. A new
+ * work-item starts each time the card boots and is alive till the card (a)
+ * shuts down (b) is reset (c) crashes (d) cosm_client driver on the card is
+ * unloaded.
+ *
+ * From the point of view of COSM interactions with SCIF during card
+ * shutdown, reset and crash are as follows:
+ *
+ * Card shutdown
+ * -
+ * 1. COSM client on the card invokes orderly_poweroff() in response to 
SHUTDOWN
+ *message from the host.
+ * 2. Card driver shutdown callback invokes scif_unregister_device(..) 
resulting
+ *in scif_remove(..) getting called on the card
+ * 3. scif_remove -> scif_stop -> scif_handle_remove_node ->
+ *scif_peer_unregister_device -> device_unregister for the host peer device
+ * 4. During device_unregister remove(..) method of cosm_client is invoked 
which
+ *closes the COSM SCIF endpoint on the card. This results in a SCIF_DISCNCT
+ *message being sent to host SCIF. SCIF_DISCNCT message processing on the
+ *host SCIF sets the host COSM SCIF endpoint state to DISCONNECTED and 
wakes
+ *up the host COSM thread blocked in scif_poll(..) resulting in
+ *scif_poll(..)  returning POLLHUP.
+ * 5. On the card, scif_peer_release_dev is next called which results in an
+ *SCIF_EXIT message being sent to the host and after receiving the
+ *SCIF_EXIT_ACK from the host the peer device teardown on the card is
+ *complete.
+ * 6. As part of the SCIF_EXIT message processing on the host, host sends a
+ *SCIF_REMOVE_NODE to itself corresponding to the card being removed. This
+ *starts a similar SCIF peer device teardown sequence on the host
+ *corresponding to the card being shut down.
+ *
+ * Card reset
+ * --
+ * The case of interest here is when the card has not been previously shut down
+ * since most of the steps below are skipped in that case:
+
+ * 1. cosm_stop(..) invokes hw_ops->stop(..) method of the base PCIe driver
+ *which unregisters the SCIF HW device resulting in scif_remove(..) being
+ *called on the host.
+ * 2. scif_remove(..) calls scif_disconnect_node(..) which results in a
+ *SCIF_EXIT message being sent to the card.
+ * 3. The card executes scif_stop() as part of SCIF_EXIT message
+ *processing. This results in the COSM endpoint on the 

[PATCH char-misc-next v2 04/22] iommu: Allow iova to be used without requiring IOMMU_SUPPORT

[Ashutosh Dixit]

From: Sudeep Dutt 

iova is a library which can be built without IOMMU_SUPPORT

Signed-off-by: Sudeep Dutt 
---
 drivers/iommu/Kconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index d9da766..71d1c46 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -12,6 +12,9 @@ menuconfig IOMMU_SUPPORT
  remap DMA requests and/or remap interrupts from other devices on the
  system.
 
+config IOMMU_IOVA
+   tristate
+
 if IOMMU_SUPPORT
 
 menu "Generic IOMMU Pagetable Support"
@@ -42,9 +45,6 @@ config IOMMU_IO_PGTABLE_LPAE_SELFTEST
 
 endmenu
 
-config IOMMU_IOVA
-   tristate
-
 config OF_IOMMU
def_bool y
depends on OF && IOMMU_API
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 06/22] misc: mic: SCIF poll

[Ashutosh Dixit]

SCIF poll allows both user and kernel mode clients to wait on
events on a SCIF endpoint. These events include availability of
space or data in the SCIF ring buffer, availability of connection
requests on a listening endpoint and completion of connections
when using async connects.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/scif/scif_api.c  | 158 +-
 drivers/misc/mic/scif/scif_epd.h  |  22 ++
 drivers/misc/mic/scif/scif_fd.c   |   9 +++
 drivers/misc/mic/scif/scif_main.h |   2 +
 include/linux/scif.h  |  74 ++
 5 files changed, 264 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index f39d313..bf2d70f 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -37,9 +37,21 @@ enum conn_async_state {
ASYNC_CONN_FLUSH_WORK   /* async work flush in progress  */
 };
 
+/*
+ * File operations for anonymous inode file associated with a SCIF endpoint,
+ * used in kernel mode SCIF poll. Kernel mode SCIF poll calls portions of the
+ * poll API in the kernel and these take in a struct file *. Since a struct
+ * file is not available to kernel mode SCIF, it uses an anonymous file for
+ * this purpose.
+ */
+const struct file_operations scif_anon_fops = {
+   .owner = THIS_MODULE,
+};
+
 scif_epd_t scif_open(void)
 {
struct scif_endpt *ep;
+   int err;
 
might_sleep();
ep = kzalloc(sizeof(*ep), GFP_KERNEL);
@@ -50,6 +62,10 @@ scif_epd_t scif_open(void)
if (!ep->qp_info.qp)
goto err_qp_alloc;
 
+   err = scif_anon_inode_getfile(ep);
+   if (err)
+   goto err_anon_inode;
+
spin_lock_init(>lock);
mutex_init(>sendlock);
mutex_init(>recvlock);
@@ -59,6 +75,8 @@ scif_epd_t scif_open(void)
"SCIFAPI open: ep %p success\n", ep);
return ep;
 
+err_anon_inode:
+   kfree(ep->qp_info.qp);
 err_qp_alloc:
kfree(ep);
 err_ep_alloc:
@@ -279,6 +297,7 @@ int scif_close(scif_epd_t epd)
}
}
scif_put_port(ep->port.port);
+   scif_anon_inode_fput(ep);
scif_teardown_ep(ep);
scif_add_epd_to_zombie_list(ep, !SCIF_EPLOCK_HELD);
return 0;
@@ -558,8 +577,10 @@ void scif_conn_handler(struct work_struct *work)
list_del(>conn_list);
}
spin_unlock(_info.nb_connect_lock);
-   if (ep)
+   if (ep) {
ep->conn_err = scif_conn_func(ep);
+   wake_up_interruptible(>conn_pend_wq);
+   }
} while (ep);
 }
 
@@ -660,6 +681,7 @@ int __scif_connect(scif_epd_t epd, struct scif_port_id 
*dst, bool non_block)
ep->remote_dev = _dev[dst->node];
ep->qp_info.qp->magic = SCIFEP_MAGIC;
if (ep->conn_async_state == ASYNC_CONN_INPROGRESS) {
+   init_waitqueue_head(>conn_pend_wq);
spin_lock(_info.nb_connect_lock);
list_add_tail(>conn_list, _info.nb_connect_list);
spin_unlock(_info.nb_connect_lock);
@@ -788,6 +810,10 @@ retry_connection:
goto scif_accept_error_qpalloc;
}
 
+   err = scif_anon_inode_getfile(cep);
+   if (err)
+   goto scif_accept_error_anon_inode;
+
cep->qp_info.qp->magic = SCIFEP_MAGIC;
spdev = scif_get_peer_dev(cep->remote_dev);
if (IS_ERR(spdev)) {
@@ -858,6 +884,8 @@ retry:
spin_unlock(>lock);
return 0;
 scif_accept_error_map:
+   scif_anon_inode_fput(cep);
+scif_accept_error_anon_inode:
scif_teardown_ep(cep);
 scif_accept_error_qpalloc:
kfree(cep);
@@ -1247,6 +1275,134 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags)
 }
 EXPORT_SYMBOL_GPL(scif_recv);
 
+static inline void _scif_poll_wait(struct file *f, wait_queue_head_t *wq,
+  poll_table *p, struct scif_endpt *ep)
+{
+   /*
+* Because poll_wait makes a GFP_KERNEL allocation, give up the lock
+* and regrab it afterwards. Because the endpoint state might have
+* changed while the lock was given up, the state must be checked
+* again after re-acquiring the lock. The code in __scif_pollfd(..)
+* does this.
+*/
+   spin_unlock(>lock);
+   poll_wait(f, wq, p);
+   spin_lock(>lock);
+}
+
+unsigned int
+__scif_pollfd(struct file *f, poll_table *wait, struct scif_endpt *ep)
+{
+   unsigned int mask = 0;
+
+   dev_dbg(scif_info.mdev.this_device,
+   "SCIFAPI pollfd: ep %p %s\n", ep, scif_ep_states[ep->state]);
+
+   spin_lock(>lock);
+
+   /* Endpoint is waiting for a non-blocking connect to complete */
+   if (ep->conn_asy

[PATCH char-misc-next v2 09/22] misc: mic: Coprocessor State Management (COSM) driver

[Ashutosh Dixit]

The COSM driver allows boot, shutdown and reset of Intel MIC devices
via sysfs. This functionality was previously present in the Intel MIC
host driver but has now been taken out into a separate driver so that
it can be shared between multiple generations of Intel MIC products.
The sysfs kernel ABI used by the COSM driver is the same as that
defined originally for the MIC host driver in
Documentation/ABI/testing/sysfs-class-mic.txt.

The COSM driver also contains support for dumping the MIC card log_buf
and doing a "force reset" for the card via debugfs. The OSPM support
present in the MIC host driver has now largely been moved to user
space and only a small required OSPM functionality is now present in
the driver.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm/Makefile   |  10 +
 drivers/misc/mic/cosm/cosm_debugfs.c | 155 
 drivers/misc/mic/cosm/cosm_main.c| 388 +
 drivers/misc/mic/cosm/cosm_main.h|  70 ++
 drivers/misc/mic/cosm/cosm_sysfs.c   | 461 +++
 5 files changed, 1084 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/Makefile
 create mode 100644 drivers/misc/mic/cosm/cosm_debugfs.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.h
 create mode 100644 drivers/misc/mic/cosm/cosm_sysfs.c

diff --git a/drivers/misc/mic/cosm/Makefile b/drivers/misc/mic/cosm/Makefile
new file mode 100644
index 000..b85d4d4
--- /dev/null
+++ b/drivers/misc/mic/cosm/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile - Intel MIC Coprocessor State Management (COSM) Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += mic_cosm.o
+
+mic_cosm-objs := cosm_main.o
+mic_cosm-objs += cosm_debugfs.o
+mic_cosm-objs += cosm_sysfs.o
+mic_cosm-objs += cosm_scif_server.o
diff --git a/drivers/misc/mic/cosm/cosm_debugfs.c 
b/drivers/misc/mic/cosm/cosm_debugfs.c
new file mode 100644
index 000..30d953d
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_debugfs.c
@@ -0,0 +1,155 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+
+#include 
+#include 
+#include "cosm_main.h"
+
+/* Debugfs parent dir */
+static struct dentry *cosm_dbg;
+
+/**
+ * cosm_log_buf_show - Display MIC kernel log buffer
+ *
+ * log_buf addr/len is read from System.map by user space
+ * and populated in sysfs entries.
+ */
+static int cosm_log_buf_show(struct seq_file *s, void *unused)
+{
+   void __iomem *log_buf_va;
+   int __iomem *log_buf_len_va;
+   struct cosm_device *cdev = s->private;
+   void *kva;
+   int size;
+   u64 aper_offset;
+
+   if (!cdev || !cdev->log_buf_addr || !cdev->log_buf_len)
+   goto done;
+
+   mutex_lock(>cosm_mutex);
+   switch (cdev->state) {
+   case MIC_BOOTING:
+   case MIC_ONLINE:
+   case MIC_SHUTTING_DOWN:
+   break;
+   default:
+   goto unlock;
+   }
+
+   /*
+* Card kernel will never be relocated and any kernel text/data mapping
+* can be translated to phys address by subtracting __START_KERNEL_map.
+*/
+   aper_offset = (u64)cdev->log_buf_len - __START_KERNEL_map;
+   log_buf_len_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+   aper_offset = (u64)cdev->log_buf_addr - __START_KERNEL_map;
+   log_buf_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+
+   size = ioread32(log_buf_len_va);
+   kva = kmalloc(size, GFP_KERNEL);
+   if (!kva)
+   goto unlock;
+
+   memcpy_fromio(kva, log_buf_va, size);
+   seq_write(s, kva, size);
+   kfree(kva);
+unlock:
+   mutex_unlock(>cosm_mutex);
+done:
+   return 0;
+}
+
+static int cosm_log_buf_open(struct inode *inode, struct file *file)
+{
+   return single_open(file, cosm_log_buf_show, inode->i_private);
+}
+
+static const struct file_operations log_buf_ops = {
+   .owner   = THIS_MODULE,
+   .open= cosm_log_buf_open,
+   .read= seq_read,
+   .llseek  = seq_lseek,
+   .release = single_release
+};
+
+/**
+ * cosm_force_reset_show - Force MIC reset
+ *
+ * Invokes the f

[PATCH char-misc-next v2 08/22] misc: mic: MIC COSM bus

[Ashutosh Dixit]

The MIC COSM bus allows the co-processor state management (COSM)
functionality to be shared between multiple generations of Intel MIC
products. The COSM driver registers itself on the COSM bus. The base
PCIe drivers implement the bus ops and register COSM devices on the
bus, resulting in the COSM driver being probed with the COSM devices.
COSM bus ops, e.g. start, stop, ready, reset, therefore abstract out
common functionality from its specific implementation for individual
generations of MIC products.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/Kconfig|  22 ++-
 drivers/misc/mic/bus/Makefile   |   1 +
 drivers/misc/mic/bus/cosm_bus.c | 141 
 drivers/misc/mic/bus/cosm_bus.h | 134 ++
 4 files changed, 296 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/mic/bus/cosm_bus.c
 create mode 100644 drivers/misc/mic/bus/cosm_bus.h

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index e9f2f56..1488232 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -36,7 +36,7 @@ comment "Intel MIC Host Driver"
 
 config INTEL_MIC_HOST
tristate "Intel MIC Host Driver"
-   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VHOST_RING
help
  This enables Host Driver support for the Intel Many Integrated
@@ -56,7 +56,7 @@ comment "Intel MIC Card Driver"
 
 config INTEL_MIC_CARD
tristate "Intel MIC Card Driver"
-   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VIRTIO
help
  This enables card driver support for the Intel Many Integrated
@@ -88,3 +88,21 @@ config SCIF
  More information about the Intel MIC family as well as the Linux
  OS and tools for MIC to use with this driver are available from
  <http://software.intel.com/en-us/mic-developer>.
+
+comment "Intel MIC Coprocessor State Management (COSM) Drivers"
+
+config MIC_COSM
+   tristate "Intel MIC Coprocessor State Management (COSM) Drivers"
+   depends on 64BIT && PCI && X86 && SCIF
+   help
+ This enables COSM driver support for the Intel Many
+ Integrated Core (MIC) family of PCIe form factor coprocessor
+ devices. COSM drivers implement functions such as boot,
+ shutdown, reset and reboot of MIC devices.
+
+ If you are building a host kernel with an Intel MIC device then
+ say M (recommended) or Y, else say N. If unsure say N.
+
+ More information about the Intel MIC family as well as the Linux
+ OS and tools for MIC to use with this driver are available from
+ <http://software.intel.com/en-us/mic-developer>.
diff --git a/drivers/misc/mic/bus/Makefile b/drivers/misc/mic/bus/Makefile
index 1ed37e2..761842b 100644
--- a/drivers/misc/mic/bus/Makefile
+++ b/drivers/misc/mic/bus/Makefile
@@ -4,3 +4,4 @@
 #
 obj-$(CONFIG_INTEL_MIC_BUS) += mic_bus.o
 obj-$(CONFIG_SCIF_BUS) += scif_bus.o
+obj-$(CONFIG_MIC_COSM) += cosm_bus.o
diff --git a/drivers/misc/mic/bus/cosm_bus.c b/drivers/misc/mic/bus/cosm_bus.c
new file mode 100644
index 000..1e36830
--- /dev/null
+++ b/drivers/misc/mic/bus/cosm_bus.c
@@ -0,0 +1,141 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Bus Driver
+ */
+#include 
+#include 
+#include 
+#include "cosm_bus.h"
+
+/* Unique numbering for cosm devices. */
+static DEFINE_IDA(cosm_index_ida);
+
+static int cosm_dev_probe(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+   return drv->probe(dev);
+}
+
+static int cosm_dev_remove(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+   drv->remove(dev);
+   return 0;
+}
+
+static struct bus_type cosm_bus = {
+

[PATCH char-misc-next v2 07/22] misc: mic: Add support for kernel mode SCIF clients

[Ashutosh Dixit]

Add support for registration/de-registration of kernel mode SCIF
clients. SCIF clients are probed with new and existing SCIF peer
devices. Similarly the client remove method is called when SCIF
peer devices are removed.

Changes to SCIF peer device framework necessitated by supporting
kernel mode SCIF clients are also included in this patch.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/scif/scif_api.c  |  43 +
 drivers/misc/mic/scif/scif_main.c |  88 --
 drivers/misc/mic/scif/scif_main.h |   5 +-
 drivers/misc/mic/scif/scif_nm.c   |  10 +--
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +-
 drivers/misc/mic/scif/scif_peer_bus.c | 165 +-
 drivers/misc/mic/scif/scif_peer_bus.h |  42 +
 include/linux/scif.h  |  58 
 8 files changed, 255 insertions(+), 221 deletions(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index bf2d70f..b47d56d 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -1430,3 +1430,46 @@ int scif_get_node_ids(u16 *nodes, int len, u16 *self)
return online;
 }
 EXPORT_SYMBOL_GPL(scif_get_node_ids);
+
+static int scif_add_client_dev(struct device *dev, struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->probe)
+   client->probe(spdev);
+   return 0;
+}
+
+static void scif_remove_client_dev(struct device *dev,
+  struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->remove)
+   client->remove(spdev);
+}
+
+void scif_client_unregister(struct scif_client *client)
+{
+   subsys_interface_unregister(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_unregister);
+
+int scif_client_register(struct scif_client *client)
+{
+   struct subsys_interface *si = >si;
+
+   si->name = client->name;
+   si->subsys = _peer_bus;
+   si->add_dev = scif_add_client_dev;
+   si->remove_dev = scif_remove_client_dev;
+
+   return subsys_interface_register(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_register);
diff --git a/drivers/misc/mic/scif/scif_main.c 
b/drivers/misc/mic/scif/scif_main.c
index 6ce851f..f90bd06 100644
--- a/drivers/misc/mic/scif/scif_main.c
+++ b/drivers/misc/mic/scif/scif_main.c
@@ -80,35 +80,6 @@ irqreturn_t scif_intr_handler(int irq, void *data)
return IRQ_HANDLED;
 }
 
-static int scif_peer_probe(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   mutex_lock(_info.conflock);
-   scif_info.total++;
-   scif_info.maxid = max_t(u32, spdev->dnode, scif_info.maxid);
-   mutex_unlock(_info.conflock);
-   rcu_assign_pointer(scifdev->spdev, spdev);
-
-   /* In the future SCIF kernel client devices will be added here */
-   return 0;
-}
-
-static void scif_peer_remove(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   /* In the future SCIF kernel client devices will be removed here */
-   spdev = rcu_dereference(scifdev->spdev);
-   if (spdev)
-   RCU_INIT_POINTER(scifdev->spdev, NULL);
-   synchronize_rcu();
-
-   mutex_lock(_info.conflock);
-   scif_info.total--;
-   mutex_unlock(_info.conflock);
-}
-
 static void scif_qp_setup_handler(struct work_struct *work)
 {
struct scif_dev *scifdev = container_of(work, struct scif_dev,
@@ -139,20 +110,13 @@ static void scif_qp_setup_handler(struct work_struct 
*work)
}
 }
 
-static int scif_setup_scifdev(struct scif_hw_dev *sdev)
+static int scif_setup_scifdev(void)
 {
+   /* We support a maximum of 129 SCIF nodes including the mgmt node */
+#define MAX_SCIF_NODES 129
int i;
-   u8 num_nodes;
+   u8 num_nodes = MAX_SCIF_NODES;
 
-   if (sdev->snode) {
-   struct mic_bootparam __iomem *bp = sdev->rdp;
-
-   num_nodes = ioread8(>tot_nodes);
-   } else {
-   struct mic_bootparam *bp = sdev->dp;
-
-   num_nodes = bp->tot_nodes;
-   }
scif_dev = kcalloc(num_nodes, sizeof(*scif_dev), GFP_KERNEL);
if (!scif_dev)
return -ENOMEM;
@@ -163,7 +127,7 @@ static int scif_setup_scifdev(struct scif_hw_dev *sdev)
scifdev->exit = OP_IDLE;
init_waitqueue_head(>disconn_wq);
mutex_init(>lock);
-   INIT_WORK(>init_msg_work, scif

[PATCH char-misc-next v2 05/22] dma: Add support to program MIC x100 status descriptiors

[Ashutosh Dixit]

From: Siva Yerramreddy 

The MIC X100 DMA engine has a special status descriptor which writes
an 8 byte value to a destination location.  This is used to signal
completion of all DMA descriptors prior to the status descriptor.
This patch add a new DMA engine API which enables updating a
destination address with an 8 byte immediate data value.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Lawrynowicz, Jacek 
Signed-off-by: Sudeep Dutt 
Signed-off-by: Siva Yerramreddy 
---
 drivers/dma/mic_x100_dma.c | 39 ++-
 include/linux/dmaengine.h  |  4 
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
index 74d9db0..068e920 100644
--- a/drivers/dma/mic_x100_dma.c
+++ b/drivers/dma/mic_x100_dma.c
@@ -193,8 +193,16 @@ static void mic_dma_prog_intr(struct mic_dma_chan *ch)
 static int mic_dma_do_dma(struct mic_dma_chan *ch, int flags, dma_addr_t src,
  dma_addr_t dst, size_t len)
 {
-   if (-ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len))
+   if (len && -ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len)) {
return -ENOMEM;
+   } else {
+   /* 3 is the maximum number of status descriptors */
+   int ret = mic_dma_avail_desc_ring_space(ch, 3);
+
+   if (ret < 0)
+   return ret;
+   }
+
/* Above mic_dma_prog_memcpy_desc() makes sure we have enough space */
if (flags & DMA_PREP_FENCE) {
mic_dma_prep_status_desc(>desc_ring[ch->head], 0,
@@ -270,6 +278,33 @@ allocate_tx(struct mic_dma_chan *ch)
return tx;
 }
 
+/* Program a status descriptor with dst as address and value to be written */
+static struct dma_async_tx_descriptor *
+mic_dma_prep_status_lock(struct dma_chan *ch, dma_addr_t dst, u64 src_val,
+unsigned long flags)
+{
+   struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
+   int result;
+
+   spin_lock(_ch->prep_lock);
+   result = mic_dma_avail_desc_ring_space(mic_ch, 4);
+   if (result < 0)
+   goto error;
+   mic_dma_prep_status_desc(_ch->desc_ring[mic_ch->head], src_val, dst,
+false);
+   mic_dma_hw_ring_inc_head(mic_ch);
+   result = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
+   if (result < 0)
+   goto error;
+
+   return allocate_tx(mic_ch);
+error:
+   dev_err(mic_dma_ch_to_device(mic_ch),
+   "Error enqueueing dma status descriptor, error=%d\n", result);
+   spin_unlock(_ch->prep_lock);
+   return NULL;
+}
+
 /*
  * Prepare a memcpy descriptor to be added to the ring.
  * Note that the temporary descriptor adds an extra overhead of copying the
@@ -587,6 +622,8 @@ static int mic_dma_register_dma_device(struct 
mic_dma_device *mic_dma_dev,
mic_dma_free_chan_resources;
mic_dma_dev->dma_dev.device_tx_status = mic_dma_tx_status;
mic_dma_dev->dma_dev.device_prep_dma_memcpy = mic_dma_prep_memcpy_lock;
+   mic_dma_dev->dma_dev.device_prep_dma_imm_data =
+   mic_dma_prep_status_lock;
mic_dma_dev->dma_dev.device_prep_dma_interrupt =
mic_dma_prep_interrupt_lock;
mic_dma_dev->dma_dev.device_issue_pending = mic_dma_issue_pending;
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 7ea9184..c47c68e 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -645,6 +645,7 @@ enum dmaengine_alignment {
  * The function takes a buffer of size buf_len. The callback function will
  * be called after period_len bytes have been transferred.
  * @device_prep_interleaved_dma: Transfer expression in a generic way.
+ * @device_prep_dma_imm_data: DMA's 8 byte immediate data to the dst address
  * @device_config: Pushes a new configuration to a channel, return 0 or an 
error
  * code
  * @device_pause: Pauses any transfer happening on a channel. Returns
@@ -727,6 +728,9 @@ struct dma_device {
struct dma_async_tx_descriptor *(*device_prep_interleaved_dma)(
struct dma_chan *chan, struct dma_interleaved_template *xt,
unsigned long flags);
+   struct dma_async_tx_descriptor *(*device_prep_dma_imm_data)(
+   struct dma_chan *chan, dma_addr_t dst, u64 data,
+   unsigned long flags);
 
int (*device_config)(struct dma_chan *chan,
 struct dma_slave_config *config);
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 03/22] iommu: Make the iova library a module

[Ashutosh Dixit]

From: Sakari Ailus 

The iova library has use outside the intel-iommu driver, thus make it a
module.

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/Kconfig | 2 +-
 drivers/iommu/iova.c  | 4 
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 4664c2a..d9da766 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -43,7 +43,7 @@ config IOMMU_IO_PGTABLE_LPAE_SELFTEST
 endmenu
 
 config IOMMU_IOVA
-   bool
+   tristate
 
 config OF_IOMMU
def_bool y
diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index dd87123..dd64d33 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -18,6 +18,7 @@
  */
 
 #include 
+#include 
 #include 
 
 void
@@ -559,3 +560,6 @@ error:
free_iova_mem(prev);
return NULL;
 }
+
+MODULE_AUTHOR("Anil S Keshavamurthy ");
+MODULE_LICENSE("GPL");
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 02/22] iommu: iova: Export symbols

[Ashutosh Dixit]

From: Sakari Ailus 

Use EXPORT_SYMBOL_GPL() to export the iova library symbols. The symbols
include:

init_iova_domain();
iova_cache_get();
iova_cache_put();
iova_cache_init();
alloc_iova();
find_iova();
__free_iova();
free_iova();
put_iova_domain();
reserve_iova();
copy_reserved_iova();

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/iova.c | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index 400f4d1..dd87123 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -38,6 +38,7 @@ init_iova_domain(struct iova_domain *iovad, unsigned long 
granule,
iovad->start_pfn = start_pfn;
iovad->dma_32bit_pfn = pfn_32bit;
 }
+EXPORT_SYMBOL_GPL(init_iova_domain);
 
 static struct rb_node *
 __get_cached_rbnode(struct iova_domain *iovad, unsigned long *limit_pfn)
@@ -243,6 +244,7 @@ int iova_cache_get(void)
 
return 0;
 }
+EXPORT_SYMBOL_GPL(iova_cache_get);
 
 void iova_cache_put(void)
 {
@@ -256,6 +258,7 @@ void iova_cache_put(void)
kmem_cache_destroy(iova_cache);
mutex_unlock(_cache_mutex);
 }
+EXPORT_SYMBOL_GPL(iova_cache_put);
 
 /**
  * alloc_iova - allocates an iova
@@ -296,6 +299,7 @@ alloc_iova(struct iova_domain *iovad, unsigned long size,
 
return new_iova;
 }
+EXPORT_SYMBOL_GPL(alloc_iova);
 
 /**
  * find_iova - find's an iova for a given pfn
@@ -336,6 +340,7 @@ struct iova *find_iova(struct iova_domain *iovad, unsigned 
long pfn)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return NULL;
 }
+EXPORT_SYMBOL_GPL(find_iova);
 
 /**
  * __free_iova - frees the given iova
@@ -354,6 +359,7 @@ __free_iova(struct iova_domain *iovad, struct iova *iova)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
free_iova_mem(iova);
 }
+EXPORT_SYMBOL_GPL(__free_iova);
 
 /**
  * free_iova - finds and frees the iova for a given pfn
@@ -371,6 +377,7 @@ free_iova(struct iova_domain *iovad, unsigned long pfn)
__free_iova(iovad, iova);
 
 }
+EXPORT_SYMBOL_GPL(free_iova);
 
 /**
  * put_iova_domain - destroys the iova doamin
@@ -393,6 +400,7 @@ void put_iova_domain(struct iova_domain *iovad)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(put_iova_domain);
 
 static int
 __is_range_overlap(struct rb_node *node,
@@ -482,6 +490,7 @@ finish:
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return iova;
 }
+EXPORT_SYMBOL_GPL(reserve_iova);
 
 /**
  * copy_reserved_iova - copies the reserved between domains
@@ -508,6 +517,7 @@ copy_reserved_iova(struct iova_domain *from, struct 
iova_domain *to)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(copy_reserved_iova);
 
 struct iova *
 split_and_remove_iova(struct iova_domain *iovad, struct iova *iova,
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 01/22] iommu: iova: Move iova cache management to the iova library

[Ashutosh Dixit]

From: Sakari Ailus 

This is necessary to separate intel-iommu from the iova library.

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/intel-iommu.c |  6 ++--
 drivers/iommu/iova.c| 83 ++---
 include/linux/iova.h|  4 +--
 3 files changed, 54 insertions(+), 39 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 2d7349a..7b27bcd 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3711,7 +3711,7 @@ static inline int iommu_devinfo_cache_init(void)
 static int __init iommu_init_mempool(void)
 {
int ret;
-   ret = iommu_iova_cache_init();
+   ret = iova_cache_get();
if (ret)
return ret;
 
@@ -3725,7 +3725,7 @@ static int __init iommu_init_mempool(void)
 
kmem_cache_destroy(iommu_domain_cache);
 domain_error:
-   iommu_iova_cache_destroy();
+   iova_cache_put();
 
return -ENOMEM;
 }
@@ -3734,7 +3734,7 @@ static void __init iommu_exit_mempool(void)
 {
kmem_cache_destroy(iommu_devinfo_cache);
kmem_cache_destroy(iommu_domain_cache);
-   iommu_iova_cache_destroy();
+   iova_cache_put();
 }
 
 static void quirk_ioat_snb_local_iommu(struct pci_dev *pdev)
diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index b7c3d92..400f4d1 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -20,40 +20,6 @@
 #include 
 #include 
 
-static struct kmem_cache *iommu_iova_cache;
-
-int iommu_iova_cache_init(void)
-{
-   int ret = 0;
-
-   iommu_iova_cache = kmem_cache_create("iommu_iova",
-sizeof(struct iova),
-0,
-SLAB_HWCACHE_ALIGN,
-NULL);
-   if (!iommu_iova_cache) {
-   pr_err("Couldn't create iova cache\n");
-   ret = -ENOMEM;
-   }
-
-   return ret;
-}
-
-void iommu_iova_cache_destroy(void)
-{
-   kmem_cache_destroy(iommu_iova_cache);
-}
-
-struct iova *alloc_iova_mem(void)
-{
-   return kmem_cache_alloc(iommu_iova_cache, GFP_ATOMIC);
-}
-
-void free_iova_mem(struct iova *iova)
-{
-   kmem_cache_free(iommu_iova_cache, iova);
-}
-
 void
 init_iova_domain(struct iova_domain *iovad, unsigned long granule,
unsigned long start_pfn, unsigned long pfn_32bit)
@@ -242,6 +208,55 @@ iova_insert_rbtree(struct rb_root *root, struct iova *iova)
rb_insert_color(>node, root);
 }
 
+static struct kmem_cache *iova_cache;
+static unsigned int iova_cache_users;
+static DEFINE_MUTEX(iova_cache_mutex);
+
+struct iova *alloc_iova_mem(void)
+{
+   return kmem_cache_alloc(iova_cache, GFP_ATOMIC);
+}
+EXPORT_SYMBOL(alloc_iova_mem);
+
+void free_iova_mem(struct iova *iova)
+{
+   kmem_cache_free(iova_cache, iova);
+}
+EXPORT_SYMBOL(free_iova_mem);
+
+int iova_cache_get(void)
+{
+   mutex_lock(_cache_mutex);
+   if (!iova_cache_users) {
+   iova_cache = kmem_cache_create(
+   "iommu_iova", sizeof(struct iova), 0,
+   SLAB_HWCACHE_ALIGN, NULL);
+   if (!iova_cache) {
+   mutex_unlock(_cache_mutex);
+   printk(KERN_ERR "Couldn't create iova cache\n");
+   return -ENOMEM;
+   }
+   }
+
+   iova_cache_users++;
+   mutex_unlock(_cache_mutex);
+
+   return 0;
+}
+
+void iova_cache_put(void)
+{
+   mutex_lock(_cache_mutex);
+   if (WARN_ON(!iova_cache_users)) {
+   mutex_unlock(_cache_mutex);
+   return;
+   }
+   iova_cache_users--;
+   if (!iova_cache_users)
+   kmem_cache_destroy(iova_cache);
+   mutex_unlock(_cache_mutex);
+}
+
 /**
  * alloc_iova - allocates an iova
  * @iovad: - iova domain in question
diff --git a/include/linux/iova.h b/include/linux/iova.h
index 3920a19..92f7177 100644
--- a/include/linux/iova.h
+++ b/include/linux/iova.h
@@ -68,8 +68,8 @@ static inline unsigned long iova_pfn(struct iova_domain 
*iovad, dma_addr_t iova)
return iova >> iova_shift(iovad);
 }
 
-int iommu_iova_cache_init(void);
-void iommu_iova_cache_destroy(void);
+int iova_cache_get(void);
+void iova_cache_put(void);
 
 struct iova *alloc_iova_mem(void);
 void free_iova_mem(struct iova *iova);
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 00/22] misc: mic: Enable COSM and remaining SCIF functionality

[Ashutosh Dixit]

Changelog:
=
v1 => v2:
* Use existing patches for IOMMU IOVA separation by Sakari Alius.
  These patches are being sent via char-misc since this patch
  series has a dependency on them. The IOMMU patches can be
  dropped once they are merged in via the iommu tree
* Add new dmaengine API instead of overloading the memcpy API
* Bug fixes and checkpatch cleanups since last post

v1: Initial post @ https://lkml.org/lkml/2015/7/27/958

The Symmetric Communication Interface (SCIF) API was accepted upstream
during the v4.2 merge window @ https://lkml.org/lkml/2015/3/30/865.
This patch series completes the implementation of the SCIF API and
also introduces the MIC Coprocessor State Management (COSM) drivers.

The first three patches break the IOMMU IOVA out of the IOMMU and make
it into a separate module. These patches were submitted in the IOMMU
tree but are being resent via char-misc since they are not available
upstream and the SCIF patches depend on them. The first three patches
can be dropped if they have been merged via the IOMMU tree.

Patch 4 solves a build issue with IOMMU IOVA and patch 5 extends the
DMA engine API to support MIC X100 status descriptor writes.

Patch 6 introduces SCIF poll which allows user and kernel mode clients
to block for events on a SCIF endpoint. Patch 7 introduces support for
SCIF kernel mode clients. SCIF clients are probed when new SCIF nodes
come online. Similarly the client remove method is called as SCIF
nodes go offline. The SCIF client framework makes use of the SCIF peer
bus infrastructure submitted previously.

Patches 8 through 14 are a re-factoring of functionality for boot,
shutdown and reset of MIC cards. This functionality, called COSM, was
previously present in the MIC host driver but has now been re-factored
into a separate driver so that it can be shared across multiple
generations of MIC products. The MIC sysfs interface documented
earlier has been updated. Further, the COSM host driver communicates
with a COSM client on the card using SCIF. COSM is therefore the first
kernel mode SCIF client and demonstrates the use of a subset of the
SCIF API.

Patch 15 onward completes the SCIF implementation by enabling the
following SCIF Remote Memory Access (RMA) functionality:
a. Memory registration to pin and unpin pages
b. Remote Memory mapping for low latency CPU accesses
c. Remote DMA (RDMA) for high bandwidth DMA transfers
d. Fence mechanisms to synchronize RDMAs

Documentation/mic/scif_overview.txt contains more information about
SCIF. This patch series is divided into 22 patches as follows:

1) iommu: iova: Move iova cache management to the iova library

2) iommu: iova: Export symbols

3) iommu: Make the iova library a module

4) Allow iova to be used without requiring IOMMU_SUPPORT

5) Add support to program MIC x100 status descriptiors

6) SCIF poll

7) Support for kernel mode SCIF clients

8) MIC COSM bus

9) Coprocessor State Management (COSM) driver

10) COSM SCIF server

11) COSM client driver

12) Remove COSM functionality from the MIC host driver

13) Remove COSM functionality from the MIC card driver

14) Update MIC host daemon with COSM changes

15) SCIF RMA header file and IOCTL changes

16) SCIF RMA header file

17) SCIF memory registration and unregistration

18) SCIF RMA list operations

19) SCIF remote memory map/unmap interface

20) SCIF DMA and CPU copy interface

21) SCIF fence

22) SCIF RMA nodeqp and minor miscellaneous changes

These patches have also been scanned by Fengguang Wu's 0-day
infrastructure and no issues have been reported.

Ashutosh Dixit (9):
  misc: mic: SCIF poll
  misc: mic: Add support for kernel mode SCIF clients
  misc: mic: MIC COSM bus
  misc: mic: Coprocessor State Management (COSM) driver
  misc: mic: COSM SCIF server
  misc: mic: COSM client driver
  misc: mic: Remove COSM functionality from the MIC host driver
  misc: mic: Remove COSM functionality from the MIC card driver
  misc: mic: Update MIC host daemon with COSM changes

Sakari Ailus (3):
  iommu: iova: Move iova cache management to the iova library
  iommu: iova: Export symbols
  iommu: Make the iova library a module

Siva Yerramreddy (1):
  dma: Add support to program MIC x100 status descriptiors

Sudeep Dutt (9):
  iommu: Allow iova to be used without requiring IOMMU_SUPPORT
  misc: mic: SCIF RMA header file and IOCTL changes
  misc: mic: SCIF RMA header file
  misc: mic: SCIF memory registration and unregistration
  misc: mic: SCIF RMA list operations
  misc: mic: SCIF remote memory map/unmap interface
  misc: mic: SCIF DMA and CPU copy interface
  misc: mic: SCIF fence
  misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

 Documentation/ABI/testing/sysfs-class-mic.txt   |   29 +-
 Documentation/mic/mic_overview.txt  |   31 +-
 Documentation/mic/mpssd/mpss|4 +-
 Documentation/mic/mpssd/mpssd.c |  362 +++--
 Documentation/mic/mpssd/mpssd.h |1 +
 drivers/dma/mic_x100_dm

[PATCH char-misc-next v2 05/22] dma: Add support to program MIC x100 status descriptiors

[Ashutosh Dixit]

From: Siva Yerramreddy <yshivakris...@gmail.com>

The MIC X100 DMA engine has a special status descriptor which writes
an 8 byte value to a destination location.  This is used to signal
completion of all DMA descriptors prior to the status descriptor.
This patch add a new DMA engine API which enables updating a
destination address with an 8 byte immediate data value.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Signed-off-by: Lawrynowicz, Jacek <jacek.lawrynow...@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Siva Yerramreddy <yshivakris...@gmail.com>
---
 drivers/dma/mic_x100_dma.c | 39 ++-
 include/linux/dmaengine.h  |  4 
 2 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
index 74d9db0..068e920 100644
--- a/drivers/dma/mic_x100_dma.c
+++ b/drivers/dma/mic_x100_dma.c
@@ -193,8 +193,16 @@ static void mic_dma_prog_intr(struct mic_dma_chan *ch)
 static int mic_dma_do_dma(struct mic_dma_chan *ch, int flags, dma_addr_t src,
  dma_addr_t dst, size_t len)
 {
-   if (-ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len))
+   if (len && -ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len)) {
return -ENOMEM;
+   } else {
+   /* 3 is the maximum number of status descriptors */
+   int ret = mic_dma_avail_desc_ring_space(ch, 3);
+
+   if (ret < 0)
+   return ret;
+   }
+
/* Above mic_dma_prog_memcpy_desc() makes sure we have enough space */
if (flags & DMA_PREP_FENCE) {
mic_dma_prep_status_desc(>desc_ring[ch->head], 0,
@@ -270,6 +278,33 @@ allocate_tx(struct mic_dma_chan *ch)
return tx;
 }
 
+/* Program a status descriptor with dst as address and value to be written */
+static struct dma_async_tx_descriptor *
+mic_dma_prep_status_lock(struct dma_chan *ch, dma_addr_t dst, u64 src_val,
+unsigned long flags)
+{
+   struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
+   int result;
+
+   spin_lock(_ch->prep_lock);
+   result = mic_dma_avail_desc_ring_space(mic_ch, 4);
+   if (result < 0)
+   goto error;
+   mic_dma_prep_status_desc(_ch->desc_ring[mic_ch->head], src_val, dst,
+false);
+   mic_dma_hw_ring_inc_head(mic_ch);
+   result = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
+   if (result < 0)
+   goto error;
+
+   return allocate_tx(mic_ch);
+error:
+   dev_err(mic_dma_ch_to_device(mic_ch),
+   "Error enqueueing dma status descriptor, error=%d\n", result);
+   spin_unlock(_ch->prep_lock);
+   return NULL;
+}
+
 /*
  * Prepare a memcpy descriptor to be added to the ring.
  * Note that the temporary descriptor adds an extra overhead of copying the
@@ -587,6 +622,8 @@ static int mic_dma_register_dma_device(struct 
mic_dma_device *mic_dma_dev,
mic_dma_free_chan_resources;
mic_dma_dev->dma_dev.device_tx_status = mic_dma_tx_status;
mic_dma_dev->dma_dev.device_prep_dma_memcpy = mic_dma_prep_memcpy_lock;
+   mic_dma_dev->dma_dev.device_prep_dma_imm_data =
+   mic_dma_prep_status_lock;
mic_dma_dev->dma_dev.device_prep_dma_interrupt =
mic_dma_prep_interrupt_lock;
mic_dma_dev->dma_dev.device_issue_pending = mic_dma_issue_pending;
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 7ea9184..c47c68e 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -645,6 +645,7 @@ enum dmaengine_alignment {
  * The function takes a buffer of size buf_len. The callback function will
  * be called after period_len bytes have been transferred.
  * @device_prep_interleaved_dma: Transfer expression in a generic way.
+ * @device_prep_dma_imm_data: DMA's 8 byte immediate data to the dst address
  * @device_config: Pushes a new configuration to a channel, return 0 or an 
error
  * code
  * @device_pause: Pauses any transfer happening on a channel. Returns
@@ -727,6 +728,9 @@ struct dma_device {
struct dma_async_tx_descriptor *(*device_prep_interleaved_dma)(
struct dma_chan *chan, struct dma_interleaved_template *xt,
unsigned long flags);
+   struct dma_async_tx_descriptor *(*device_prep_dma_imm_data)(
+   struct dma_chan *chan, dma_addr_t dst, u64 data,
+   unsigned long flags);
 
int (*device_config)(struct dma_chan *chan,
 struct dma_slave_config *config);
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 07/22] misc: mic: Add support for kernel mode SCIF clients

[Ashutosh Dixit]

Add support for registration/de-registration of kernel mode SCIF
clients. SCIF clients are probed with new and existing SCIF peer
devices. Similarly the client remove method is called when SCIF
peer devices are removed.

Changes to SCIF peer device framework necessitated by supporting
kernel mode SCIF clients are also included in this patch.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/scif/scif_api.c  |  43 +
 drivers/misc/mic/scif/scif_main.c |  88 --
 drivers/misc/mic/scif/scif_main.h |   5 +-
 drivers/misc/mic/scif/scif_nm.c   |  10 +--
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +-
 drivers/misc/mic/scif/scif_peer_bus.c | 165 +-
 drivers/misc/mic/scif/scif_peer_bus.h |  42 +
 include/linux/scif.h  |  58 
 8 files changed, 255 insertions(+), 221 deletions(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index bf2d70f..b47d56d 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -1430,3 +1430,46 @@ int scif_get_node_ids(u16 *nodes, int len, u16 *self)
return online;
 }
 EXPORT_SYMBOL_GPL(scif_get_node_ids);
+
+static int scif_add_client_dev(struct device *dev, struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->probe)
+   client->probe(spdev);
+   return 0;
+}
+
+static void scif_remove_client_dev(struct device *dev,
+  struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->remove)
+   client->remove(spdev);
+}
+
+void scif_client_unregister(struct scif_client *client)
+{
+   subsys_interface_unregister(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_unregister);
+
+int scif_client_register(struct scif_client *client)
+{
+   struct subsys_interface *si = >si;
+
+   si->name = client->name;
+   si->subsys = _peer_bus;
+   si->add_dev = scif_add_client_dev;
+   si->remove_dev = scif_remove_client_dev;
+
+   return subsys_interface_register(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_register);
diff --git a/drivers/misc/mic/scif/scif_main.c 
b/drivers/misc/mic/scif/scif_main.c
index 6ce851f..f90bd06 100644
--- a/drivers/misc/mic/scif/scif_main.c
+++ b/drivers/misc/mic/scif/scif_main.c
@@ -80,35 +80,6 @@ irqreturn_t scif_intr_handler(int irq, void *data)
return IRQ_HANDLED;
 }
 
-static int scif_peer_probe(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   mutex_lock(_info.conflock);
-   scif_info.total++;
-   scif_info.maxid = max_t(u32, spdev->dnode, scif_info.maxid);
-   mutex_unlock(_info.conflock);
-   rcu_assign_pointer(scifdev->spdev, spdev);
-
-   /* In the future SCIF kernel client devices will be added here */
-   return 0;
-}
-
-static void scif_peer_remove(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   /* In the future SCIF kernel client devices will be removed here */
-   spdev = rcu_dereference(scifdev->spdev);
-   if (spdev)
-   RCU_INIT_POINTER(scifdev->spdev, NULL);
-   synchronize_rcu();
-
-   mutex_lock(_info.conflock);
-   scif_info.total--;
-   mutex_unlock(_info.conflock);
-}
-
 static void scif_qp_setup_handler(struct work_struct *work)
 {
struct scif_dev *scifdev = container_of(work, struct scif_dev,
@@ -139,20 +110,13 @@ static void scif_qp_setup_handler(struct work_struct 
*work)
}
 }
 
-static int scif_setup_scifdev(struct scif_hw_dev *sdev)
+static int scif_setup_scifdev(void)
 {
+   /* We support a maximum of 129 SCIF nodes including the mgmt node */
+#define MAX_SCIF_NODES 129
int i;
-   u8 num_nodes;
+   u8 num_nodes = MAX_SCIF_NODES;
 
-   if (sdev->snode) {
-   struct mic_bootparam __iomem *bp = sdev->rdp;
-
-   num_nodes = ioread8(>tot_nodes);
-   } else {
-   struct mic_bootparam *bp = sdev->dp;
-
-   num_nodes = bp->tot_nodes;
-   }
scif_dev = kcalloc(num_nodes, sizeof(*scif_dev), GFP_KERNEL);
if (!scif_dev)
return -ENOMEM;
@@ -163,7 +127,7 @@ static int scif_setup_scifdev(struct scif_hw_dev *sdev)
scifdev->exit = OP_IDLE;
init_waitqueue_head(>disconn_wq);

[PATCH char-misc-next v2 09/22] misc: mic: Coprocessor State Management (COSM) driver

[Ashutosh Dixit]

The COSM driver allows boot, shutdown and reset of Intel MIC devices
via sysfs. This functionality was previously present in the Intel MIC
host driver but has now been taken out into a separate driver so that
it can be shared between multiple generations of Intel MIC products.
The sysfs kernel ABI used by the COSM driver is the same as that
defined originally for the MIC host driver in
Documentation/ABI/testing/sysfs-class-mic.txt.

The COSM driver also contains support for dumping the MIC card log_buf
and doing a "force reset" for the card via debugfs. The OSPM support
present in the MIC host driver has now largely been moved to user
space and only a small required OSPM functionality is now present in
the driver.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Dasaratharaman Chandramouli 
<dasaratharaman.chandramo...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/cosm/Makefile   |  10 +
 drivers/misc/mic/cosm/cosm_debugfs.c | 155 
 drivers/misc/mic/cosm/cosm_main.c| 388 +
 drivers/misc/mic/cosm/cosm_main.h|  70 ++
 drivers/misc/mic/cosm/cosm_sysfs.c   | 461 +++
 5 files changed, 1084 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/Makefile
 create mode 100644 drivers/misc/mic/cosm/cosm_debugfs.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.h
 create mode 100644 drivers/misc/mic/cosm/cosm_sysfs.c

diff --git a/drivers/misc/mic/cosm/Makefile b/drivers/misc/mic/cosm/Makefile
new file mode 100644
index 000..b85d4d4
--- /dev/null
+++ b/drivers/misc/mic/cosm/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile - Intel MIC Coprocessor State Management (COSM) Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += mic_cosm.o
+
+mic_cosm-objs := cosm_main.o
+mic_cosm-objs += cosm_debugfs.o
+mic_cosm-objs += cosm_sysfs.o
+mic_cosm-objs += cosm_scif_server.o
diff --git a/drivers/misc/mic/cosm/cosm_debugfs.c 
b/drivers/misc/mic/cosm/cosm_debugfs.c
new file mode 100644
index 000..30d953d
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_debugfs.c
@@ -0,0 +1,155 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+
+#include 
+#include 
+#include "cosm_main.h"
+
+/* Debugfs parent dir */
+static struct dentry *cosm_dbg;
+
+/**
+ * cosm_log_buf_show - Display MIC kernel log buffer
+ *
+ * log_buf addr/len is read from System.map by user space
+ * and populated in sysfs entries.
+ */
+static int cosm_log_buf_show(struct seq_file *s, void *unused)
+{
+   void __iomem *log_buf_va;
+   int __iomem *log_buf_len_va;
+   struct cosm_device *cdev = s->private;
+   void *kva;
+   int size;
+   u64 aper_offset;
+
+   if (!cdev || !cdev->log_buf_addr || !cdev->log_buf_len)
+   goto done;
+
+   mutex_lock(>cosm_mutex);
+   switch (cdev->state) {
+   case MIC_BOOTING:
+   case MIC_ONLINE:
+   case MIC_SHUTTING_DOWN:
+   break;
+   default:
+   goto unlock;
+   }
+
+   /*
+* Card kernel will never be relocated and any kernel text/data mapping
+* can be translated to phys address by subtracting __START_KERNEL_map.
+*/
+   aper_offset = (u64)cdev->log_buf_len - __START_KERNEL_map;
+   log_buf_len_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+   aper_offset = (u64)cdev->log_buf_addr - __START_KERNEL_map;
+   log_buf_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+
+   size = ioread32(log_buf_len_va);
+   kva = kmalloc(size, GFP_KERNEL);
+   if (!kva)
+   goto unlock;
+
+   memcpy_fromio(kva, log_buf_va, size);
+   seq_write(s, kva, size);
+   kfree(kva);
+unlock:
+   mutex_unlock(>cosm_mutex);
+done:
+   return 0;
+}
+
+static int cosm_log_buf_open(struct inode *inode, struct file *file)
+{
+   return single_open(file, cosm_log_buf_show, inode->i_private);
+}
+
+static const struct file_operations log_buf_ops = {
+   .owner   = THIS_MODULE,
+   .open= cosm_log_buf_open,
+   .read= seq_read,
+

[PATCH char-misc-next v2 08/22] misc: mic: MIC COSM bus

[Ashutosh Dixit]

The MIC COSM bus allows the co-processor state management (COSM)
functionality to be shared between multiple generations of Intel MIC
products. The COSM driver registers itself on the COSM bus. The base
PCIe drivers implement the bus ops and register COSM devices on the
bus, resulting in the COSM driver being probed with the COSM devices.
COSM bus ops, e.g. start, stop, ready, reset, therefore abstract out
common functionality from its specific implementation for individual
generations of MIC products.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/Kconfig|  22 ++-
 drivers/misc/mic/bus/Makefile   |   1 +
 drivers/misc/mic/bus/cosm_bus.c | 141 
 drivers/misc/mic/bus/cosm_bus.h | 134 ++
 4 files changed, 296 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/mic/bus/cosm_bus.c
 create mode 100644 drivers/misc/mic/bus/cosm_bus.h

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index e9f2f56..1488232 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -36,7 +36,7 @@ comment "Intel MIC Host Driver"
 
 config INTEL_MIC_HOST
tristate "Intel MIC Host Driver"
-   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VHOST_RING
help
  This enables Host Driver support for the Intel Many Integrated
@@ -56,7 +56,7 @@ comment "Intel MIC Card Driver"
 
 config INTEL_MIC_CARD
tristate "Intel MIC Card Driver"
-   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VIRTIO
help
  This enables card driver support for the Intel Many Integrated
@@ -88,3 +88,21 @@ config SCIF
  More information about the Intel MIC family as well as the Linux
  OS and tools for MIC to use with this driver are available from
  <http://software.intel.com/en-us/mic-developer>.
+
+comment "Intel MIC Coprocessor State Management (COSM) Drivers"
+
+config MIC_COSM
+   tristate "Intel MIC Coprocessor State Management (COSM) Drivers"
+   depends on 64BIT && PCI && X86 && SCIF
+   help
+ This enables COSM driver support for the Intel Many
+ Integrated Core (MIC) family of PCIe form factor coprocessor
+ devices. COSM drivers implement functions such as boot,
+ shutdown, reset and reboot of MIC devices.
+
+ If you are building a host kernel with an Intel MIC device then
+ say M (recommended) or Y, else say N. If unsure say N.
+
+ More information about the Intel MIC family as well as the Linux
+ OS and tools for MIC to use with this driver are available from
+ <http://software.intel.com/en-us/mic-developer>.
diff --git a/drivers/misc/mic/bus/Makefile b/drivers/misc/mic/bus/Makefile
index 1ed37e2..761842b 100644
--- a/drivers/misc/mic/bus/Makefile
+++ b/drivers/misc/mic/bus/Makefile
@@ -4,3 +4,4 @@
 #
 obj-$(CONFIG_INTEL_MIC_BUS) += mic_bus.o
 obj-$(CONFIG_SCIF_BUS) += scif_bus.o
+obj-$(CONFIG_MIC_COSM) += cosm_bus.o
diff --git a/drivers/misc/mic/bus/cosm_bus.c b/drivers/misc/mic/bus/cosm_bus.c
new file mode 100644
index 000..1e36830
--- /dev/null
+++ b/drivers/misc/mic/bus/cosm_bus.c
@@ -0,0 +1,141 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Bus Driver
+ */
+#include 
+#include 
+#include 
+#include "cosm_bus.h"
+
+/* Unique numbering for cosm devices. */
+static DEFINE_IDA(cosm_index_ida);
+
+static int cosm_dev_probe(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+   return drv->probe(dev);
+}
+
+static int cosm_dev_remove(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+

[PATCH char-misc-next v2 06/22] misc: mic: SCIF poll

[Ashutosh Dixit]

SCIF poll allows both user and kernel mode clients to wait on
events on a SCIF endpoint. These events include availability of
space or data in the SCIF ring buffer, availability of connection
requests on a listening endpoint and completion of connections
when using async connects.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/scif/scif_api.c  | 158 +-
 drivers/misc/mic/scif/scif_epd.h  |  22 ++
 drivers/misc/mic/scif/scif_fd.c   |   9 +++
 drivers/misc/mic/scif/scif_main.h |   2 +
 include/linux/scif.h  |  74 ++
 5 files changed, 264 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index f39d313..bf2d70f 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -37,9 +37,21 @@ enum conn_async_state {
ASYNC_CONN_FLUSH_WORK   /* async work flush in progress  */
 };
 
+/*
+ * File operations for anonymous inode file associated with a SCIF endpoint,
+ * used in kernel mode SCIF poll. Kernel mode SCIF poll calls portions of the
+ * poll API in the kernel and these take in a struct file *. Since a struct
+ * file is not available to kernel mode SCIF, it uses an anonymous file for
+ * this purpose.
+ */
+const struct file_operations scif_anon_fops = {
+   .owner = THIS_MODULE,
+};
+
 scif_epd_t scif_open(void)
 {
struct scif_endpt *ep;
+   int err;
 
might_sleep();
ep = kzalloc(sizeof(*ep), GFP_KERNEL);
@@ -50,6 +62,10 @@ scif_epd_t scif_open(void)
if (!ep->qp_info.qp)
goto err_qp_alloc;
 
+   err = scif_anon_inode_getfile(ep);
+   if (err)
+   goto err_anon_inode;
+
spin_lock_init(>lock);
mutex_init(>sendlock);
mutex_init(>recvlock);
@@ -59,6 +75,8 @@ scif_epd_t scif_open(void)
"SCIFAPI open: ep %p success\n", ep);
return ep;
 
+err_anon_inode:
+   kfree(ep->qp_info.qp);
 err_qp_alloc:
kfree(ep);
 err_ep_alloc:
@@ -279,6 +297,7 @@ int scif_close(scif_epd_t epd)
}
}
scif_put_port(ep->port.port);
+   scif_anon_inode_fput(ep);
scif_teardown_ep(ep);
scif_add_epd_to_zombie_list(ep, !SCIF_EPLOCK_HELD);
return 0;
@@ -558,8 +577,10 @@ void scif_conn_handler(struct work_struct *work)
list_del(>conn_list);
}
spin_unlock(_info.nb_connect_lock);
-   if (ep)
+   if (ep) {
ep->conn_err = scif_conn_func(ep);
+   wake_up_interruptible(>conn_pend_wq);
+   }
} while (ep);
 }
 
@@ -660,6 +681,7 @@ int __scif_connect(scif_epd_t epd, struct scif_port_id 
*dst, bool non_block)
ep->remote_dev = _dev[dst->node];
ep->qp_info.qp->magic = SCIFEP_MAGIC;
if (ep->conn_async_state == ASYNC_CONN_INPROGRESS) {
+   init_waitqueue_head(>conn_pend_wq);
spin_lock(_info.nb_connect_lock);
list_add_tail(>conn_list, _info.nb_connect_list);
spin_unlock(_info.nb_connect_lock);
@@ -788,6 +810,10 @@ retry_connection:
goto scif_accept_error_qpalloc;
}
 
+   err = scif_anon_inode_getfile(cep);
+   if (err)
+   goto scif_accept_error_anon_inode;
+
cep->qp_info.qp->magic = SCIFEP_MAGIC;
spdev = scif_get_peer_dev(cep->remote_dev);
if (IS_ERR(spdev)) {
@@ -858,6 +884,8 @@ retry:
spin_unlock(>lock);
return 0;
 scif_accept_error_map:
+   scif_anon_inode_fput(cep);
+scif_accept_error_anon_inode:
scif_teardown_ep(cep);
 scif_accept_error_qpalloc:
kfree(cep);
@@ -1247,6 +1275,134 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags)
 }
 EXPORT_SYMBOL_GPL(scif_recv);
 
+static inline void _scif_poll_wait(struct file *f, wait_queue_head_t *wq,
+  poll_table *p, struct scif_endpt *ep)
+{
+   /*
+* Because poll_wait makes a GFP_KERNEL allocation, give up the lock
+* and regrab it afterwards. Because the endpoint state might have
+* changed while the lock was given up, the state must be checked
+* again after re-acquiring the lock. The code in __scif_pollfd(..)
+* does this.
+*/
+   spin_unlock(>lock);
+   poll_wait(f, wq, p);
+   spin_lock(>lock);
+}
+
+unsigned int
+__scif_pollfd(struct file *f, poll_table *wait, struct scif_endpt *ep)
+{
+   unsigned int mask = 0;
+
+   dev_dbg(scif_info.mdev.this_device,
+   "SCIFAPI pollfd: ep %p %s\n", ep, scif_ep_states[ep->state]);
+
+   spin_lock(>lock);
+
+   /* Endpoint 

[PATCH char-misc-next v2 10/22] misc: mic: COSM SCIF server

[Ashutosh Dixit]

The COSM driver communicates with the MIC cards over SCIF. A SCIF
"server" listens for incoming connections from "client" MIC cards as
they boot. After the connection is accepted a separate work item is
scheduled for each MIC card. This work item normally stays blocked in
scif_poll but wakes up to process messages from the card.

The SCIF connection between the host and card COSM components is used
to (a) send the command to shut down the card (b) receive shutdown
status back from the card upon completion of shutdown (c) receive
periodic heartbeat messages to detect card crashes (d) send host time
to the card to enable the card to sync its time to the host.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/cosm/cosm_scif_server.c | 405 +++
 1 file changed, 405 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/cosm_scif_server.c

diff --git a/drivers/misc/mic/cosm/cosm_scif_server.c 
b/drivers/misc/mic/cosm/cosm_scif_server.c
new file mode 100644
index 000..5696df4
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_scif_server.c
@@ -0,0 +1,405 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+#include 
+#include "cosm_main.h"
+
+/*
+ * The COSM driver uses SCIF to communicate between the management node and the
+ * MIC cards. SCIF is used to (a) Send a shutdown command to the card (b)
+ * receive a shutdown status back from the card upon completion of shutdown and
+ * (c) receive periodic heartbeat messages from the card used to deduce if the
+ * card has crashed.
+ *
+ * A COSM server consisting of a SCIF listening endpoint waits for incoming
+ * connections from the card. Upon acceptance of the connection, a separate
+ * work-item is scheduled to handle SCIF message processing for that card. The
+ * life-time of this work-item is therefore the time from which the connection
+ * from a card is accepted to the time at which the connection is closed. A new
+ * work-item starts each time the card boots and is alive till the card (a)
+ * shuts down (b) is reset (c) crashes (d) cosm_client driver on the card is
+ * unloaded.
+ *
+ * From the point of view of COSM interactions with SCIF during card
+ * shutdown, reset and crash are as follows:
+ *
+ * Card shutdown
+ * -
+ * 1. COSM client on the card invokes orderly_poweroff() in response to 
SHUTDOWN
+ *message from the host.
+ * 2. Card driver shutdown callback invokes scif_unregister_device(..) 
resulting
+ *in scif_remove(..) getting called on the card
+ * 3. scif_remove -> scif_stop -> scif_handle_remove_node ->
+ *scif_peer_unregister_device -> device_unregister for the host peer device
+ * 4. During device_unregister remove(..) method of cosm_client is invoked 
which
+ *closes the COSM SCIF endpoint on the card. This results in a SCIF_DISCNCT
+ *message being sent to host SCIF. SCIF_DISCNCT message processing on the
+ *host SCIF sets the host COSM SCIF endpoint state to DISCONNECTED and 
wakes
+ *up the host COSM thread blocked in scif_poll(..) resulting in
+ *scif_poll(..)  returning POLLHUP.
+ * 5. On the card, scif_peer_release_dev is next called which results in an
+ *SCIF_EXIT message being sent to the host and after receiving the
+ *SCIF_EXIT_ACK from the host the peer device teardown on the card is
+ *complete.
+ * 6. As part of the SCIF_EXIT message processing on the host, host sends a
+ *SCIF_REMOVE_NODE to itself corresponding to the card being removed. This
+ *starts a similar SCIF peer device teardown sequence on the host
+ *corresponding to the card being shut down.
+ *
+ * Card reset
+ * --
+ * The case of interest here is when the card has not been previously shut down
+ * since most of the steps below are skipped in that case:
+
+ * 1. cosm_stop(..) invokes hw_ops->stop(..) method of the base PCIe driver
+ *which unregisters the SCIF HW device resulting in scif_remove(..) being
+ *called on the host.
+ * 2. scif_remove(..) calls scif_disconnect_node(..) which results in a
+ *SCIF_EXIT message being sent to the card.
+ * 3. The card executes scif_stop()

[PATCH char-misc-next v2 04/22] iommu: Allow iova to be used without requiring IOMMU_SUPPORT

[Ashutosh Dixit]

From: Sudeep Dutt 

iova is a library which can be built without IOMMU_SUPPORT

Signed-off-by: Sudeep Dutt 
---
 drivers/iommu/Kconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index d9da766..71d1c46 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -12,6 +12,9 @@ menuconfig IOMMU_SUPPORT
  remap DMA requests and/or remap interrupts from other devices on the
  system.
 
+config IOMMU_IOVA
+   tristate
+
 if IOMMU_SUPPORT
 
 menu "Generic IOMMU Pagetable Support"
@@ -42,9 +45,6 @@ config IOMMU_IO_PGTABLE_LPAE_SELFTEST
 
 endmenu
 
-config IOMMU_IOVA
-   tristate
-
 config OF_IOMMU
def_bool y
depends on OF && IOMMU_API
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 22/22] misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch adds the SCIF kernel node QP control messages required to
enable SCIF RMAs. Examples of such node QP control messages include
registration, unregistration, remote memory allocation requests,
remote memory unmap and SCIF remote fence requests.

The patch also updates the SCIF driver with minor changes required to
enable SCIF RMAs by adding the new files to the build, initializing
RMA specific information during SCIF endpoint creation, reserving SCIF
DMA channels, initializing SCIF RMA specific global data structures,
adding the IOCTL hooks required for SCIF RMAs and updating RMA
specific debugfs hooks.

Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Reviewed-by: Nikhil Rao <nikhil@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/Kconfig  |   1 +
 drivers/misc/mic/scif/Makefile|   5 +
 drivers/misc/mic/scif/scif_api.c  |  33 +--
 drivers/misc/mic/scif/scif_debugfs.c  |  85 -
 drivers/misc/mic/scif/scif_epd.c  |  26 +++---
 drivers/misc/mic/scif/scif_epd.h  |  28 ++
 drivers/misc/mic/scif/scif_fd.c   | 169 +-
 drivers/misc/mic/scif/scif_main.c |  23 -
 drivers/misc/mic/scif/scif_main.h |  30 +-
 drivers/misc/mic/scif/scif_map.h  |  25 -
 drivers/misc/mic/scif/scif_nm.c   |  10 +-
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +++--
 drivers/misc/mic/scif/scif_nodeqp.h   |  42 -
 drivers/misc/mic/scif/scif_peer_bus.c |  14 +++
 14 files changed, 516 insertions(+), 40 deletions(-)

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index 1488232..60376fb 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -75,6 +75,7 @@ comment "SCIF Driver"
 config SCIF
tristate "SCIF Driver"
depends on 64BIT && PCI && X86 && SCIF_BUS
+   select IOMMU_IOVA
help
  This enables SCIF Driver support for the Intel Many Integrated
  Core (MIC) family of PCIe form factor coprocessor devices that
diff --git a/drivers/misc/mic/scif/Makefile b/drivers/misc/mic/scif/Makefile
index bf10bb7..29cfc3e 100644
--- a/drivers/misc/mic/scif/Makefile
+++ b/drivers/misc/mic/scif/Makefile
@@ -13,3 +13,8 @@ scif-objs += scif_epd.o
 scif-objs += scif_rb.o
 scif-objs += scif_nodeqp.o
 scif-objs += scif_nm.o
+scif-objs += scif_dma.o
+scif-objs += scif_fence.o
+scif-objs += scif_mmap.o
+scif-objs += scif_rma.o
+scif-objs += scif_rma_list.o
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index b47d56d..ddc9e4b 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -70,6 +70,7 @@ scif_epd_t scif_open(void)
mutex_init(>sendlock);
mutex_init(>recvlock);
 
+   scif_rma_ep_init(ep);
ep->state = SCIFEP_UNBOUND;
dev_dbg(scif_info.mdev.this_device,
"SCIFAPI open: ep %p success\n", ep);
@@ -184,8 +185,11 @@ int scif_close(scif_epd_t epd)
 
switch (oldstate) {
case SCIFEP_ZOMBIE:
+   dev_err(scif_info.mdev.this_device,
+   "SCIFAPI close: zombie state unexpected\n");
case SCIFEP_DISCONNECTED:
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
/* Remove from the disconnected list */
mutex_lock(_info.connlock);
list_for_each_safe(pos, tmpq, _info.disconnected) {
@@ -207,6 +211,7 @@ int scif_close(scif_epd_t epd)
case SCIFEP_CLOSING:
{
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
scif_disconnect_ep(ep);
break;
}
@@ -218,7 +223,7 @@ int scif_close(scif_epd_t epd)
struct scif_endpt *aep;
 
spin_unlock(>lock);
-   spin_lock(_info.eplock);
+   mutex_lock(_info.eplock);
 
/* remove from listen list */
list_for_each_safe(pos, tmpq, _info.listen) {
@@ -240,7 +245,7 @@ int scif_close(scif_epd_t epd)
break;
}
}
-   spin_unlock(_info.eplock);
+   mutex_unlock(_info.eplock);
mutex_lock(_info.connlock);
list_for_each_safe(pos, tmpq, _info.connected) {
tmpep = list_entry(pos,
@@ -260,13 +265,13 @@ int scif_close(scif_epd_t epd)
}
mutex_unlock(_info.connlock);
scif_teardown_ep(aep);
-   spin_lock(_info.eplock);
+   mutex_lock(_info.eplock);
scif_add_epd_to_zombie_list(aep, SCIF_

[PATCH char-misc-next v2 00/22] misc: mic: Enable COSM and remaining SCIF functionality

[Ashutosh Dixit]

Changelog:
=
v1 => v2:
* Use existing patches for IOMMU IOVA separation by Sakari Alius.
  These patches are being sent via char-misc since this patch
  series has a dependency on them. The IOMMU patches can be
  dropped once they are merged in via the iommu tree
* Add new dmaengine API instead of overloading the memcpy API
* Bug fixes and checkpatch cleanups since last post

v1: Initial post @ https://lkml.org/lkml/2015/7/27/958

The Symmetric Communication Interface (SCIF) API was accepted upstream
during the v4.2 merge window @ https://lkml.org/lkml/2015/3/30/865.
This patch series completes the implementation of the SCIF API and
also introduces the MIC Coprocessor State Management (COSM) drivers.

The first three patches break the IOMMU IOVA out of the IOMMU and make
it into a separate module. These patches were submitted in the IOMMU
tree but are being resent via char-misc since they are not available
upstream and the SCIF patches depend on them. The first three patches
can be dropped if they have been merged via the IOMMU tree.

Patch 4 solves a build issue with IOMMU IOVA and patch 5 extends the
DMA engine API to support MIC X100 status descriptor writes.

Patch 6 introduces SCIF poll which allows user and kernel mode clients
to block for events on a SCIF endpoint. Patch 7 introduces support for
SCIF kernel mode clients. SCIF clients are probed when new SCIF nodes
come online. Similarly the client remove method is called as SCIF
nodes go offline. The SCIF client framework makes use of the SCIF peer
bus infrastructure submitted previously.

Patches 8 through 14 are a re-factoring of functionality for boot,
shutdown and reset of MIC cards. This functionality, called COSM, was
previously present in the MIC host driver but has now been re-factored
into a separate driver so that it can be shared across multiple
generations of MIC products. The MIC sysfs interface documented
earlier has been updated. Further, the COSM host driver communicates
with a COSM client on the card using SCIF. COSM is therefore the first
kernel mode SCIF client and demonstrates the use of a subset of the
SCIF API.

Patch 15 onward completes the SCIF implementation by enabling the
following SCIF Remote Memory Access (RMA) functionality:
a. Memory registration to pin and unpin pages
b. Remote Memory mapping for low latency CPU accesses
c. Remote DMA (RDMA) for high bandwidth DMA transfers
d. Fence mechanisms to synchronize RDMAs

Documentation/mic/scif_overview.txt contains more information about
SCIF. This patch series is divided into 22 patches as follows:

1) iommu: iova: Move iova cache management to the iova library

2) iommu: iova: Export symbols

3) iommu: Make the iova library a module

4) Allow iova to be used without requiring IOMMU_SUPPORT

5) Add support to program MIC x100 status descriptiors

6) SCIF poll

7) Support for kernel mode SCIF clients

8) MIC COSM bus

9) Coprocessor State Management (COSM) driver

10) COSM SCIF server

11) COSM client driver

12) Remove COSM functionality from the MIC host driver

13) Remove COSM functionality from the MIC card driver

14) Update MIC host daemon with COSM changes

15) SCIF RMA header file and IOCTL changes

16) SCIF RMA header file

17) SCIF memory registration and unregistration

18) SCIF RMA list operations

19) SCIF remote memory map/unmap interface

20) SCIF DMA and CPU copy interface

21) SCIF fence

22) SCIF RMA nodeqp and minor miscellaneous changes

These patches have also been scanned by Fengguang Wu's 0-day
infrastructure and no issues have been reported.

Ashutosh Dixit (9):
  misc: mic: SCIF poll
  misc: mic: Add support for kernel mode SCIF clients
  misc: mic: MIC COSM bus
  misc: mic: Coprocessor State Management (COSM) driver
  misc: mic: COSM SCIF server
  misc: mic: COSM client driver
  misc: mic: Remove COSM functionality from the MIC host driver
  misc: mic: Remove COSM functionality from the MIC card driver
  misc: mic: Update MIC host daemon with COSM changes

Sakari Ailus (3):
  iommu: iova: Move iova cache management to the iova library
  iommu: iova: Export symbols
  iommu: Make the iova library a module

Siva Yerramreddy (1):
  dma: Add support to program MIC x100 status descriptiors

Sudeep Dutt (9):
  iommu: Allow iova to be used without requiring IOMMU_SUPPORT
  misc: mic: SCIF RMA header file and IOCTL changes
  misc: mic: SCIF RMA header file
  misc: mic: SCIF memory registration and unregistration
  misc: mic: SCIF RMA list operations
  misc: mic: SCIF remote memory map/unmap interface
  misc: mic: SCIF DMA and CPU copy interface
  misc: mic: SCIF fence
  misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

 Documentation/ABI/testing/sysfs-class-mic.txt   |   29 +-
 Documentation/mic/mic_overview.txt  |   31 +-
 Documentation/mic/mpssd/mpss|4 +-
 Documentation/mic/mpssd/mpssd.c |  362 +++--
 Documentation/mic/mpssd/mpssd.h |1 +
 drivers/dma/mic_x100_dm

[PATCH char-misc-next v2 01/22] iommu: iova: Move iova cache management to the iova library

[Ashutosh Dixit]

From: Sakari Ailus 

This is necessary to separate intel-iommu from the iova library.

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/intel-iommu.c |  6 ++--
 drivers/iommu/iova.c| 83 ++---
 include/linux/iova.h|  4 +--
 3 files changed, 54 insertions(+), 39 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 2d7349a..7b27bcd 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3711,7 +3711,7 @@ static inline int iommu_devinfo_cache_init(void)
 static int __init iommu_init_mempool(void)
 {
int ret;
-   ret = iommu_iova_cache_init();
+   ret = iova_cache_get();
if (ret)
return ret;
 
@@ -3725,7 +3725,7 @@ static int __init iommu_init_mempool(void)
 
kmem_cache_destroy(iommu_domain_cache);
 domain_error:
-   iommu_iova_cache_destroy();
+   iova_cache_put();
 
return -ENOMEM;
 }
@@ -3734,7 +3734,7 @@ static void __init iommu_exit_mempool(void)
 {
kmem_cache_destroy(iommu_devinfo_cache);
kmem_cache_destroy(iommu_domain_cache);
-   iommu_iova_cache_destroy();
+   iova_cache_put();
 }
 
 static void quirk_ioat_snb_local_iommu(struct pci_dev *pdev)
diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index b7c3d92..400f4d1 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -20,40 +20,6 @@
 #include 
 #include 
 
-static struct kmem_cache *iommu_iova_cache;
-
-int iommu_iova_cache_init(void)
-{
-   int ret = 0;
-
-   iommu_iova_cache = kmem_cache_create("iommu_iova",
-sizeof(struct iova),
-0,
-SLAB_HWCACHE_ALIGN,
-NULL);
-   if (!iommu_iova_cache) {
-   pr_err("Couldn't create iova cache\n");
-   ret = -ENOMEM;
-   }
-
-   return ret;
-}
-
-void iommu_iova_cache_destroy(void)
-{
-   kmem_cache_destroy(iommu_iova_cache);
-}
-
-struct iova *alloc_iova_mem(void)
-{
-   return kmem_cache_alloc(iommu_iova_cache, GFP_ATOMIC);
-}
-
-void free_iova_mem(struct iova *iova)
-{
-   kmem_cache_free(iommu_iova_cache, iova);
-}
-
 void
 init_iova_domain(struct iova_domain *iovad, unsigned long granule,
unsigned long start_pfn, unsigned long pfn_32bit)
@@ -242,6 +208,55 @@ iova_insert_rbtree(struct rb_root *root, struct iova *iova)
rb_insert_color(>node, root);
 }
 
+static struct kmem_cache *iova_cache;
+static unsigned int iova_cache_users;
+static DEFINE_MUTEX(iova_cache_mutex);
+
+struct iova *alloc_iova_mem(void)
+{
+   return kmem_cache_alloc(iova_cache, GFP_ATOMIC);
+}
+EXPORT_SYMBOL(alloc_iova_mem);
+
+void free_iova_mem(struct iova *iova)
+{
+   kmem_cache_free(iova_cache, iova);
+}
+EXPORT_SYMBOL(free_iova_mem);
+
+int iova_cache_get(void)
+{
+   mutex_lock(_cache_mutex);
+   if (!iova_cache_users) {
+   iova_cache = kmem_cache_create(
+   "iommu_iova", sizeof(struct iova), 0,
+   SLAB_HWCACHE_ALIGN, NULL);
+   if (!iova_cache) {
+   mutex_unlock(_cache_mutex);
+   printk(KERN_ERR "Couldn't create iova cache\n");
+   return -ENOMEM;
+   }
+   }
+
+   iova_cache_users++;
+   mutex_unlock(_cache_mutex);
+
+   return 0;
+}
+
+void iova_cache_put(void)
+{
+   mutex_lock(_cache_mutex);
+   if (WARN_ON(!iova_cache_users)) {
+   mutex_unlock(_cache_mutex);
+   return;
+   }
+   iova_cache_users--;
+   if (!iova_cache_users)
+   kmem_cache_destroy(iova_cache);
+   mutex_unlock(_cache_mutex);
+}
+
 /**
  * alloc_iova - allocates an iova
  * @iovad: - iova domain in question
diff --git a/include/linux/iova.h b/include/linux/iova.h
index 3920a19..92f7177 100644
--- a/include/linux/iova.h
+++ b/include/linux/iova.h
@@ -68,8 +68,8 @@ static inline unsigned long iova_pfn(struct iova_domain 
*iovad, dma_addr_t iova)
return iova >> iova_shift(iovad);
 }
 
-int iommu_iova_cache_init(void);
-void iommu_iova_cache_destroy(void);
+int iova_cache_get(void);
+void iova_cache_put(void);
 
 struct iova *alloc_iova_mem(void);
 void free_iova_mem(struct iova *iova);
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 02/22] iommu: iova: Export symbols

[Ashutosh Dixit]

From: Sakari Ailus 

Use EXPORT_SYMBOL_GPL() to export the iova library symbols. The symbols
include:

init_iova_domain();
iova_cache_get();
iova_cache_put();
iova_cache_init();
alloc_iova();
find_iova();
__free_iova();
free_iova();
put_iova_domain();
reserve_iova();
copy_reserved_iova();

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/iova.c | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index 400f4d1..dd87123 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -38,6 +38,7 @@ init_iova_domain(struct iova_domain *iovad, unsigned long 
granule,
iovad->start_pfn = start_pfn;
iovad->dma_32bit_pfn = pfn_32bit;
 }
+EXPORT_SYMBOL_GPL(init_iova_domain);
 
 static struct rb_node *
 __get_cached_rbnode(struct iova_domain *iovad, unsigned long *limit_pfn)
@@ -243,6 +244,7 @@ int iova_cache_get(void)
 
return 0;
 }
+EXPORT_SYMBOL_GPL(iova_cache_get);
 
 void iova_cache_put(void)
 {
@@ -256,6 +258,7 @@ void iova_cache_put(void)
kmem_cache_destroy(iova_cache);
mutex_unlock(_cache_mutex);
 }
+EXPORT_SYMBOL_GPL(iova_cache_put);
 
 /**
  * alloc_iova - allocates an iova
@@ -296,6 +299,7 @@ alloc_iova(struct iova_domain *iovad, unsigned long size,
 
return new_iova;
 }
+EXPORT_SYMBOL_GPL(alloc_iova);
 
 /**
  * find_iova - find's an iova for a given pfn
@@ -336,6 +340,7 @@ struct iova *find_iova(struct iova_domain *iovad, unsigned 
long pfn)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return NULL;
 }
+EXPORT_SYMBOL_GPL(find_iova);
 
 /**
  * __free_iova - frees the given iova
@@ -354,6 +359,7 @@ __free_iova(struct iova_domain *iovad, struct iova *iova)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
free_iova_mem(iova);
 }
+EXPORT_SYMBOL_GPL(__free_iova);
 
 /**
  * free_iova - finds and frees the iova for a given pfn
@@ -371,6 +377,7 @@ free_iova(struct iova_domain *iovad, unsigned long pfn)
__free_iova(iovad, iova);
 
 }
+EXPORT_SYMBOL_GPL(free_iova);
 
 /**
  * put_iova_domain - destroys the iova doamin
@@ -393,6 +400,7 @@ void put_iova_domain(struct iova_domain *iovad)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(put_iova_domain);
 
 static int
 __is_range_overlap(struct rb_node *node,
@@ -482,6 +490,7 @@ finish:
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return iova;
 }
+EXPORT_SYMBOL_GPL(reserve_iova);
 
 /**
  * copy_reserved_iova - copies the reserved between domains
@@ -508,6 +517,7 @@ copy_reserved_iova(struct iova_domain *from, struct 
iova_domain *to)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(copy_reserved_iova);
 
 struct iova *
 split_and_remove_iova(struct iova_domain *iovad, struct iova *iova,
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 19/22] misc: mic: SCIF remote memory map/unmap interface

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch implements the SCIF mmap/munmap interface. A similar
capability is provided to kernel clients via the
scif_get_pages()/scif_put_pages() APIs. The SCIF mmap interface
queries to check if a window is valid and then remaps the local
virtual address to the remote physical pages. These mappings are
subsequently destroyed upon receipt of the VMA close operation or
scif_get_pages().  This functionality allows SCIF users to directly
access remote memory without any driver interaction once the mappings
are created thereby providing bare-metal PCIe latency. These mappings
are zapped to avoid RMA accesses from user space, if a Coprocessor is
reset.

Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Reviewed-by: Nikhil Rao <nikhil@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_mmap.c | 699 ++
 1 file changed, 699 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_mmap.c

diff --git a/drivers/misc/mic/scif/scif_mmap.c 
b/drivers/misc/mic/scif/scif_mmap.c
new file mode 100644
index 000..49cb8f7
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_mmap.c
@@ -0,0 +1,699 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+
+/*
+ * struct scif_vma_info - Information about a remote memory mapping
+ *   created via scif_mmap(..)
+ * @vma: VM area struct
+ * @list: link to list of active vmas
+ */
+struct scif_vma_info {
+   struct vm_area_struct *vma;
+   struct list_head list;
+};
+
+void scif_recv_munmap(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_rma_req req;
+   struct scif_window *window = NULL;
+   struct scif_window *recv_window =
+   (struct scif_window *)msg->payload[0];
+   struct scif_endpt *ep;
+
+   ep = (struct scif_endpt *)recv_window->ep;
+   req.out_window = 
+   req.offset = recv_window->offset;
+   req.prot = recv_window->prot;
+   req.nr_bytes = recv_window->nr_pages << PAGE_SHIFT;
+   req.type = SCIF_WINDOW_FULL;
+   req.head = >rma_info.reg_list;
+   msg->payload[0] = ep->remote_ep;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Does a valid window exist? */
+   if (scif_query_window()) {
+   dev_err(>sdev->dev,
+   "%s %d -ENXIO\n", __func__, __LINE__);
+   msg->uop = SCIF_UNREGISTER_ACK;
+   goto error;
+   }
+
+   scif_put_window(window, window->nr_pages);
+
+   if (!window->ref_count) {
+   atomic_inc(>rma_info.tw_refcount);
+   ep->rma_info.async_list_del = 1;
+   list_del_init(>list);
+   scif_free_window_offset(ep, window, window->offset);
+   }
+error:
+   mutex_unlock(>rma_info.rma_lock);
+   if (window && !window->ref_count)
+   scif_queue_for_cleanup(window, _info.rma);
+}
+
+/*
+ * Remove valid remote memory mappings created via scif_mmap(..) from the
+ * process address space since the remote node is lost
+ */
+static void __scif_zap_mmaps(struct scif_endpt *ep)
+{
+   struct list_head *item;
+   struct scif_vma_info *info;
+   struct vm_area_struct *vma;
+   unsigned long size;
+
+   spin_lock(>lock);
+   list_for_each(item, >rma_info.vma_list) {
+   info = list_entry(item, struct scif_vma_info, list);
+   vma = info->vma;
+   size = vma->vm_end - vma->vm_start;
+   zap_vma_ptes(vma, vma->vm_start, size);
+   dev_dbg(scif_info.mdev.this_device,
+   "%s ep %p zap vma %p size 0x%lx\n",
+   __func__, ep, info->vma, size);
+   }
+   spin_unlock(>lock);
+}
+
+/*
+ * Traverse the list of endpoints for a particular remote node and
+ * zap valid remote memory mappings since the remote node is lost
+ */
+static void _scif_zap_mmaps(int node, struct list_head *head)
+{
+   struct scif_endpt *ep;
+   struct list_head *item;
+
+   mutex_lock(_info.connlock);
+   list_for_each(item, head) {
+   ep = list_entry(item, struct scif_endpt, list);
+   if (ep->remote_dev->node == node)
+   __scif_zap_mmaps(ep);

[PATCH char-misc-next v2 03/22] iommu: Make the iova library a module

[Ashutosh Dixit]

From: Sakari Ailus 

The iova library has use outside the intel-iommu driver, thus make it a
module.

Signed-off-by: Sakari Ailus 
Signed-off-by: David Woodhouse 
---
 drivers/iommu/Kconfig | 2 +-
 drivers/iommu/iova.c  | 4 
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index 4664c2a..d9da766 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -43,7 +43,7 @@ config IOMMU_IO_PGTABLE_LPAE_SELFTEST
 endmenu
 
 config IOMMU_IOVA
-   bool
+   tristate
 
 config OF_IOMMU
def_bool y
diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index dd87123..dd64d33 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -18,6 +18,7 @@
  */
 
 #include 
+#include 
 #include 
 
 void
@@ -559,3 +560,6 @@ error:
free_iova_mem(prev);
return NULL;
 }
+
+MODULE_AUTHOR("Anil S Keshavamurthy ");
+MODULE_LICENSE("GPL");
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next v2 16/22] misc: mic: SCIF RMA header file

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch adds the internal data structures required to perform SCIF
RMAs. The data structures required to maintain per SCIF endpoint, RMA
information are contained in scif_endpt_rma_info. scif_pinned_pages
describes a set of SCIF pinned pages maintained locally. The
scif_window is a data structure which contains all the fields required
to describe a SCIF registered window on self and remote nodes. It
contains an offset which is used as a key to perform SCIF DMAs and CPU
copies between self and remote registered windows.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_rma.h | 464 +++
 1 file changed, 464 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.h

diff --git a/drivers/misc/mic/scif/scif_rma.h b/drivers/misc/mic/scif/scif_rma.h
new file mode 100644
index 000..fa67222
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.h
@@ -0,0 +1,464 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ *   notice, this list of conditions and the following disclaimer in
+ *   the documentation and/or other materials provided with the
+ *   distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ *   contributors may be used to endorse or promote products derived
+ *   from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#ifndef SCIF_RMA_H
+#define SCIF_RMA_H
+
+#include 
+#include 
+
+#include "../bus/scif_bus.h"
+
+/* If this bit is set then the mark is a remote fence mark */
+#define SCIF_REMOTE_FENCE_BIT  31
+/* Magic value used to indicate a remote fence request */
+#define SCIF_REMOTE_FENCE BIT_ULL(SCIF_REMOTE_FENCE_BIT)
+
+#define SCIF_MAX_UNALIGNED_BUF_SIZE (1024 * 1024ULL)
+#define SCIF_KMEM_UNALIGNED_BUF_SIZE (SCIF_MAX_UNALIGNED_BUF_SIZE + \
+ (L1_CACHE_BYTES << 1))
+
+#define SCIF_IOVA_START_PFN(1)
+#define SCIF_IOVA_PFN(addr) ((addr) >> PAGE_SHIFT)
+#define SCIF_DMA_64BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(64))
+#define SCIF_DMA_63BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(63))
+
+/*
+ * struct scif_endpt_rma_info - Per Endpoint Remote Memory Access Information
+ *
+ * @reg_list: List of registration windows for self
+ * @remote_reg_list: List of registration windows for peer
+ * @iovad: Offset generator
+ * @rma_lock: Synchronizes access to self/remote list and also protects the
+ *   window from being destroyed while RMAs are in progress.
+ * @tc_lock: Synchronizes access to temporary cached windows list
+ *  for SCIF Registration Caching.
+ * @mmn_lock: Synchronizes access to the list of MMU notifiers registered
+ * @tw_refcount: Keeps track of number of outstanding temporary registered
+ *  windows created by scif_vreadfrom/scif_vwriteto which have
+ *  not been destroyed.
+ * @tcw_refcou

[PATCH char-misc-next v2 12/22] misc: mic: Remove COSM functionality from the MIC host driver

[Ashutosh Dixit]

Since COSM functionality is now moved into a separate COSM driver
drivers, this patch removes this functionality from the base MIC host
driver. The MIC host driver now implements cosm_hw_ops and registers a
COSM device which allows the COSM driver to trigger
boot/shutdown/reset of the MIC devices via the cosm_hw_ops.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Dasaratharaman Chandramouli 
<dasaratharaman.chandramo...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/Makefile   |   2 -
 drivers/misc/mic/host/Makefile  |   1 -
 drivers/misc/mic/host/mic_boot.c| 317 +++--
 drivers/misc/mic/host/mic_debugfs.c | 114 +
 drivers/misc/mic/host/mic_device.h  |  88 +--
 drivers/misc/mic/host/mic_fops.c|   4 +-
 drivers/misc/mic/host/mic_intr.c|  46 ++--
 drivers/misc/mic/host/mic_main.c| 223 +++---
 drivers/misc/mic/host/mic_smpt.c|  30 ++-
 drivers/misc/mic/host/mic_sysfs.c   | 459 
 drivers/misc/mic/host/mic_virtio.c  |  17 +-
 drivers/misc/mic/host/mic_virtio.h  |   2 +-
 drivers/misc/mic/host/mic_x100.c|  46 ++--
 13 files changed, 204 insertions(+), 1145 deletions(-)
 delete mode 100644 drivers/misc/mic/host/mic_sysfs.c

diff --git a/drivers/misc/mic/Makefile b/drivers/misc/mic/Makefile
index a74042c..a2e75f4 100644
--- a/drivers/misc/mic/Makefile
+++ b/drivers/misc/mic/Makefile
@@ -2,7 +2,5 @@
 # Makefile - Intel MIC Linux driver.
 # Copyright(c) 2013, Intel Corporation.
 #
-obj-$(CONFIG_INTEL_MIC_HOST) += host/
-obj-$(CONFIG_INTEL_MIC_CARD) += card/
 obj-y += bus/
 obj-$(CONFIG_SCIF) += scif/
diff --git a/drivers/misc/mic/host/Makefile b/drivers/misc/mic/host/Makefile
index c2197f9..004d3db 100644
--- a/drivers/misc/mic/host/Makefile
+++ b/drivers/misc/mic/host/Makefile
@@ -5,7 +5,6 @@
 obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
 mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
-mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_intr.o
 mic_host-objs += mic_boot.o
diff --git a/drivers/misc/mic/host/mic_boot.c b/drivers/misc/mic/host/mic_boot.c
index e5f6a5e..7845564 100644
--- a/drivers/misc/mic/host/mic_boot.c
+++ b/drivers/misc/mic/host/mic_boot.c
@@ -22,9 +22,9 @@
 #include 
 #include 
 #include 
-
 #include 
 #include 
+#include "../bus/scif_bus.h"
 #include "../common/mic_dev.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
@@ -99,7 +99,7 @@ static int __mic_dma_map_sg(struct device *dev, struct 
scatterlist *sg,
int i, j, ret;
dma_addr_t da;
 
-   ret = dma_map_sg(mdev->sdev->parent, sg, nents, dir);
+   ret = dma_map_sg(>pdev->dev, sg, nents, dir);
if (ret <= 0)
return 0;
 
@@ -115,7 +115,7 @@ err:
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = mic_to_dma_addr(mdev, sg_dma_address(s));
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
return 0;
 }
 
@@ -135,7 +135,7 @@ static void __mic_dma_unmap_sg(struct device *dev,
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = da;
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
 }
 
 static struct dma_map_ops __mic_dma_ops = {
@@ -270,48 +270,13 @@ static struct mbus_hw_ops mbus_hw_ops = {
.ack_interrupt = _mic_ack_interrupt,
 };
 
-/**
- * mic_reset - Reset the MIC device.
- * @mdev: pointer to mic_device instance
- */
-static void mic_reset(struct mic_device *mdev)
-{
-   int i;
-
-#define MIC_RESET_TO (45)
-
-   reinit_completion(>reset_wait);
-   mdev->ops->reset_fw_ready(mdev);
-   mdev->ops->reset(mdev);
-
-   for (i = 0; i < MIC_RESET_TO; i++) {
-   if (mdev->ops->is_fw_ready(mdev))
-   goto done;
-   /*
-* Resets typically take 10s of seconds to complete.
-* Since an MMIO read is required to check if the
-* firmware is ready or not, a 1 second delay works nicely.
-*/
-   msleep(1000);
-   }
-   mic_set_state(mdev, MIC_RESET_FAILED);
-done:
-   complete_all(>reset_wait);
-}
-
 /* Initialize the MIC bootparams */
 void mic_bootparam_init(struct mic_device *mdev)
 {
struct mic_bootparam *bootparam = mdev->dp;
 
bootparam->magic = cpu_to_le32(MIC_MAGIC);
-   bootparam->c2h_shutdown_db = mdev->shutdown_db;
-   bootparam->h2c_shutdown_db = -1;
bootparam->h2c_config_db = -1;
-   bootparam->shutdown_status = 0;
-   bootparam->shutdown_card = 0;
-

[PATCH char-misc-next v2 18/22] misc: mic: SCIF RMA list operations

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch adds the implementation for operations performed on the
list of SCIF windows. Examples of such operations includes adding the
windows to the list of registered (or cached) windows, querying the
list of self or remote windows and unregistering windows. The query
operation is used by SCIF APIs which initiate DMAs, CPU copies or
fences to ensure that a window remains valid during a transfer.

Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Signed-off-by: Nikhil Rao <nikhil@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_rma_list.c | 291 ++
 drivers/misc/mic/scif/scif_rma_list.h |  57 +++
 2 files changed, 348 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.c
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.h

diff --git a/drivers/misc/mic/scif/scif_rma_list.c 
b/drivers/misc/mic/scif/scif_rma_list.c
new file mode 100644
index 000..e1ef8da
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma_list.c
@@ -0,0 +1,291 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include 
+#include 
+
+/*
+ * scif_insert_tcw:
+ *
+ * Insert a temp window to the temp registration list sorted by va_for_temp.
+ * RMA lock must be held.
+ */
+void scif_insert_tcw(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL;
+   struct scif_window *prev = list_entry(head, struct scif_window, list);
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   /* Compare with tail and if the entry is new tail add it to the end */
+   if (!list_empty(head)) {
+   curr = list_entry(head->prev, struct scif_window, list);
+   if (curr->va_for_temp < window->va_for_temp) {
+   list_add_tail(>list, head);
+   return;
+   }
+   }
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->va_for_temp > window->va_for_temp)
+   break;
+   prev = curr;
+   }
+   list_add(>list, >list);
+}
+
+/*
+ * scif_insert_window:
+ *
+ * Insert a window to the self registration list sorted by offset.
+ * RMA lock must be held.
+ */
+void scif_insert_window(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL, *prev = NULL;
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->offset > window->offset)
+   break;
+   prev = curr;
+   }
+   if (!prev)
+   list_add(>list, head);
+   else
+   list_add(>list, >list);
+   scif_set_window_ref(window, window->nr_pages);
+}
+
+/*
+ * scif_query_tcw:
+ *
+ * Query the temp cached registration list of ep for an overlapping window
+ * in case of permission mismatch, destroy the previous window. if permissions
+ * match and overlap is partial, destroy the window but return the new range
+ * RMA lock must be held.
+ */
+int scif_query_tcw(struct scif_endpt *ep, struct scif_rma_req *req)
+{
+   struct list_head *item, *temp, *head = req->head;
+   struct scif_window *window;
+   u64 start_va_window, start_va_req = req->va_for_temp;
+   u64 end_va_window, end_va_req = start_va_req + req->nr_bytes;
+
+   if (!req->nr_bytes)
+   return -EINVAL;
+   /*
+* Avoid traversing the entire list to find out that there
+* is no entry that matches
+*/
+   if (!list_empty(head)) {
+   window = list_last_entry(head, struct scif_window, list);
+   end_va_window = window->va_for_temp +
+   (window->nr_pages << PAGE_SHIFT);
+   if (start_va_req > end_va_window)
+   return -ENXIO;
+   }
+   list_for_each_safe(item, temp, head) {
+   window = list_entry(item, struct scif_window, list);
+   start_va_window = window->va_for_temp;
+   end_va_window = window->va_for_temp +
+   (window->

[PATCH char-misc-next v2 20/22] misc: mic: SCIF DMA and CPU copy interface

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

SCIF allows users to read from or write to registered remote memory
via CPU copies or DMA. The API verifies that both local and remote
windows are valid before initiating the CPU or DMA transfers. SCIF has
optimized algorithms for handling byte aligned as well as cache line
aligned DMA engines. A registration cache is maintained to avoid the
overhead of pinning pages repeatedly if buffers are reused. The
registration cache is invalidated upon receipt of MMU notifier
callbacks.  SCIF windows are destroyed and the pages are unpinned only
once all prior DMAs initiated using that window are drained. Users can
request synchronous DMA operations as well as tail byte ordering if
required. CPU copies are always performed synchronously.

Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Reviewed-by: Nikhil Rao <nikhil@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_dma.c | 1979 ++
 1 file changed, 1979 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_dma.c

diff --git a/drivers/misc/mic/scif/scif_dma.c b/drivers/misc/mic/scif/scif_dma.c
new file mode 100644
index 000..95a13c6
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_dma.c
@@ -0,0 +1,1979 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include "scif_map.h"
+
+/*
+ * struct scif_dma_comp_cb - SCIF DMA completion callback
+ *
+ * @dma_completion_func: DMA completion callback
+ * @cb_cookie: DMA completion callback cookie
+ * @temp_buf: Temporary buffer
+ * @temp_buf_to_free: Temporary buffer to be freed
+ * @is_cache: Is a kmem_cache allocated buffer
+ * @dst_offset: Destination registration offset
+ * @dst_window: Destination registration window
+ * @len: Length of the temp buffer
+ * @temp_phys: DMA address of the temp buffer
+ * @sdev: The SCIF device
+ * @header_padding: padding for cache line alignment
+ */
+struct scif_dma_comp_cb {
+   void (*dma_completion_func)(void *cookie);
+   void *cb_cookie;
+   u8 *temp_buf;
+   u8 *temp_buf_to_free;
+   bool is_cache;
+   s64 dst_offset;
+   struct scif_window *dst_window;
+   size_t len;
+   dma_addr_t temp_phys;
+   struct scif_dev *sdev;
+   int header_padding;
+};
+
+/**
+ * struct scif_copy_work - Work for DMA copy
+ *
+ * @src_offset: Starting source offset
+ * @dst_offset: Starting destination offset
+ * @src_window: Starting src registered window
+ * @dst_window: Starting dst registered window
+ * @loopback: true if this is a loopback DMA transfer
+ * @len: Length of the transfer
+ * @comp_cb: DMA copy completion callback
+ * @remote_dev: The remote SCIF peer device
+ * @fence_type: polling or interrupt based
+ * @ordered: is this a tail byte ordered DMA transfer
+ */
+struct scif_copy_work {
+   s64 src_offset;
+   s64 dst_offset;
+   struct scif_window *src_window;
+   struct scif_window *dst_window;
+   int loopback;
+   size_t len;
+   struct scif_dma_comp_cb   *comp_cb;
+   struct scif_dev *remote_dev;
+   int fence_type;
+   bool ordered;
+};
+
+#ifndef list_entry_next
+#define list_entry_next(pos, member) \
+   list_entry(pos->member.next, typeof(*pos), member)
+#endif
+
+/**
+ * scif_reserve_dma_chan:
+ * @ep: Endpoint Descriptor.
+ *
+ * This routine reserves a DMA channel for a particular
+ * endpoint. All DMA transfers for an endpoint are always
+ * programmed on the same DMA channel.
+ */
+int scif_reserve_dma_chan(struct scif_endpt *ep)
+{
+   int err = 0;
+   struct scif_dev *scifdev;
+   struct scif_hw_dev *sdev;
+   struct dma_chan *chan;
+
+   /* Loopback DMAs are not supported on the management node */
+   if (!scif_info.nodeid && scifdev_self(ep->remote_dev))
+   return 0;
+   if (scif_info.nodeid)
+   scifdev = _dev[0];
+   else
+   scifdev = ep->remote_dev;
+   sdev = scifdev->sdev;
+   if (!sdev->num_dma_ch)
+   return -ENODEV;
+   chan = sdev->dma_ch[scifdev->dma_ch_idx];
+   scifdev->dma_ch_idx = (scifdev->dma_ch_idx + 1) % sdev->num_dma_ch;
+   mutex_lock(>rma_info.rma_lock);
+   ep->rma_info.dma_chan = chan;
+   mutex_unlock(>rma_info.rma_lock);
+   return err;
+}
+
+#ifdef CONFIG_MMU_NOT

[PATCH char-misc-next v2 11/22] misc: mic: COSM client driver

[Ashutosh Dixit]

The COSM client driver running on the MIC cards is implemented as a
kernel mode SCIF client. It responds to a "shutdown" message from the
host by triggering a card shutdown and also communicates the shutdown
or reboot status back the host. It is also responsible for syncing the
card time to that of the host. Because SCIF messaging cannot be used
in a panic context, the COSM client driver also periodically sends a
heartbeat SCIF message to the host thereby enabling the host to detect
card crashes.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/cosm_client/Makefile   |   7 +
 drivers/misc/mic/cosm_client/cosm_scif_client.c | 275 
 2 files changed, 282 insertions(+)
 create mode 100644 drivers/misc/mic/cosm_client/Makefile
 create mode 100644 drivers/misc/mic/cosm_client/cosm_scif_client.c

diff --git a/drivers/misc/mic/cosm_client/Makefile 
b/drivers/misc/mic/cosm_client/Makefile
new file mode 100644
index 000..6f751a5
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/Makefile
@@ -0,0 +1,7 @@
+#
+# Makefile - Intel MIC COSM Client Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += cosm_client.o
+
+cosm_client-objs += cosm_scif_client.o
diff --git a/drivers/misc/mic/cosm_client/cosm_scif_client.c 
b/drivers/misc/mic/cosm_client/cosm_scif_client.c
new file mode 100644
index 000..03e98bf
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/cosm_scif_client.c
@@ -0,0 +1,275 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Client Driver
+ *
+ */
+#include 
+#include 
+#include 
+#include 
+#include "../cosm/cosm_main.h"
+
+#define COSM_SCIF_MAX_RETRIES 10
+#define COSM_HEARTBEAT_SEND_MSEC (COSM_HEARTBEAT_SEND_SEC * MSEC_PER_SEC)
+
+static struct task_struct *client_thread;
+static scif_epd_t client_epd;
+static struct scif_peer_dev *client_spdev;
+
+/*
+ * Reboot notifier: receives shutdown status from the OS and communicates it
+ * back to the COSM process on the host
+ */
+static int cosm_reboot_event(struct notifier_block *this, unsigned long event,
+void *ptr)
+{
+   struct cosm_msg msg = { .id = COSM_MSG_SHUTDOWN_STATUS };
+   int rc;
+
+   event = (event == SYS_RESTART) ? SYSTEM_RESTART : event;
+   dev_info(_spdev->dev, "%s %d received event %ld\n",
+__func__, __LINE__, event);
+
+   msg.shutdown_status = event;
+   rc = scif_send(client_epd, , sizeof(msg), SCIF_SEND_BLOCK);
+   if (rc < 0)
+   dev_err(_spdev->dev, "%s %d scif_send rc %d\n",
+   __func__, __LINE__, rc);
+
+   return NOTIFY_DONE;
+}
+
+static struct notifier_block cosm_reboot = {
+   .notifier_call  = cosm_reboot_event,
+};
+
+/* Set system time from timespec value received from the host */
+static void cosm_set_time(struct cosm_msg *msg)
+{
+   int rc = do_settimeofday64(>timespec);
+
+   if (rc)
+   dev_err(_spdev->dev, "%s: %d settimeofday rc %d\n",
+   __func__, __LINE__, rc);
+}
+
+/* COSM client receive message processing */
+static void cosm_client_recv(void)
+{
+   struct cosm_msg msg;
+   int rc;
+
+   while (1) {
+   rc = scif_recv(client_epd, , sizeof(msg), 0);
+   if (!rc) {
+   return;
+   } else if (rc < 0) {
+   dev_err(_spdev->dev, "%s: %d rc %d\n",
+   __func__, __LINE__, rc);
+   return;
+   }
+
+   dev_dbg(_spdev->dev, "%s: %d rc %d id 0x%llx\n",
+   __func__, __LINE__, rc, msg.id);
+
+   switch (msg.id) {
+   case COSM_MSG_SYNC_TIME:
+   cosm_set_time();
+   break;
+   case COSM_MSG_SHUTDOWN:
+   orderly_poweroff(true);
+   break;
+   default:
+   dev_err(_spdev->dev, "%s: %d unknown id %lld\n",
+   __func__, __LINE__, msg.id);
+   break;
+   }
+   

[PATCH char-misc-next v2 14/22] misc: mic: Update MIC host daemon with COSM changes

[Ashutosh Dixit]

This patch updates the MIC host daemon to work with corresponding
changes in COSM. Other MIC daemon fixes, cleanups and enhancements as
are also rolled into this patch. Changes to MIC sysfs ABI which go
into effect with this patch are also documented.

Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Dasaratharaman Chandramouli 
<dasaratharaman.chandramo...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 Documentation/ABI/testing/sysfs-class-mic.txt |  29 ++-
 Documentation/mic/mic_overview.txt|  31 ++-
 Documentation/mic/mpssd/mpss  |   4 +-
 Documentation/mic/mpssd/mpssd.c   | 362 --
 Documentation/mic/mpssd/mpssd.h   |   1 +
 drivers/misc/mic/Makefile |   4 +
 drivers/misc/mic/bus/scif_bus.c   |   7 +-
 drivers/misc/mic/bus/scif_bus.h   |   6 +-
 include/uapi/linux/mic_common.h   |  16 +-
 9 files changed, 285 insertions(+), 175 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-class-mic.txt 
b/Documentation/ABI/testing/sysfs-class-mic.txt
index 13f48af..d45eed2 100644
--- a/Documentation/ABI/testing/sysfs-class-mic.txt
+++ b/Documentation/ABI/testing/sysfs-class-mic.txt
@@ -41,18 +41,15 @@ Description:
When read, this entry provides the current state of an Intel
MIC device in the context of the card OS. Possible values that
will be read are:
-   "offline" - The MIC device is ready to boot the card OS. On
+   "ready" - The MIC device is ready to boot the card OS. On
reading this entry after an OSPM resume, a "boot" has to be
written to this entry if the card was previously shutdown
during OSPM suspend.
-   "online" - The MIC device has initiated booting a card OS.
+   "booting" - The MIC device has initiated booting a card OS.
+   "online" - The MIC device has completed boot and is online
"shutting_down" - The card OS is shutting down.
+   "resetting" - A reset has been initiated for the MIC device
"reset_failed" - The MIC device has failed to reset.
-   "suspending" - The MIC device is currently being prepared for
-   suspend. On reading this entry, a "suspend" has to be written
-   to the state sysfs entry to ensure the card is shutdown during
-   OSPM suspend.
-   "suspended" - The MIC device has been suspended.
 
When written, this sysfs entry triggers different state change
operations depending upon the current state of the card OS.
@@ -62,8 +59,6 @@ Description:
sysfs entries.
"reset" - Initiates device reset.
"shutdown" - Initiates card OS shutdown.
-   "suspend" - Initiates card OS shutdown and also marks the card
-   as suspended.
 
 What:  /sys/class/mic/mic(x)/shutdown_status
 Date:  October 2013
@@ -126,7 +121,7 @@ Description:
the card. This sysfs entry can be written with the following
valid strings:
a) linux - Boot a Linux image.
-   b) elf - Boot an elf image for flash updates.
+   b) flash - Boot an image for flash updates.
 
 What:  /sys/class/mic/mic(x)/log_buf_addr
 Date:  October 2013
@@ -155,3 +150,17 @@ Description:
daemon to set the log buffer length address. The correct log
buffer length address to be written can be found in the
    System.map file of the card OS.
+
+What:  /sys/class/mic/mic(x)/heartbeat_enable
+Date:  March 2015
+KernelVersion: 3.20
+Contact:   Ashutosh Dixit <ashutosh.di...@intel.com>
+Description:
+   The MIC drivers detect and inform user space about card crashes
+   via a heartbeat mechanism (see the description of
+   shutdown_status above). User space can turn off this
+   notification by setting heartbeat_enable to 0 and enable it by
+   setting this entry to 1. If this notification is disabled it is
+   the responsibility of user space to detect card crashes via
+   alternative means such as a network ping. This setting is
+   enabled by default.
diff --git a/Documentation/mic/mic_overview.txt 
b/Documentation/mic/mic_overview.txt
index 1a2f2c8..73f44fc 100644
--- a/Documentation/mic/mic_overview.txt
+++ b/Documentation/mic/mic_overview.txt
@@ -28,6 +28,10 @@ The Symmetric Communication Interface (SCIF (pronounced as 
skiff)) is a
 low level communications API across PCIe curre

[PATCH char-misc-next v2 21/22] misc: mic: SCIF fence

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch implements the fence APIs required to synchronize
DMAs. SCIF provides an interface to return a "mark" for all DMAs
programmed at the instant the API was called. Users can then "wait" on
the mark provided previously by blocking inside the kernel. Upon
receipt of a DMA completion interrupt the waiting thread is woken
up. There is also an interface to signal DMA completion by polling for
a location to be updated via a "signal" cookie to avoid the interrupt
overhead in the mark/wait interface. SCIF allows programming fences on
both the local and the remote node for both the mark/wait or the fence
signal APIs.

Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Reviewed-by: Nikhil Rao <nikhil@intel.com>
Signed-off-by: Jacek Lawrynowicz <jacek.lawrynow...@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_fence.c | 771 +
 1 file changed, 771 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_fence.c

diff --git a/drivers/misc/mic/scif/scif_fence.c 
b/drivers/misc/mic/scif/scif_fence.c
new file mode 100644
index 000..7f2c96f
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_fence.c
@@ -0,0 +1,771 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+
+#include "scif_main.h"
+
+/**
+ * scif_recv_mark: Handle SCIF_MARK request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested a mark.
+ */
+void scif_recv_mark(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   int mark, err;
+
+   err = _scif_fence_mark(ep, );
+   if (err)
+   msg->uop = SCIF_MARK_NACK;
+   else
+   msg->uop = SCIF_MARK_ACK;
+   msg->payload[0] = ep->remote_ep;
+   msg->payload[2] = mark;
+   scif_nodeqp_send(ep->remote_dev, msg);
+}
+
+/**
+ * scif_recv_mark_resp: Handle SCIF_MARK_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_MARK message.
+ */
+void scif_recv_mark_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if (msg->uop == SCIF_MARK_ACK) {
+   fence_req->state = OP_COMPLETED;
+   fence_req->dma_mark = (int)msg->payload[2];
+   } else {
+   fence_req->state = OP_FAILED;
+   }
+   mutex_unlock(>rma_info.rma_lock);
+   complete(_req->comp);
+}
+
+/**
+ * scif_recv_wait: Handle SCIF_WAIT request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested waiting on a fence.
+ */
+void scif_recv_wait(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_remote_fence_info *fence;
+
+   /*
+* Allocate structure for remote fence information and
+* send a NACK if the allocation failed. The peer will
+* return ENOMEM upon receiving a NACK.
+*/
+   fence = kmalloc(sizeof(*fence), GFP_KERNEL);
+   if (!fence) {
+   msg->payload[0] = ep->remote_ep;
+   msg->uop = SCIF_WAIT_NACK;
+   scif_nodeqp_send(ep->remote_dev, msg);
+   return;
+   }
+
+   /* Prepare the fence request */
+   memcpy(>msg, msg, sizeof(struct scifmsg));
+   INIT_LIST_HEAD(>list);
+
+   /* Insert to the global remote fence request list */
+   mutex_lock(_info.fencelock);
+   atomic_inc(>rma_info.fence_refcount);
+   list_add_tail(>list, _info.fence);
+   mutex_unlock(_info.fencelock);
+
+   schedule_work(_info.misc_work);
+}
+
+/**
+ * scif_recv_wait_resp: Handle SCIF_WAIT_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_WAIT message.
+ */
+void scif_recv_wait_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if

[PATCH char-misc-next v2 13/22] misc: mic: Remove COSM functionality from the MIC card driver

[Ashutosh Dixit]

Since card side COSM functionality, to trigger MIC device shutdowns
and communicate shutdown status to the host, is now moved into a
separate COSM client driver, this patch removes this functionality
from the base MIC card driver. The mic_bus driver is also updated to
use the device index provided by COSM rather than maintain its own
device index.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Sudeep Dutt <sudeep.d...@intel.com>
Signed-off-by: Ashutosh Dixit <ashutosh.di...@intel.com>
---
 drivers/misc/mic/bus/mic_bus.c | 22 ++
 drivers/misc/mic/card/mic_device.c | 88 ++
 drivers/misc/mic/card/mic_x100.c   |  2 +-
 drivers/misc/mic/common/mic_dev.h  | 13 ++
 include/linux/mic_bus.h|  3 +-
 5 files changed, 23 insertions(+), 105 deletions(-)

diff --git a/drivers/misc/mic/bus/mic_bus.c b/drivers/misc/mic/bus/mic_bus.c
index 961ae90..c64955d 100644
--- a/drivers/misc/mic/bus/mic_bus.c
+++ b/drivers/misc/mic/bus/mic_bus.c
@@ -25,9 +25,6 @@
 #include 
 #include 
 
-/* Unique numbering for mbus devices. */
-static DEFINE_IDA(mbus_index_ida);
-
 static ssize_t device_show(struct device *d,
   struct device_attribute *attr, char *buf)
 {
@@ -147,7 +144,8 @@ static void mbus_release_dev(struct device *d)
 
 struct mbus_device *
 mbus_register_device(struct device *pdev, int id, struct dma_map_ops *dma_ops,
-struct mbus_hw_ops *hw_ops, void __iomem *mmio_va)
+struct mbus_hw_ops *hw_ops, int index,
+void __iomem *mmio_va)
 {
int ret;
struct mbus_device *mbdev;
@@ -166,13 +164,7 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
mbdev->dev.release = mbus_release_dev;
mbdev->hw_ops = hw_ops;
mbdev->dev.bus = _bus;
-
-   /* Assign a unique device index and hence name. */
-   ret = ida_simple_get(_index_ida, 0, 0, GFP_KERNEL);
-   if (ret < 0)
-   goto free_mbdev;
-
-   mbdev->index = ret;
+   mbdev->index = index;
dev_set_name(>dev, "mbus-dev%u", mbdev->index);
/*
 * device_register() causes the bus infrastructure to look for a
@@ -180,10 +172,8 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
 */
ret = device_register(>dev);
if (ret)
-   goto ida_remove;
+   goto free_mbdev;
return mbdev;
-ida_remove:
-   ida_simple_remove(_index_ida, mbdev->index);
 free_mbdev:
kfree(mbdev);
return ERR_PTR(ret);
@@ -192,10 +182,7 @@ EXPORT_SYMBOL_GPL(mbus_register_device);
 
 void mbus_unregister_device(struct mbus_device *mbdev)
 {
-   int index = mbdev->index; /* save for after device release */
-
device_unregister(>dev);
-   ida_simple_remove(_index_ida, index);
 }
 EXPORT_SYMBOL_GPL(mbus_unregister_device);
 
@@ -207,7 +194,6 @@ static int __init mbus_init(void)
 static void __exit mbus_exit(void)
 {
bus_unregister(_bus);
-   ida_destroy(_index_ida);
 }
 
 core_initcall(mbus_init);
diff --git a/drivers/misc/mic/card/mic_device.c 
b/drivers/misc/mic/card/mic_device.c
index 6338908..d0edaf7 100644
--- a/drivers/misc/mic/card/mic_device.c
+++ b/drivers/misc/mic/card/mic_device.c
@@ -37,71 +37,6 @@
 #include "mic_virtio.h"
 
 static struct mic_driver *g_drv;
-static struct mic_irq *shutdown_cookie;
-
-static void mic_notify_host(u8 state)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(state, >shutdown_status);
-   dev_dbg(mdrv->dev, "%s %d system_state %d\n",
-   __func__, __LINE__, state);
-   mic_send_intr(>mdev, ioread8(>c2h_shutdown_db));
-}
-
-static int mic_panic_event(struct notifier_block *this, unsigned long event,
-   void *ptr)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(-1, >h2c_config_db);
-   iowrite8(-1, >h2c_shutdown_db);
-   mic_notify_host(MIC_CRASHED);
-   return NOTIFY_DONE;
-}
-
-static struct notifier_block mic_panic = {
-   .notifier_call  = mic_panic_event,
-};
-
-static irqreturn_t mic_shutdown_isr(int irq, void *data)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   mic_ack_interrupt(_drv->mdev);
-   if (ioread8(>shutdown_card))
-   orderly_poweroff(true);
-   return IRQ_HANDLED;
-}
-
-static int mic_shutdown_init(void)
-{
-   int rc = 0;
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-   int shutdown_db;
-
-   shutdown_db = mic_next_card_db();
-   shutdown_cookie = mic_request_card_irq(mic_shutdown_isr,

[PATCH char-misc-next v2 15/22] misc: mic: SCIF RMA header file and IOCTL changes

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch updates the SCIF header file and IOCTL interface with the
changes required to support RMAs.  APIs added include the ability to
pin pages and register those pages with SCIF. SCIF kernel clients can
also add references to remote registered pages and access them via the
CPU. The user space IOCTL interface has been updated to enable SCIF
registration, RDMA/CPU copies and fence APIs for RDMA synchronization.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 include/linux/scif.h| 234 ++--
 include/uapi/linux/scif_ioctl.h |  85 +++
 2 files changed, 309 insertions(+), 10 deletions(-)

diff --git a/include/linux/scif.h b/include/linux/scif.h
index fd62c05..49a35d6 100644
--- a/include/linux/scif.h
+++ b/include/linux/scif.h
@@ -93,6 +93,27 @@ enum {
 #define SCIF_PORT_RSVD 1088
 
 typedef struct scif_endpt *scif_epd_t;
+typedef struct scif_pinned_pages *scif_pinned_pages_t;
+
+/**
+ * struct scif_range - SCIF registered range used in kernel mode
+ * @cookie: cookie used internally by SCIF
+ * @nr_pages: number of pages of PAGE_SIZE
+ * @prot_flags: R/W protection
+ * @phys_addr: Array of bus addresses
+ * @va: Array of kernel virtual addresses backed by the pages in the phys_addr
+ * array. The va is populated only when called on the host for a remote
+ * SCIF connection on MIC. This is required to support the use case of DMA
+ * between MIC and another device which is not a SCIF node e.g., an IB or
+ * ethernet NIC.
+ */
+struct scif_range {
+   void *cookie;
+   int nr_pages;
+   int prot_flags;
+   dma_addr_t *phys_addr;
+   void __iomem **va;
+};
 
 /**
  * struct scif_pollepd - SCIF endpoint to be monitored via scif_poll
@@ -389,7 +410,6 @@ int scif_close(scif_epd_t epd);
  * Errors:
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -442,7 +462,6 @@ int scif_send(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The destination node is returning from a low power state
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -505,9 +524,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * SCIF_PROT_READ - allow read operations from the window
  * SCIF_PROT_WRITE - allow write operations to the window
  *
- * The map_flags argument can be set to SCIF_MAP_FIXED which interprets a
- * fixed offset.
- *
  * Return:
  * Upon successful completion, scif_register() returns the offset at which the
  * mapping was placed (po); otherwise in user mode SCIF_REGISTER_FAILED (that
@@ -520,7 +536,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The mapping could not be performed due to lack of resources
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - map_flags is invalid, or prot_flags is invalid, or SCIF_MAP_FIXED 
is
  * set in flags, and offset is not a multiple of the page size, or addr is not 
a
  * multiple of the page size, or len is not a multiple of the page size, or is
@@ -803,7 +818,6 @@ int scif_writeto(scif_epd_t epd, off_t loffset, size_t len, 
off_t
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -884,7 +898,6 @@ int scif_vreadfrom(scif_epd_t epd, void *addr, size_t len, 
off_t roffset,
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -1028,11 +1041,212 @@ int scif_fence_signal(scif_epd_t epd, off_t loff, u64 
lval, off_t roff,
  * online nodes in the SCIF network including 'self'; otherwise in user mode
  * -1 is returned and errno is s

[PATCH char-misc-next v2 17/22] misc: mic: SCIF memory registration and unregistration

[Ashutosh Dixit]

From: Sudeep Dutt <sudeep.d...@intel.com>

This patch implements the SCIF APIs required to pin and unpin
pages. SCIF registration locks down the pages. It then sends a remote
window allocation request to the peer. Once the peer has allocated
memory, the local SCIF endpoint copies the pinned page information to
the peer and notifies the peer once the copy has complete. The peer
upon receipt of the registration notification adds the new remote
window to its list. At this point the window page information is
available on both self and remote nodes so that they can start
performing SCIF DMAs, CPU copies and fences. The unregistration API
tears down the registration at both self and remote nodes.

Reviewed-by: Nikhil Rao <nikhil@intel.com>
Reviewed-by: Ashutosh Dixit <ashutosh.di...@intel.com>
Signed-off-by: Sudeep Dutt <sudeep.d...@intel.com>
---
 drivers/misc/mic/scif/scif_rma.c | 1770 ++
 1 file changed, 1770 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.c

diff --git a/drivers/misc/mic/scif/scif_rma.c b/drivers/misc/mic/scif/scif_rma.c
new file mode 100644
index 000..e288996
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.c
@@ -0,0 +1,1770 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include 
+#include 
+#include "scif_main.h"
+#include "scif_map.h"
+
+/* Used to skip ulimit checks for registrations with SCIF_MAP_KERNEL flag */
+#define SCIF_MAP_ULIMIT 0x40
+
+bool scif_ulimit_check = 1;
+
+/**
+ * scif_rma_ep_init:
+ * @ep: end point
+ *
+ * Initialize RMA per EP data structures.
+ */
+void scif_rma_ep_init(struct scif_endpt *ep)
+{
+   struct scif_endpt_rma_info *rma = >rma_info;
+
+   mutex_init(>rma_lock);
+   init_iova_domain(>iovad, PAGE_SIZE, SCIF_IOVA_START_PFN,
+SCIF_DMA_64BIT_PFN);
+   spin_lock_init(>tc_lock);
+   mutex_init(>mmn_lock);
+   INIT_LIST_HEAD(>reg_list);
+   INIT_LIST_HEAD(>remote_reg_list);
+   atomic_set(>tw_refcount, 0);
+   atomic_set(>tcw_refcount, 0);
+   atomic_set(>tcw_total_pages, 0);
+   atomic_set(>fence_refcount, 0);
+
+   rma->async_list_del = 0;
+   rma->dma_chan = NULL;
+   INIT_LIST_HEAD(>mmn_list);
+   INIT_LIST_HEAD(>vma_list);
+   init_waitqueue_head(>markwq);
+}
+
+/**
+ * scif_rma_ep_can_uninit:
+ * @ep: end point
+ *
+ * Returns 1 if an endpoint can be uninitialized and 0 otherwise.
+ */
+int scif_rma_ep_can_uninit(struct scif_endpt *ep)
+{
+   int ret = 0;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Destroy RMA Info only if both lists are empty */
+   if (list_empty(>rma_info.reg_list) &&
+   list_empty(>rma_info.remote_reg_list) &&
+   list_empty(>rma_info.mmn_list) &&
+   !atomic_read(>rma_info.tw_refcount) &&
+   !atomic_read(>rma_info.tcw_refcount) &&
+   !atomic_read(>rma_info.fence_refcount))
+   ret = 1;
+   mutex_unlock(>rma_info.rma_lock);
+   return ret;
+}
+
+/**
+ * scif_create_pinned_pages:
+ * @nr_pages: number of pages in window
+ * @prot: read/write protection
+ *
+ * Allocate and prepare a set of pinned pages.
+ */
+static struct scif_pinned_pages *
+scif_create_pinned_pages(int nr_pages, int prot)
+{
+   struct scif_pinned_pages *pin;
+
+   might_sleep();
+   pin = scif_zalloc(sizeof(*pin));
+   if (!pin)
+   goto error;
+
+   pin->pages = scif_zalloc(nr_pages * sizeof(*pin->pages));
+   if (!pin->pages)
+   goto error_free_pinned_pages;
+
+   pin->prot = prot;
+   pin->magic = SCIFEP_MAGIC;
+   return pin;
+
+error_free_pinned_pages:
+   scif_free(pin, sizeof(*pin));
+error:
+   return NULL;
+}
+
+/**
+ * scif_destroy_pinned_pages:
+ * @pin: A set of pinned pages.
+ *
+ * Deallocate resources for pinned pages.
+ */
+static int scif_destroy_pinned_pages(struct scif_pinned_pages *pin)
+{
+   int j;
+   int writeable = pin->prot & SCIF_PROT_WRITE;
+   int kernel = SCIF_MAP_KERNEL & pin->map_flags;
+
+   for (j = 0; j < pin->nr_pages; j++) {
+   if (pin->pages[j] && !kernel) {
+   if (writeable)
+   SetPageDirty(pin->pages[j]);
+ 

Re: [PATCH char-misc-next 10/19] lib: convert iova.c into a library

[Ashutosh Dixit]

On Tue, Jul 28 2015 at 01:40:19 PM, Andrew Morton  
wrote:
> On Mon, 27 Jul 2015 16:57:32 -0700 Ashutosh Dixit  
> wrote:
>
>> From: Harish Chegondi 
>>
>> This patch converts iova.c into a library, moving it from
>> drivers/iommu/ to lib/, and exports its virtual address allocation
>> and management functions so that other modules can reuse them.
>
> From the following emails it is unclear to me whether we'll actually
> be going ahead with this, but whatever.  It's a chance to do some code
> reading.

Thanks for the review. Either us or the IOMMU team will submit a patch
incorporating your suggestions.

Ashutosh
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH char-misc-next 10/19] lib: convert iova.c into a library

[Ashutosh Dixit]

On Tue, Jul 28 2015 at 01:40:19 PM, Andrew Morton a...@linux-foundation.org 
wrote:
 On Mon, 27 Jul 2015 16:57:32 -0700 Ashutosh Dixit ashutosh.di...@intel.com 
 wrote:

 From: Harish Chegondi harish.chego...@intel.com

 This patch converts iova.c into a library, moving it from
 drivers/iommu/ to lib/, and exports its virtual address allocation
 and management functions so that other modules can reuse them.

 From the following emails it is unclear to me whether we'll actually
 be going ahead with this, but whatever.  It's a chance to do some code
 reading.

Thanks for the review. Either us or the IOMMU team will submit a patch
incorporating your suggestions.

Ashutosh
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next 06/19] misc: mic: COSM client driver

[Ashutosh Dixit]

The COSM client driver running on the MIC cards is implemented as a
kernel mode SCIF client. It responds to a "shutdown" message from the
host by triggering a card shutdown and also communicates the shutdown
or reboot status back the host. It is also responsible for syncing the
card time to that of the host. Because SCIF messaging cannot be used
in a panic context, the COSM client driver also periodically sends a
heartbeat SCIF message to the host thereby enabling the host to detect
card crashes.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm_client/Makefile   |   7 +
 drivers/misc/mic/cosm_client/cosm_scif_client.c | 275 
 2 files changed, 282 insertions(+)
 create mode 100644 drivers/misc/mic/cosm_client/Makefile
 create mode 100644 drivers/misc/mic/cosm_client/cosm_scif_client.c

diff --git a/drivers/misc/mic/cosm_client/Makefile 
b/drivers/misc/mic/cosm_client/Makefile
new file mode 100644
index 000..6f751a5
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/Makefile
@@ -0,0 +1,7 @@
+#
+# Makefile - Intel MIC COSM Client Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += cosm_client.o
+
+cosm_client-objs += cosm_scif_client.o
diff --git a/drivers/misc/mic/cosm_client/cosm_scif_client.c 
b/drivers/misc/mic/cosm_client/cosm_scif_client.c
new file mode 100644
index 000..03e98bf
--- /dev/null
+++ b/drivers/misc/mic/cosm_client/cosm_scif_client.c
@@ -0,0 +1,275 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Client Driver
+ *
+ */
+#include 
+#include 
+#include 
+#include 
+#include "../cosm/cosm_main.h"
+
+#define COSM_SCIF_MAX_RETRIES 10
+#define COSM_HEARTBEAT_SEND_MSEC (COSM_HEARTBEAT_SEND_SEC * MSEC_PER_SEC)
+
+static struct task_struct *client_thread;
+static scif_epd_t client_epd;
+static struct scif_peer_dev *client_spdev;
+
+/*
+ * Reboot notifier: receives shutdown status from the OS and communicates it
+ * back to the COSM process on the host
+ */
+static int cosm_reboot_event(struct notifier_block *this, unsigned long event,
+void *ptr)
+{
+   struct cosm_msg msg = { .id = COSM_MSG_SHUTDOWN_STATUS };
+   int rc;
+
+   event = (event == SYS_RESTART) ? SYSTEM_RESTART : event;
+   dev_info(_spdev->dev, "%s %d received event %ld\n",
+__func__, __LINE__, event);
+
+   msg.shutdown_status = event;
+   rc = scif_send(client_epd, , sizeof(msg), SCIF_SEND_BLOCK);
+   if (rc < 0)
+   dev_err(_spdev->dev, "%s %d scif_send rc %d\n",
+   __func__, __LINE__, rc);
+
+   return NOTIFY_DONE;
+}
+
+static struct notifier_block cosm_reboot = {
+   .notifier_call  = cosm_reboot_event,
+};
+
+/* Set system time from timespec value received from the host */
+static void cosm_set_time(struct cosm_msg *msg)
+{
+   int rc = do_settimeofday64(>timespec);
+
+   if (rc)
+   dev_err(_spdev->dev, "%s: %d settimeofday rc %d\n",
+   __func__, __LINE__, rc);
+}
+
+/* COSM client receive message processing */
+static void cosm_client_recv(void)
+{
+   struct cosm_msg msg;
+   int rc;
+
+   while (1) {
+   rc = scif_recv(client_epd, , sizeof(msg), 0);
+   if (!rc) {
+   return;
+   } else if (rc < 0) {
+   dev_err(_spdev->dev, "%s: %d rc %d\n",
+   __func__, __LINE__, rc);
+   return;
+   }
+
+   dev_dbg(_spdev->dev, "%s: %d rc %d id 0x%llx\n",
+   __func__, __LINE__, rc, msg.id);
+
+   switch (msg.id) {
+   case COSM_MSG_SYNC_TIME:
+   cosm_set_time();
+   break;
+   case COSM_MSG_SHUTDOWN:
+   orderly_poweroff(true);
+   break;
+   default:
+   dev_err(_spdev->dev, "%s: %d unknown id %lld\n",
+   __func__, __LINE__, msg.id);
+   break;
+   }
+   }
+}
+
+/* Initiate connection to the COSM server on the host */
+static in

[PATCH char-misc-next 08/19] misc: mic: Remove COSM functionality from the MIC card driver

[Ashutosh Dixit]

Since card side COSM functionality, to trigger MIC device shutdowns
and communicate shutdown status to the host, is now moved into a
separate COSM client driver, this patch removes this functionality
from the base MIC card driver. The mic_bus driver is also updated to
use the device index provided by COSM rather than maintain its own
device index.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/bus/mic_bus.c | 22 ++
 drivers/misc/mic/card/mic_device.c | 88 ++
 drivers/misc/mic/card/mic_x100.c   |  2 +-
 drivers/misc/mic/common/mic_dev.h  | 13 ++
 include/linux/mic_bus.h|  3 +-
 5 files changed, 23 insertions(+), 105 deletions(-)

diff --git a/drivers/misc/mic/bus/mic_bus.c b/drivers/misc/mic/bus/mic_bus.c
index 961ae90..c64955d 100644
--- a/drivers/misc/mic/bus/mic_bus.c
+++ b/drivers/misc/mic/bus/mic_bus.c
@@ -25,9 +25,6 @@
 #include 
 #include 
 
-/* Unique numbering for mbus devices. */
-static DEFINE_IDA(mbus_index_ida);
-
 static ssize_t device_show(struct device *d,
   struct device_attribute *attr, char *buf)
 {
@@ -147,7 +144,8 @@ static void mbus_release_dev(struct device *d)
 
 struct mbus_device *
 mbus_register_device(struct device *pdev, int id, struct dma_map_ops *dma_ops,
-struct mbus_hw_ops *hw_ops, void __iomem *mmio_va)
+struct mbus_hw_ops *hw_ops, int index,
+void __iomem *mmio_va)
 {
int ret;
struct mbus_device *mbdev;
@@ -166,13 +164,7 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
mbdev->dev.release = mbus_release_dev;
mbdev->hw_ops = hw_ops;
mbdev->dev.bus = _bus;
-
-   /* Assign a unique device index and hence name. */
-   ret = ida_simple_get(_index_ida, 0, 0, GFP_KERNEL);
-   if (ret < 0)
-   goto free_mbdev;
-
-   mbdev->index = ret;
+   mbdev->index = index;
dev_set_name(>dev, "mbus-dev%u", mbdev->index);
/*
 * device_register() causes the bus infrastructure to look for a
@@ -180,10 +172,8 @@ mbus_register_device(struct device *pdev, int id, struct 
dma_map_ops *dma_ops,
 */
ret = device_register(>dev);
if (ret)
-   goto ida_remove;
+   goto free_mbdev;
return mbdev;
-ida_remove:
-   ida_simple_remove(_index_ida, mbdev->index);
 free_mbdev:
kfree(mbdev);
return ERR_PTR(ret);
@@ -192,10 +182,7 @@ EXPORT_SYMBOL_GPL(mbus_register_device);
 
 void mbus_unregister_device(struct mbus_device *mbdev)
 {
-   int index = mbdev->index; /* save for after device release */
-
device_unregister(>dev);
-   ida_simple_remove(_index_ida, index);
 }
 EXPORT_SYMBOL_GPL(mbus_unregister_device);
 
@@ -207,7 +194,6 @@ static int __init mbus_init(void)
 static void __exit mbus_exit(void)
 {
bus_unregister(_bus);
-   ida_destroy(_index_ida);
 }
 
 core_initcall(mbus_init);
diff --git a/drivers/misc/mic/card/mic_device.c 
b/drivers/misc/mic/card/mic_device.c
index 6338908..d0edaf7 100644
--- a/drivers/misc/mic/card/mic_device.c
+++ b/drivers/misc/mic/card/mic_device.c
@@ -37,71 +37,6 @@
 #include "mic_virtio.h"
 
 static struct mic_driver *g_drv;
-static struct mic_irq *shutdown_cookie;
-
-static void mic_notify_host(u8 state)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(state, >shutdown_status);
-   dev_dbg(mdrv->dev, "%s %d system_state %d\n",
-   __func__, __LINE__, state);
-   mic_send_intr(>mdev, ioread8(>c2h_shutdown_db));
-}
-
-static int mic_panic_event(struct notifier_block *this, unsigned long event,
-   void *ptr)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   iowrite8(-1, >h2c_config_db);
-   iowrite8(-1, >h2c_shutdown_db);
-   mic_notify_host(MIC_CRASHED);
-   return NOTIFY_DONE;
-}
-
-static struct notifier_block mic_panic = {
-   .notifier_call  = mic_panic_event,
-};
-
-static irqreturn_t mic_shutdown_isr(int irq, void *data)
-{
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-
-   mic_ack_interrupt(_drv->mdev);
-   if (ioread8(>shutdown_card))
-   orderly_poweroff(true);
-   return IRQ_HANDLED;
-}
-
-static int mic_shutdown_init(void)
-{
-   int rc = 0;
-   struct mic_driver *mdrv = g_drv;
-   struct mic_bootparam __iomem *bootparam = mdrv->dp;
-   int shutdown_db;
-
-   shutdown_db = mic_next_card_db();
-   shutdown_cookie = mic_request_card_irq(mic_shutdown_isr, NULL,
-  "Shutdown", mdrv

[PATCH char-misc-next 16/19] misc: mic: SCIF remote memory map/unmap interface

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the SCIF mmap/munmap interface. A similar
capability is provided to kernel clients via the
scif_get_pages()/scif_put_pages() APIs. The SCIF mmap interface
queries to check if a window is valid and then remaps the local
virtual address to the remote physical pages. These mappings are
subsequently destroyed upon receipt of the VMA close operation or
scif_get_pages().  This functionality allows SCIF users to directly
access remote memory without any driver interaction once the mappings
are created thereby providing bare-metal PCIe latency. These mappings
are zapped to avoid RMA accesses from user space, if a Coprocessor is
reset.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_mmap.c | 711 ++
 1 file changed, 711 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_mmap.c

diff --git a/drivers/misc/mic/scif/scif_mmap.c 
b/drivers/misc/mic/scif/scif_mmap.c
new file mode 100644
index 000..c6c728e
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_mmap.c
@@ -0,0 +1,711 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+
+/*
+ * struct scif_vma_info - Information about a remote memory mapping
+ *   created via scif_mmap(..)
+ * @vma: VM area struct
+ * @list: link to list of active vmas
+ */
+struct scif_vma_info {
+   struct vm_area_struct *vma;
+   struct list_head list;
+};
+
+void scif_recv_munmap(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_rma_req req;
+   struct scif_window *window = NULL;
+   struct scif_window *recv_window =
+   (struct scif_window *)msg->payload[0];
+   struct scif_endpt *ep;
+
+   ep = (struct scif_endpt *)recv_window->ep;
+   req.out_window = 
+   req.offset = recv_window->offset;
+   req.prot = recv_window->prot;
+   req.nr_bytes = recv_window->nr_pages << PAGE_SHIFT;
+   req.type = SCIF_WINDOW_FULL;
+   req.head = >rma_info.reg_list;
+   msg->payload[0] = ep->remote_ep;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Does a valid window exist? */
+   if (scif_query_window()) {
+   dev_err(>sdev->dev,
+   "%s %d -ENXIO\n", __func__, __LINE__);
+   msg->uop = SCIF_UNREGISTER_ACK;
+   goto error;
+   }
+
+   scif_put_window(window, window->nr_pages);
+
+   if (!window->ref_count) {
+   atomic_inc(>rma_info.tw_refcount);
+   ep->rma_info.async_list_del = 1;
+   list_del_init(>list);
+   scif_free_window_offset(ep, window, window->offset);
+   }
+error:
+   mutex_unlock(>rma_info.rma_lock);
+   if (window && !window->ref_count)
+   scif_queue_for_cleanup(window, _info.rma);
+}
+
+/*
+ * Remove valid remote memory mappings created via scif_mmap(..) from the
+ * process address space since the remote node is lost
+ */
+static void __scif_zap_mmaps(struct scif_endpt *ep)
+{
+   struct list_head *item;
+   struct scif_vma_info *info;
+   struct vm_area_struct *vma;
+   unsigned long size;
+
+   spin_lock(>lock);
+   list_for_each(item, >rma_info.vma_list) {
+   info = list_entry(item, struct scif_vma_info, list);
+   vma = info->vma;
+   size = vma->vm_end - vma->vm_start;
+   zap_vma_ptes(vma, vma->vm_start, size);
+   dev_dbg(scif_info.mdev.this_device,
+   "%s ep %p zap vma %p size 0x%lx\n",
+   __func__, ep, info->vma, size);
+   }
+   spin_unlock(>lock);
+}
+
+/*
+ * Traverse the list of endpoints for a particular remote node and
+ * zap valid remote memory mappings since the remote node is lost
+ */
+static void _scif_zap_mmaps(int node, struct list_head *head)
+{
+   struct scif_endpt *ep;
+   struct list_head *item;
+
+   mutex_lock(_info.connlock);
+   list_for_each(item, head) {
+   ep = list_entry(item, struct scif_endpt, list);
+   if (ep->remote_dev->node == node)
+   __scif_zap_mmaps(ep);
+   }
+   mutex_unlock(_info.connlock);
+}
+
+/*
+ * Wrapper for removing remote memory ma

[PATCH char-misc-next 19/19] misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the SCIF kernel node QP control messages required to
enable SCIF RMAs. Examples of such node QP control messages include
registration, unregistration, remote memory allocation requests,
remote memory unmap and SCIF remote fence requests.

The patch also updates the SCIF driver with minor changes required to
enable SCIF RMAs by adding the new files to the build, initializing
RMA specific information during SCIF endpoint creation, reserving SCIF
DMA channels, initializing SCIF RMA specific global data structures,
adding the IOCTL hooks required for SCIF RMAs and updating RMA
specific debugfs hooks.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/Kconfig  |   1 +
 drivers/misc/mic/scif/Makefile|   5 ++
 drivers/misc/mic/scif/scif_api.c  |  21 +
 drivers/misc/mic/scif/scif_debugfs.c  |  85 +-
 drivers/misc/mic/scif/scif_epd.c  |   9 +-
 drivers/misc/mic/scif/scif_epd.h  |  28 ++
 drivers/misc/mic/scif/scif_fd.c   | 159 ++
 drivers/misc/mic/scif/scif_main.c |  19 
 drivers/misc/mic/scif/scif_main.h |  28 +-
 drivers/misc/mic/scif/scif_map.h  |  25 +-
 drivers/misc/mic/scif/scif_nm.c   |  10 ++-
 drivers/misc/mic/scif/scif_nodeqp.c   |  53 +++-
 drivers/misc/mic/scif/scif_nodeqp.h   |  42 -
 drivers/misc/mic/scif/scif_peer_bus.c |  14 +++
 14 files changed, 484 insertions(+), 15 deletions(-)

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index 1488232..e3bf95c 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -75,6 +75,7 @@ comment "SCIF Driver"
 config SCIF
tristate "SCIF Driver"
depends on 64BIT && PCI && X86 && SCIF_BUS
+   select IOVA
help
  This enables SCIF Driver support for the Intel Many Integrated
  Core (MIC) family of PCIe form factor coprocessor devices that
diff --git a/drivers/misc/mic/scif/Makefile b/drivers/misc/mic/scif/Makefile
index bf10bb7..29cfc3e 100644
--- a/drivers/misc/mic/scif/Makefile
+++ b/drivers/misc/mic/scif/Makefile
@@ -13,3 +13,8 @@ scif-objs += scif_epd.o
 scif-objs += scif_rb.o
 scif-objs += scif_nodeqp.o
 scif-objs += scif_nm.o
+scif-objs += scif_dma.o
+scif-objs += scif_fence.o
+scif-objs += scif_mmap.o
+scif-objs += scif_rma.o
+scif-objs += scif_rma_list.o
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index 900c86d..7fd0fe3 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -70,6 +70,7 @@ scif_epd_t scif_open(void)
mutex_init(>sendlock);
mutex_init(>recvlock);
 
+   scif_rma_ep_init(ep);
ep->state = SCIFEP_UNBOUND;
dev_dbg(scif_info.mdev.this_device,
"SCIFAPI open: ep %p success\n", ep);
@@ -184,8 +185,11 @@ int scif_close(scif_epd_t epd)
 
switch (oldstate) {
case SCIFEP_ZOMBIE:
+   dev_err(scif_info.mdev.this_device,
+   "SCIFAPI close: zombie state unexpected\n");
case SCIFEP_DISCONNECTED:
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
/* Remove from the disconnected list */
mutex_lock(_info.connlock);
list_for_each_safe(pos, tmpq, _info.disconnected) {
@@ -207,6 +211,7 @@ int scif_close(scif_epd_t epd)
case SCIFEP_CLOSING:
{
spin_unlock(>lock);
+   scif_unregister_all_windows(epd);
scif_disconnect_ep(ep);
break;
}
@@ -469,6 +474,13 @@ static int scif_conn_func(struct scif_endpt *ep)
struct scifmsg msg;
struct device *spdev;
 
+   err = scif_reserve_dma_chan(ep);
+   if (err) {
+   dev_err(>remote_dev->sdev->dev,
+   "%s %d err %d\n", __func__, __LINE__, err);
+   ep->state = SCIFEP_BOUND;
+   goto connect_error_simple;
+   }
/* Initiate the first part of the endpoint QP setup */
err = scif_setup_qp_connect(ep->qp_info.qp, >qp_info.qp_offset,
SCIF_ENDPT_QP_SIZE, ep->remote_dev);
@@ -804,6 +816,15 @@ retry_connection:
cep->remote_dev = _dev[peer->node];
cep->remote_ep = conreq->msg.payload[0];
 
+   scif_rma_ep_init(cep);
+
+   err = scif_reserve_dma_chan(cep);
+   if (err) {
+   dev_err(scif_info.mdev.this_device,
+   "%s %d err %d\n", __func__, __LINE__, err);
+   goto scif_accept_error_qpalloc;
+   }
+
cep->qp_info.qp = kzalloc(sizeof(*cep->qp_info.qp), GFP_KERNEL);
if (!cep->qp_info.qp) {
err = -ENOMEM;
diff --git a/drivers/misc/m

[PATCH char-misc-next 17/19] misc: mic: SCIF DMA and CPU copy interface

[Ashutosh Dixit]

From: Sudeep Dutt 

SCIF allows users to read from or write to registered remote memory
via CPU copies or DMA. The API verifies that both local and remote
windows are valid before initiating the CPU or DMA transfers. SCIF has
optimized algorithms for handling byte aligned as well as cache line
aligned DMA engines. A registration cache is maintained to avoid the
overhead of pinning pages repeatedly if buffers are reused. The
registration cache is invalidated upon receipt of MMU notifier
callbacks.  SCIF windows are destroyed and the pages are unpinned only
once all prior DMAs initiated using that window are drained. Users can
request synchronous DMA operations as well as tail byte ordering if
required. CPU copies are always performed synchronously.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_dma.c | 1979 ++
 1 file changed, 1979 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_dma.c

diff --git a/drivers/misc/mic/scif/scif_dma.c b/drivers/misc/mic/scif/scif_dma.c
new file mode 100644
index 000..bf750b3
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_dma.c
@@ -0,0 +1,1979 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include "scif_map.h"
+
+/*
+ * struct scif_dma_comp_cb - SCIF DMA completion callback
+ *
+ * @dma_completion_func: DMA completion callback
+ * @cb_cookie: DMA completion callback cookie
+ * @temp_buf: Temporary buffer
+ * @temp_buf_to_free: Temporary buffer to be freed
+ * @is_cache: Is a kmem_cache allocated buffer
+ * @dst_offset: Destination registration offset
+ * @dst_window: Destination registration window
+ * @len: Length of the temp buffer
+ * @temp_phys: DMA address of the temp buffer
+ * @sdev: The SCIF device
+ * @header_padding: padding for cache line alignment
+ */
+struct scif_dma_comp_cb {
+   void (*dma_completion_func)(void *cookie);
+   void *cb_cookie;
+   u8 *temp_buf;
+   u8 *temp_buf_to_free;
+   bool is_cache;
+   s64 dst_offset;
+   struct scif_window *dst_window;
+   size_t len;
+   dma_addr_t temp_phys;
+   struct scif_dev *sdev;
+   int header_padding;
+};
+
+/**
+ * struct scif_copy_work - Work for DMA copy
+ *
+ * @src_offset: Starting source offset
+ * @dst_offset: Starting destination offset
+ * @src_window: Starting src registered window
+ * @dst_window: Starting dst registered window
+ * @loopback: true if this is a loopback DMA transfer
+ * @len: Length of the transfer
+ * @comp_cb: DMA copy completion callback
+ * @remote_dev: The remote SCIF peer device
+ * @fence_type: polling or interrupt based
+ * @ordered: is this a tail byte ordered DMA transfer
+ */
+struct scif_copy_work {
+   s64 src_offset;
+   s64 dst_offset;
+   struct scif_window *src_window;
+   struct scif_window *dst_window;
+   int loopback;
+   size_t len;
+   struct scif_dma_comp_cb   *comp_cb;
+   struct scif_dev *remote_dev;
+   int fence_type;
+   bool ordered;
+};
+
+#ifndef list_entry_next
+#define list_entry_next(pos, member) \
+   list_entry(pos->member.next, typeof(*pos), member)
+#endif
+
+/**
+ * scif_reserve_dma_chan:
+ * @ep: Endpoint Descriptor.
+ *
+ * This routine reserves a DMA channel for a particular
+ * endpoint. All DMA transfers for an endpoint are always
+ * programmed on the same DMA channel.
+ */
+int scif_reserve_dma_chan(struct scif_endpt *ep)
+{
+   int err = 0;
+   struct scif_dev *scifdev;
+   struct scif_hw_dev *sdev;
+   struct dma_chan *chan;
+
+   /* Loopback DMAs are not supported on the management node */
+   if (!scif_info.nodeid && scifdev_self(ep->remote_dev))
+   return 0;
+   if (scif_info.nodeid)
+   scifdev = _dev[0];
+   else
+   scifdev = ep->remote_dev;
+   sdev = scifdev->sdev;
+   if (!sdev->num_dma_ch)
+   return -ENODEV;
+   chan = sdev->dma_ch[scifdev->dma_ch_idx];
+   scifdev->dma_ch_idx = (scifdev->dma_ch_idx + 1) % sdev->num_dma_ch;
+   mutex_lock(>rma_info.rma_lock);
+   ep->rma_info.dma_chan = chan;
+   mutex_unlock(>rma_info.rma_lock);
+   return err;
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+/**
+ * scif_rma_destroy_tcw:
+ *
+ * This routine destroys temporary cached windows
+

[PATCH char-misc-next 18/19] misc: mic: SCIF fence

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the fence APIs required to synchronize
DMAs. SCIF provides an interface to return a "mark" for all DMAs
programmed at the instant the API was called. Users can then "wait" on
the mark provided previously by blocking inside the kernel. Upon
receipt of a DMA completion interrupt the waiting thread is woken
up. There is also an interface to signal DMA completion by polling for
a location to be updated via a "signal" cookie to avoid the interrupt
overhead in the mark/wait interface. SCIF allows programming fences on
both the local and the remote node for both the mark/wait or the fence
signal APIs.

Reviewed-by: Ashutosh Dixit 
Reviewed-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_fence.c | 773 +
 1 file changed, 773 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_fence.c

diff --git a/drivers/misc/mic/scif/scif_fence.c 
b/drivers/misc/mic/scif/scif_fence.c
new file mode 100644
index 000..1620aaa
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_fence.c
@@ -0,0 +1,773 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+
+#include "scif_main.h"
+
+/**
+ * scif_recv_mark: Handle SCIF_MARK request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested a mark.
+ */
+void scif_recv_mark(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   int mark, err;
+
+   err = _scif_fence_mark(ep, );
+   if (err)
+   msg->uop = SCIF_MARK_NACK;
+   else
+   msg->uop = SCIF_MARK_ACK;
+   msg->payload[0] = ep->remote_ep;
+   msg->payload[2] = mark;
+   scif_nodeqp_send(ep->remote_dev, msg);
+}
+
+/**
+ * scif_recv_mark_resp: Handle SCIF_MARK_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_MARK message.
+ */
+void scif_recv_mark_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if (SCIF_MARK_ACK == msg->uop) {
+   fence_req->state = OP_COMPLETED;
+   fence_req->dma_mark = (int)msg->payload[2];
+   } else {
+   fence_req->state = OP_FAILED;
+   }
+   mutex_unlock(>rma_info.rma_lock);
+   complete(_req->comp);
+}
+
+/**
+ * scif_recv_wait: Handle SCIF_WAIT request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested waiting on a fence.
+ */
+void scif_recv_wait(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_remote_fence_info *fence;
+
+   /*
+* Allocate structure for remote fence information and
+* send a NACK if the allocation failed. The peer will
+* return ENOMEM upon receiving a NACK.
+*/
+   fence = kmalloc(sizeof(*fence), GFP_KERNEL);
+   if (!fence) {
+   msg->payload[0] = ep->remote_ep;
+   msg->uop = SCIF_WAIT_NACK;
+   scif_nodeqp_send(ep->remote_dev, msg);
+   return;
+   }
+
+   /* Prepare the fence request */
+   memcpy(>msg, msg, sizeof(struct scifmsg));
+   INIT_LIST_HEAD(>list);
+
+   /* Insert to the global remote fence request list */
+   mutex_lock(_info.fencelock);
+   atomic_inc(>rma_info.fence_refcount);
+   list_add_tail(>list, _info.fence);
+   mutex_unlock(_info.fencelock);
+
+   schedule_work(_info.misc_work);
+}
+
+/**
+ * scif_recv_wait_resp: Handle SCIF_WAIT_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_WAIT message.
+ */
+void scif_recv_wait_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg->payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg->payload[1];
+
+   mutex_lock(>rma_info.rma_lock);
+   if (SCIF_WAIT_ACK == msg->uop)
+   fence_req->state = OP_COMPLETED;
+   else
+   fence_req->state = OP_FAILED;
+   mutex_unlock(>rma_info.rma_lock);
+ 

[PATCH char-misc-next 15/19] misc: mic: SCIF RMA list operations

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the implementation for operations performed on the
list of SCIF windows. Examples of such operations includes adding the
windows to the list of registered (or cached) windows, querying the
list of self or remote windows and unregistering windows. The query
operation is used by SCIF APIs which initiate DMAs, CPU copies or
fences to ensure that a window remains valid during a transfer.

Reviewed-by: Ashutosh Dixit 
Signed-off-by: Nikhil Rao 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma_list.c | 291 ++
 drivers/misc/mic/scif/scif_rma_list.h |  57 +++
 2 files changed, 348 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.c
 create mode 100644 drivers/misc/mic/scif/scif_rma_list.h

diff --git a/drivers/misc/mic/scif/scif_rma_list.c 
b/drivers/misc/mic/scif/scif_rma_list.c
new file mode 100644
index 000..e1ef8da
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma_list.c
@@ -0,0 +1,291 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include "scif_main.h"
+#include 
+#include 
+
+/*
+ * scif_insert_tcw:
+ *
+ * Insert a temp window to the temp registration list sorted by va_for_temp.
+ * RMA lock must be held.
+ */
+void scif_insert_tcw(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL;
+   struct scif_window *prev = list_entry(head, struct scif_window, list);
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   /* Compare with tail and if the entry is new tail add it to the end */
+   if (!list_empty(head)) {
+   curr = list_entry(head->prev, struct scif_window, list);
+   if (curr->va_for_temp < window->va_for_temp) {
+   list_add_tail(>list, head);
+   return;
+   }
+   }
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->va_for_temp > window->va_for_temp)
+   break;
+   prev = curr;
+   }
+   list_add(>list, >list);
+}
+
+/*
+ * scif_insert_window:
+ *
+ * Insert a window to the self registration list sorted by offset.
+ * RMA lock must be held.
+ */
+void scif_insert_window(struct scif_window *window, struct list_head *head)
+{
+   struct scif_window *curr = NULL, *prev = NULL;
+   struct list_head *item;
+
+   INIT_LIST_HEAD(>list);
+   list_for_each(item, head) {
+   curr = list_entry(item, struct scif_window, list);
+   if (curr->offset > window->offset)
+   break;
+   prev = curr;
+   }
+   if (!prev)
+   list_add(>list, head);
+   else
+   list_add(>list, >list);
+   scif_set_window_ref(window, window->nr_pages);
+}
+
+/*
+ * scif_query_tcw:
+ *
+ * Query the temp cached registration list of ep for an overlapping window
+ * in case of permission mismatch, destroy the previous window. if permissions
+ * match and overlap is partial, destroy the window but return the new range
+ * RMA lock must be held.
+ */
+int scif_query_tcw(struct scif_endpt *ep, struct scif_rma_req *req)
+{
+   struct list_head *item, *temp, *head = req->head;
+   struct scif_window *window;
+   u64 start_va_window, start_va_req = req->va_for_temp;
+   u64 end_va_window, end_va_req = start_va_req + req->nr_bytes;
+
+   if (!req->nr_bytes)
+   return -EINVAL;
+   /*
+* Avoid traversing the entire list to find out that there
+* is no entry that matches
+*/
+   if (!list_empty(head)) {
+   window = list_last_entry(head, struct scif_window, list);
+   end_va_window = window->va_for_temp +
+   (window->nr_pages << PAGE_SHIFT);
+   if (start_va_req > end_va_window)
+   return -ENXIO;
+   }
+   list_for_each_safe(item, temp, head) {
+   window = list_entry(item, struct scif_window, list);
+   start_va_window = window->va_for_temp;
+   end_va_window = window->va_for_temp +
+   (window->nr_pages << PAGE_SHIFT);
+   if (start_va_req < start_va_window &&
+   end_va_req < start_va

[PATCH char-misc-next 14/19] misc: mic: SCIF memory registration and unregistration

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch implements the SCIF APIs required to pin and unpin
pages. SCIF registration locks down the pages. It then sends a remote
window allocation request to the peer. Once the peer has allocated
memory, the local SCIF endpoint copies the pinned page information to
the peer and notifies the peer once the copy has complete. The peer
upon receipt of the registration notification adds the new remote
window to its list. At this point the window page information is
available on both self and remote nodes so that they can start
performing SCIF DMAs, CPU copies and fences. The unregistration API
tears down the registration at both self and remote nodes.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma.c | 1761 ++
 1 file changed, 1761 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.c

diff --git a/drivers/misc/mic/scif/scif_rma.c b/drivers/misc/mic/scif/scif_rma.c
new file mode 100644
index 000..d2acb0e
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.c
@@ -0,0 +1,1761 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include 
+#include 
+#include "scif_main.h"
+#include "scif_map.h"
+
+/* Used to skip ulimit checks for registrations with SCIF_MAP_KERNEL flag */
+#define SCIF_MAP_ULIMIT 0x40
+
+bool scif_ulimit_check = 1;
+
+/**
+ * scif_rma_ep_init:
+ * @ep: end point
+ *
+ * Initialize RMA per EP data structures.
+ */
+void scif_rma_ep_init(struct scif_endpt *ep)
+{
+   struct scif_endpt_rma_info *rma = >rma_info;
+
+   mutex_init(>rma_lock);
+   init_iova_domain(>iovad, PAGE_SIZE, SCIF_IOVA_START_PFN,
+SCIF_DMA_64BIT_PFN);
+   spin_lock_init(>tc_lock);
+   mutex_init(>mmn_lock);
+   INIT_LIST_HEAD(>reg_list);
+   INIT_LIST_HEAD(>remote_reg_list);
+   atomic_set(>tw_refcount, 0);
+   atomic_set(>tcw_refcount, 0);
+   atomic_set(>tcw_total_pages, 0);
+   atomic_set(>fence_refcount, 0);
+
+   rma->async_list_del = 0;
+   rma->dma_chan = NULL;
+   INIT_LIST_HEAD(>mmn_list);
+   INIT_LIST_HEAD(>vma_list);
+   init_waitqueue_head(>markwq);
+}
+
+/**
+ * scif_rma_ep_can_uninit:
+ * @ep: end point
+ *
+ * Returns 1 if an endpoint can be uninitialized and 0 otherwise.
+ */
+int scif_rma_ep_can_uninit(struct scif_endpt *ep)
+{
+   int ret = 0;
+
+   mutex_lock(>rma_info.rma_lock);
+   /* Destroy RMA Info only if both lists are empty */
+   if (list_empty(>rma_info.reg_list) &&
+   list_empty(>rma_info.remote_reg_list) &&
+   list_empty(>rma_info.mmn_list) &&
+   !atomic_read(>rma_info.tw_refcount) &&
+   !atomic_read(>rma_info.tcw_refcount) &&
+   !atomic_read(>rma_info.fence_refcount))
+   ret = 1;
+   mutex_unlock(>rma_info.rma_lock);
+   return ret;
+}
+
+/**
+ * scif_create_pinned_pages:
+ * @nr_pages: number of pages in window
+ * @prot: read/write protection
+ *
+ * Allocate and prepare a set of pinned pages.
+ */
+static struct scif_pinned_pages *
+scif_create_pinned_pages(int nr_pages, int prot)
+{
+   struct scif_pinned_pages *pin;
+
+   might_sleep();
+   pin = scif_zalloc(sizeof(*pin));
+   if (!pin)
+   goto error;
+
+   pin->pages = scif_zalloc(nr_pages * sizeof(*pin->pages));
+   if (!pin->pages)
+   goto error_free_pinned_pages;
+
+   pin->prot = prot;
+   pin->magic = SCIFEP_MAGIC;
+   return pin;
+
+error_free_pinned_pages:
+   scif_free(pin, sizeof(*pin));
+error:
+   return NULL;
+}
+
+/**
+ * scif_destroy_pinned_pages:
+ * @pin: A set of pinned pages.
+ *
+ * Deallocate resources for pinned pages.
+ */
+static int scif_destroy_pinned_pages(struct scif_pinned_pages *pin)
+{
+   int j;
+   int writeable = pin->prot & SCIF_PROT_WRITE;
+   int kernel = SCIF_MAP_KERNEL & pin->map_flags;
+
+   for (j = 0; j < pin->nr_pages; j++) {
+   if (pin->pages[j] && !kernel) {
+   if (writeable)
+   SetPageDirty(pin->pages[j]);
+   put_page(pin->pages[j]);
+   }
+   }
+
+   scif_free(pin->pages,
+ p

[PATCH char-misc-next 13/19] misc: mic: SCIF RMA header file

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch adds the internal data structures required to perform SCIF
RMAs. The data structures required to maintain per SCIF endpoint, RMA
information are contained in scif_endpt_rma_info. scif_pinned_pages
describes a set of SCIF pinned pages maintained locally. The
scif_window is a data structure which contains all the fields required
to describe a SCIF registered window on self and remote nodes. It
contains an offset which is used as a key to perform SCIF DMAs and CPU
copies between self and remote registered windows.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 drivers/misc/mic/scif/scif_rma.h | 464 +++
 1 file changed, 464 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_rma.h

diff --git a/drivers/misc/mic/scif/scif_rma.h b/drivers/misc/mic/scif/scif_rma.h
new file mode 100644
index 000..fa67222
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_rma.h
@@ -0,0 +1,464 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above copyright
+ *   notice, this list of conditions and the following disclaimer in
+ *   the documentation and/or other materials provided with the
+ *   distribution.
+ * * Neither the name of Intel Corporation nor the names of its
+ *   contributors may be used to endorse or promote products derived
+ *   from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#ifndef SCIF_RMA_H
+#define SCIF_RMA_H
+
+#include 
+#include 
+
+#include "../bus/scif_bus.h"
+
+/* If this bit is set then the mark is a remote fence mark */
+#define SCIF_REMOTE_FENCE_BIT  31
+/* Magic value used to indicate a remote fence request */
+#define SCIF_REMOTE_FENCE BIT_ULL(SCIF_REMOTE_FENCE_BIT)
+
+#define SCIF_MAX_UNALIGNED_BUF_SIZE (1024 * 1024ULL)
+#define SCIF_KMEM_UNALIGNED_BUF_SIZE (SCIF_MAX_UNALIGNED_BUF_SIZE + \
+ (L1_CACHE_BYTES << 1))
+
+#define SCIF_IOVA_START_PFN(1)
+#define SCIF_IOVA_PFN(addr) ((addr) >> PAGE_SHIFT)
+#define SCIF_DMA_64BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(64))
+#define SCIF_DMA_63BIT_PFN SCIF_IOVA_PFN(DMA_BIT_MASK(63))
+
+/*
+ * struct scif_endpt_rma_info - Per Endpoint Remote Memory Access Information
+ *
+ * @reg_list: List of registration windows for self
+ * @remote_reg_list: List of registration windows for peer
+ * @iovad: Offset generator
+ * @rma_lock: Synchronizes access to self/remote list and also protects the
+ *   window from being destroyed while RMAs are in progress.
+ * @tc_lock: Synchronizes access to temporary cached windows list
+ *  for SCIF Registration Caching.
+ * @mmn_lock: Synchronizes access to the list of MMU notifiers registered
+ * @tw_refcount: Keeps track of number of outstanding temporary registered
+ *  windows created by scif_vreadfrom/scif_vwriteto which have
+ *  not been destroyed.
+ * @tcw_refcount: Same as tw_refcount but for temporary cached windows
+ * @tcw_total_pages: Same as tcw_refcount but in terms of

[PATCH char-misc-next 11/19] dma: Add support to program MIC x100 status descriptiors

[Ashutosh Dixit]

From: Siva Yerramreddy 

The MIC X100 DMA engine has a special status descriptor which writes
an 8 byte value to a destination location.  This is used to signal
completion of all DMA descriptors prior to the status descriptor. This
patch uses the src field as a value since the DMA engine API does not
allow passing a value as an argument, if the length is 8 bytes.  This
special case works since the MIC x100 DMA engine requires cache line
(64 byte) aligned src/dst/len so we can use the 8 byte length as a
special case to enable programming status descriptors without
requiring a change to the DMA engine API.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Siva Yerramreddy 
---
 drivers/dma/mic_x100_dma.c | 44 +++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/drivers/dma/mic_x100_dma.c b/drivers/dma/mic_x100_dma.c
index 74d9db0..8d8acfc 100644
--- a/drivers/dma/mic_x100_dma.c
+++ b/drivers/dma/mic_x100_dma.c
@@ -193,8 +193,16 @@ static void mic_dma_prog_intr(struct mic_dma_chan *ch)
 static int mic_dma_do_dma(struct mic_dma_chan *ch, int flags, dma_addr_t src,
  dma_addr_t dst, size_t len)
 {
-   if (-ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len))
+   if (len && -ENOMEM == mic_dma_prog_memcpy_desc(ch, src, dst, len)) {
return -ENOMEM;
+   } else {
+   /* 3 is the maximum number of status descriptors */
+   int ret = mic_dma_avail_desc_ring_space(ch, 3);
+
+   if (ret < 0)
+   return ret;
+   }
+
/* Above mic_dma_prog_memcpy_desc() makes sure we have enough space */
if (flags & DMA_PREP_FENCE) {
mic_dma_prep_status_desc(>desc_ring[ch->head], 0,
@@ -208,6 +216,21 @@ static int mic_dma_do_dma(struct mic_dma_chan *ch, int 
flags, dma_addr_t src,
return 0;
 }
 
+/* Program a status descriptor with phys as address and value to be written */
+static int mic_dma_do_status_update(struct mic_dma_chan *ch, dma_addr_t phys,
+   u64 value)
+{
+   int ret = mic_dma_avail_desc_ring_space(ch, 4);
+
+   if (ret < 0)
+   return ret;
+   ret = 0;
+   mic_dma_prep_status_desc(>desc_ring[ch->head],
+value, phys, false);
+   mic_dma_hw_ring_inc_head(ch);
+   return ret;
+}
+
 static inline void mic_dma_issue_pending(struct dma_chan *ch)
 {
struct mic_dma_chan *mic_ch = to_mic_dma_chan(ch);
@@ -287,9 +310,28 @@ mic_dma_prep_memcpy_lock(struct dma_chan *ch, dma_addr_t 
dma_dest,
return NULL;
 
spin_lock(_ch->prep_lock);
+   if (len == 8) {
+   /*
+* This is a hack to program status descriptor since
+* DMA engine API doesn't have support for this.
+*/
+   result = mic_dma_do_status_update(mic_ch, dma_dest, dma_src);
+   if (result)
+   goto error;
+
+   result = mic_dma_do_dma(mic_ch, DMA_PREP_FENCE, 0, 0, 0);
+   if (result < 0)
+   goto error;
+   result = mic_dma_do_dma(mic_ch, flags, 0, 0, 0);
+   if (result < 0)
+   goto error;
+   return allocate_tx(mic_ch);
+   }
+
result = mic_dma_do_dma(mic_ch, flags, dma_src, dma_dest, len);
if (result >= 0)
return allocate_tx(mic_ch);
+error:
dev_err(dev, "Error enqueueing dma, error=%d\n", result);
spin_unlock(_ch->prep_lock);
return NULL;
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH char-misc-next 12/19] misc: mic: SCIF RMA header file and IOCTL changes

[Ashutosh Dixit]

From: Sudeep Dutt 

This patch updates the SCIF header file and IOCTL interface with the
changes required to support RMAs.  APIs added include the ability to
pin pages and register those pages with SCIF. SCIF kernel clients can
also add references to remote registered pages and access them via the
CPU. The user space IOCTL interface has been updated to enable SCIF
registration, RDMA/CPU copies and fence APIs for RDMA synchronization.

Reviewed-by: Nikhil Rao 
Reviewed-by: Ashutosh Dixit 
Signed-off-by: Sudeep Dutt 
---
 include/linux/scif.h| 232 ++--
 include/uapi/linux/scif_ioctl.h |  85 +++
 2 files changed, 307 insertions(+), 10 deletions(-)

diff --git a/include/linux/scif.h b/include/linux/scif.h
index fd62c05..bd7b722 100644
--- a/include/linux/scif.h
+++ b/include/linux/scif.h
@@ -93,6 +93,25 @@ enum {
 #define SCIF_PORT_RSVD 1088
 
 typedef struct scif_endpt *scif_epd_t;
+typedef struct scif_pinned_pages *scif_pinned_pages_t;
+
+/**
+ * struct scif_range - SCIF registered range used in kernel mode
+ * @nr_pages: number of pages of PAGE_SIZE
+ * @prot_flags: R/W protection
+ * @phys_addr: Array of bus addresses
+ * @va: Array of kernel virtual addresses backed by the pages in the phys_addr
+ * array. The va is populated only when called on the host for a remote
+ * SCIF connection on MIC. This is required to support the use case of DMA
+ * between MIC and another device which is not a SCIF node e.g., an IB or
+ * ethernet NIC.
+ */
+struct scif_range {
+   int nr_pages;
+   int prot_flags;
+   dma_addr_t *phys_addr;
+   void __iomem **va;
+};
 
 /**
  * struct scif_pollepd - SCIF endpoint to be monitored via scif_poll
@@ -389,7 +408,6 @@ int scif_close(scif_epd_t epd);
  * Errors:
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -442,7 +460,6 @@ int scif_send(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The destination node is returning from a low power state
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - An invalid address was specified for a parameter
  * EINVAL - flags is invalid, or len is negative
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -505,9 +522,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * SCIF_PROT_READ - allow read operations from the window
  * SCIF_PROT_WRITE - allow write operations to the window
  *
- * The map_flags argument can be set to SCIF_MAP_FIXED which interprets a
- * fixed offset.
- *
  * Return:
  * Upon successful completion, scif_register() returns the offset at which the
  * mapping was placed (po); otherwise in user mode SCIF_REGISTER_FAILED (that
@@ -520,7 +534,6 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags);
  * EAGAIN - The mapping could not be performed due to lack of resources
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - map_flags is invalid, or prot_flags is invalid, or SCIF_MAP_FIXED 
is
  * set in flags, and offset is not a multiple of the page size, or addr is not 
a
  * multiple of the page size, or len is not a multiple of the page size, or is
@@ -803,7 +816,6 @@ int scif_writeto(scif_epd_t epd, off_t loffset, size_t len, 
off_t
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -884,7 +896,6 @@ int scif_vreadfrom(scif_epd_t epd, void *addr, size_t len, 
off_t roffset,
  * EACCESS - Attempt to write to a read-only range
  * EBADF, ENOTTY - epd is not a valid endpoint descriptor
  * ECONNRESET - Connection reset by peer
- * EFAULT - Addresses in the range [addr, addr + len - 1] are invalid
  * EINVAL - rma_flags is invalid
  * ENODEV - The remote node is lost or existed, but is not currently in the
  * network since it may have crashed
@@ -1028,11 +1039,212 @@ int scif_fence_signal(scif_epd_t epd, off_t loff, u64 
lval, off_t roff,
  * online nodes in the SCIF network including 'self'; otherwise in user mode
  * -1 is returned and errno is set to indicate the error; in kernel mode no
  * errors are returned.
+ */
+int scif_get_node_ids(u16 *nodes, int len, u16 *self);
+
+/**
+ * scif_pin_pages() - Pin a set of pages
+ * @addr

[PATCH char-misc-next 10/19] lib: convert iova.c into a library

[Ashutosh Dixit]

From: Harish Chegondi 

This patch converts iova.c into a library, moving it from
drivers/iommu/ to lib/, and exports its virtual address allocation and
management functions so that other modules can reuse them.

Cc: Joerg Roedel 
Reviewed-by: Anil S Keshavamurthy 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Harish Chegondi 
---
 drivers/iommu/Kconfig | 5 +
 drivers/iommu/Makefile| 1 -
 lib/Kconfig   | 6 ++
 lib/Makefile  | 2 ++
 {drivers/iommu => lib}/iova.c | 9 +
 5 files changed, 18 insertions(+), 5 deletions(-)
 rename {drivers/iommu => lib}/iova.c (97%)

diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig
index f1fb1d3..c711889 100644
--- a/drivers/iommu/Kconfig
+++ b/drivers/iommu/Kconfig
@@ -41,9 +41,6 @@ config IOMMU_IO_PGTABLE_LPAE_SELFTEST
 
 endmenu
 
-config IOMMU_IOVA
-   bool
-
 config OF_IOMMU
def_bool y
depends on OF && IOMMU_API
@@ -125,8 +122,8 @@ config INTEL_IOMMU
bool "Support for Intel IOMMU using DMA Remapping Devices"
depends on PCI_MSI && ACPI && (X86 || IA64_GENERIC)
select IOMMU_API
-   select IOMMU_IOVA
select DMAR_TABLE
+   select IOVA
help
  DMA remapping (DMAR) devices support enables independent address
  translations for Direct Memory Access (DMA) from devices.
diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile
index c6dcc51..fbb1372 100644
--- a/drivers/iommu/Makefile
+++ b/drivers/iommu/Makefile
@@ -3,7 +3,6 @@ obj-$(CONFIG_IOMMU_API) += iommu-traces.o
 obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o
 obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o
-obj-$(CONFIG_IOMMU_IOVA) += iova.o
 obj-$(CONFIG_OF_IOMMU) += of_iommu.o
 obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o msm_iommu_dev.o
 obj-$(CONFIG_AMD_IOMMU) += amd_iommu.o amd_iommu_init.o
diff --git a/lib/Kconfig b/lib/Kconfig
index 3a2ef67..62ba145 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -62,6 +62,12 @@ config ARCH_USE_CMPXCHG_LOCKREF
 config ARCH_HAS_FAST_MULTIPLIER
bool
 
+config IOVA
+   bool
+   help
+ This option enables a 64 bit virtual address allocation and management
+ library.
+
 config CRC_CCITT
tristate "CRC-CCITT functions"
help
diff --git a/lib/Makefile b/lib/Makefile
index 6897b52..058722d 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -205,3 +205,5 @@ quiet_cmd_build_OID_registry = GEN $@
 clean-files+= oid_registry_data.c
 
 obj-$(CONFIG_UCS2_STRING) += ucs2_string.o
+
+obj-$(CONFIG_IOVA) += iova.o
diff --git a/drivers/iommu/iova.c b/lib/iova.c
similarity index 97%
rename from drivers/iommu/iova.c
rename to lib/iova.c
index b7c3d92..7753006 100644
--- a/drivers/iommu/iova.c
+++ b/lib/iova.c
@@ -72,6 +72,7 @@ init_iova_domain(struct iova_domain *iovad, unsigned long 
granule,
iovad->start_pfn = start_pfn;
iovad->dma_32bit_pfn = pfn_32bit;
 }
+EXPORT_SYMBOL_GPL(init_iova_domain);
 
 static struct rb_node *
 __get_cached_rbnode(struct iova_domain *iovad, unsigned long *limit_pfn)
@@ -281,6 +282,7 @@ alloc_iova(struct iova_domain *iovad, unsigned long size,
 
return new_iova;
 }
+EXPORT_SYMBOL_GPL(alloc_iova);
 
 /**
  * find_iova - find's an iova for a given pfn
@@ -321,6 +323,7 @@ struct iova *find_iova(struct iova_domain *iovad, unsigned 
long pfn)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return NULL;
 }
+EXPORT_SYMBOL_GPL(find_iova);
 
 /**
  * __free_iova - frees the given iova
@@ -339,6 +342,7 @@ __free_iova(struct iova_domain *iovad, struct iova *iova)
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
free_iova_mem(iova);
 }
+EXPORT_SYMBOL_GPL(__free_iova);
 
 /**
  * free_iova - finds and frees the iova for a given pfn
@@ -356,6 +360,7 @@ free_iova(struct iova_domain *iovad, unsigned long pfn)
__free_iova(iovad, iova);
 
 }
+EXPORT_SYMBOL_GPL(free_iova);
 
 /**
  * put_iova_domain - destroys the iova doamin
@@ -378,6 +383,7 @@ void put_iova_domain(struct iova_domain *iovad)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(put_iova_domain);
 
 static int
 __is_range_overlap(struct rb_node *node,
@@ -467,6 +473,7 @@ finish:
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
return iova;
 }
+EXPORT_SYMBOL_GPL(reserve_iova);
 
 /**
  * copy_reserved_iova - copies the reserved between domains
@@ -493,6 +500,7 @@ copy_reserved_iova(struct iova_domain *from, struct 
iova_domain *to)
}
spin_unlock_irqrestore(>iova_rbtree_lock, flags);
 }
+EXPORT_SYMBOL_GPL(copy_reserved_iova);
 
 struct iova *
 split_and_remove_iova(struct iova_domain *iovad, struct iova *iova,
@@ -534,3 +542,4 @@ error:
free_iova_mem(prev);
return NULL;
 }
+EXPORT_SYMBOL_GPL(split_and_remove_iova);
-- 
2.0.0.rc3.2.g998f840

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the 

[PATCH char-misc-next 09/19] misc: mic: Update MIC host daemon with COSM changes

[Ashutosh Dixit]

This patch updates the MIC host daemon to work with corresponding
changes in COSM. Other MIC daemon fixes, cleanups and enhancements as
are also rolled into this patch. Changes to MIC sysfs ABI which go
into effect with this patch are also documented.

Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 Documentation/ABI/testing/sysfs-class-mic.txt |  29 ++-
 Documentation/mic/mic_overview.txt|  31 ++-
 Documentation/mic/mpssd/mpss  |   4 +-
 Documentation/mic/mpssd/mpssd.c   | 362 --
 Documentation/mic/mpssd/mpssd.h   |   1 +
 drivers/misc/mic/Makefile |   4 +
 drivers/misc/mic/bus/scif_bus.c   |   7 +-
 drivers/misc/mic/bus/scif_bus.h   |   6 +-
 include/uapi/linux/mic_common.h   |  16 +-
 9 files changed, 285 insertions(+), 175 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-class-mic.txt 
b/Documentation/ABI/testing/sysfs-class-mic.txt
index 13f48af..d45eed2 100644
--- a/Documentation/ABI/testing/sysfs-class-mic.txt
+++ b/Documentation/ABI/testing/sysfs-class-mic.txt
@@ -41,18 +41,15 @@ Description:
When read, this entry provides the current state of an Intel
MIC device in the context of the card OS. Possible values that
will be read are:
-   "offline" - The MIC device is ready to boot the card OS. On
+   "ready" - The MIC device is ready to boot the card OS. On
reading this entry after an OSPM resume, a "boot" has to be
written to this entry if the card was previously shutdown
during OSPM suspend.
-   "online" - The MIC device has initiated booting a card OS.
+   "booting" - The MIC device has initiated booting a card OS.
+   "online" - The MIC device has completed boot and is online
"shutting_down" - The card OS is shutting down.
+   "resetting" - A reset has been initiated for the MIC device
"reset_failed" - The MIC device has failed to reset.
-   "suspending" - The MIC device is currently being prepared for
-   suspend. On reading this entry, a "suspend" has to be written
-   to the state sysfs entry to ensure the card is shutdown during
-   OSPM suspend.
-   "suspended" - The MIC device has been suspended.
 
When written, this sysfs entry triggers different state change
operations depending upon the current state of the card OS.
@@ -62,8 +59,6 @@ Description:
sysfs entries.
"reset" - Initiates device reset.
"shutdown" - Initiates card OS shutdown.
-   "suspend" - Initiates card OS shutdown and also marks the card
-   as suspended.
 
 What:  /sys/class/mic/mic(x)/shutdown_status
 Date:  October 2013
@@ -126,7 +121,7 @@ Description:
the card. This sysfs entry can be written with the following
valid strings:
a) linux - Boot a Linux image.
-   b) elf - Boot an elf image for flash updates.
+   b) flash - Boot an image for flash updates.
 
 What:  /sys/class/mic/mic(x)/log_buf_addr
 Date:  October 2013
@@ -155,3 +150,17 @@ Description:
daemon to set the log buffer length address. The correct log
buffer length address to be written can be found in the
System.map file of the card OS.
+
+What:  /sys/class/mic/mic(x)/heartbeat_enable
+Date:  March 2015
+KernelVersion: 3.20
+Contact:   Ashutosh Dixit 
+Description:
+   The MIC drivers detect and inform user space about card crashes
+   via a heartbeat mechanism (see the description of
+   shutdown_status above). User space can turn off this
+   notification by setting heartbeat_enable to 0 and enable it by
+   setting this entry to 1. If this notification is disabled it is
+   the responsibility of user space to detect card crashes via
+   alternative means such as a network ping. This setting is
+   enabled by default.
diff --git a/Documentation/mic/mic_overview.txt 
b/Documentation/mic/mic_overview.txt
index 1a2f2c8..73f44fc 100644
--- a/Documentation/mic/mic_overview.txt
+++ b/Documentation/mic/mic_overview.txt
@@ -28,6 +28,10 @@ The Symmetric Communication Interface (SCIF (pronounced as 
skiff)) is a
 low level communications API across PCIe currently implemented for MIC.
 More details are available at scif_overview.txt.
 
+The Coprocessor State Management (COSM) driver on 

[PATCH char-misc-next 07/19] misc: mic: Remove COSM functionality from the MIC host driver

[Ashutosh Dixit]

Since COSM functionality is now moved into a separate COSM driver
drivers, this patch removes this functionality from the base MIC host
driver. The MIC host driver now implements cosm_hw_ops and registers a
COSM device which allows the COSM driver to trigger
boot/shutdown/reset of the MIC devices via the cosm_hw_ops.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/Makefile   |   2 -
 drivers/misc/mic/host/Makefile  |   1 -
 drivers/misc/mic/host/mic_boot.c| 317 +++--
 drivers/misc/mic/host/mic_debugfs.c | 114 +
 drivers/misc/mic/host/mic_device.h  |  88 +--
 drivers/misc/mic/host/mic_fops.c|   4 +-
 drivers/misc/mic/host/mic_intr.c|  46 ++--
 drivers/misc/mic/host/mic_main.c| 223 +++---
 drivers/misc/mic/host/mic_smpt.c|  30 ++-
 drivers/misc/mic/host/mic_sysfs.c   | 459 
 drivers/misc/mic/host/mic_virtio.c  |  17 +-
 drivers/misc/mic/host/mic_virtio.h  |   2 +-
 drivers/misc/mic/host/mic_x100.c|  46 ++--
 13 files changed, 204 insertions(+), 1145 deletions(-)
 delete mode 100644 drivers/misc/mic/host/mic_sysfs.c

diff --git a/drivers/misc/mic/Makefile b/drivers/misc/mic/Makefile
index a74042c..a2e75f4 100644
--- a/drivers/misc/mic/Makefile
+++ b/drivers/misc/mic/Makefile
@@ -2,7 +2,5 @@
 # Makefile - Intel MIC Linux driver.
 # Copyright(c) 2013, Intel Corporation.
 #
-obj-$(CONFIG_INTEL_MIC_HOST) += host/
-obj-$(CONFIG_INTEL_MIC_CARD) += card/
 obj-y += bus/
 obj-$(CONFIG_SCIF) += scif/
diff --git a/drivers/misc/mic/host/Makefile b/drivers/misc/mic/host/Makefile
index c2197f9..004d3db 100644
--- a/drivers/misc/mic/host/Makefile
+++ b/drivers/misc/mic/host/Makefile
@@ -5,7 +5,6 @@
 obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
 mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
-mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_intr.o
 mic_host-objs += mic_boot.o
diff --git a/drivers/misc/mic/host/mic_boot.c b/drivers/misc/mic/host/mic_boot.c
index e5f6a5e..7845564 100644
--- a/drivers/misc/mic/host/mic_boot.c
+++ b/drivers/misc/mic/host/mic_boot.c
@@ -22,9 +22,9 @@
 #include 
 #include 
 #include 
-
 #include 
 #include 
+#include "../bus/scif_bus.h"
 #include "../common/mic_dev.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
@@ -99,7 +99,7 @@ static int __mic_dma_map_sg(struct device *dev, struct 
scatterlist *sg,
int i, j, ret;
dma_addr_t da;
 
-   ret = dma_map_sg(mdev->sdev->parent, sg, nents, dir);
+   ret = dma_map_sg(>pdev->dev, sg, nents, dir);
if (ret <= 0)
return 0;
 
@@ -115,7 +115,7 @@ err:
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = mic_to_dma_addr(mdev, sg_dma_address(s));
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
return 0;
 }
 
@@ -135,7 +135,7 @@ static void __mic_dma_unmap_sg(struct device *dev,
mic_unmap(mdev, sg_dma_address(s), s->length);
sg_dma_address(s) = da;
}
-   dma_unmap_sg(mdev->sdev->parent, sg, nents, dir);
+   dma_unmap_sg(>pdev->dev, sg, nents, dir);
 }
 
 static struct dma_map_ops __mic_dma_ops = {
@@ -270,48 +270,13 @@ static struct mbus_hw_ops mbus_hw_ops = {
.ack_interrupt = _mic_ack_interrupt,
 };
 
-/**
- * mic_reset - Reset the MIC device.
- * @mdev: pointer to mic_device instance
- */
-static void mic_reset(struct mic_device *mdev)
-{
-   int i;
-
-#define MIC_RESET_TO (45)
-
-   reinit_completion(>reset_wait);
-   mdev->ops->reset_fw_ready(mdev);
-   mdev->ops->reset(mdev);
-
-   for (i = 0; i < MIC_RESET_TO; i++) {
-   if (mdev->ops->is_fw_ready(mdev))
-   goto done;
-   /*
-* Resets typically take 10s of seconds to complete.
-* Since an MMIO read is required to check if the
-* firmware is ready or not, a 1 second delay works nicely.
-*/
-   msleep(1000);
-   }
-   mic_set_state(mdev, MIC_RESET_FAILED);
-done:
-   complete_all(>reset_wait);
-}
-
 /* Initialize the MIC bootparams */
 void mic_bootparam_init(struct mic_device *mdev)
 {
struct mic_bootparam *bootparam = mdev->dp;
 
bootparam->magic = cpu_to_le32(MIC_MAGIC);
-   bootparam->c2h_shutdown_db = mdev->shutdown_db;
-   bootparam->h2c_shutdown_db = -1;
bootparam->h2c_config_db = -1;
-   bootparam->shutdown_status = 0;
-   bootparam->shutdown_card = 0;
-   /* Total nodes = number of MICs + 1 for self node */
-   bootparam->tot_nodes = atomic_read(_num_mics) + 1;
bootparam-&

[PATCH char-misc-next 05/19] misc: mic: COSM SCIF server

[Ashutosh Dixit]

The COSM driver communicates with the MIC cards over SCIF. A SCIF
"server" listens for incoming connections from "client" MIC cards as
they boot. After the connection is accepted a separate work item is
scheduled for each MIC card. This work item normally stays blocked in
scif_poll but wakes up to process messages from the card.

The SCIF connection between the host and card COSM components is used
to (a) send the command to shut down the card (b) receive shutdown
status back from the card upon completion of shutdown (c) receive
periodic heartbeat messages to detect card crashes (d) send host time
to the card to enable the card to sync its time to the host.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm/cosm_scif_server.c | 405 +++
 1 file changed, 405 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/cosm_scif_server.c

diff --git a/drivers/misc/mic/cosm/cosm_scif_server.c 
b/drivers/misc/mic/cosm/cosm_scif_server.c
new file mode 100644
index 000..62a73fb
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_scif_server.c
@@ -0,0 +1,405 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+#include 
+#include "cosm_main.h"
+
+/*
+ * The COSM driver uses SCIF to communicate between the management node and the
+ * MIC cards. SCIF is used to (a) Send a shutdown command to the card (b)
+ * receive a shutdown status back from the card upon completion of shutdown and
+ * (c) receive periodic heartbeat messages from the card used to deduce if the
+ * card has crashed.
+ *
+ * A COSM server consisting of a SCIF listening endpoint waits for incoming
+ * connections from the card. Upon acceptance of the connection, a separate
+ * work-item is scheduled to handle SCIF message processing for that card. The
+ * life-time of this work-item is therefore the time from which the connection
+ * from a card is accepted to the time at which the connection is closed. A new
+ * work-item starts each time the card boots and is alive till the card (a)
+ * shuts down (b) is reset (c) crashes (d) cosm_client driver on the card is
+ * unloaded.
+ *
+ * From the point of view of COSM interactions with SCIF during card
+ * shutdown, reset and crash are as follows:
+ *
+ * Card shutdown
+ * -
+ * 1. COSM client on the card invokes orderly_poweroff() in response to 
SHUTDOWN
+ *message from the host.
+ * 2. Card driver shutdown callback invokes scif_unregister_device(..) 
resulting
+ *in scif_remove(..) getting called on the card
+ * 3. scif_remove -> scif_stop -> scif_handle_remove_node ->
+ *scif_peer_unregister_device -> device_unregister for the host peer device
+ * 4. During device_unregister remove(..) method of cosm_client is invoked 
which
+ *closes the COSM SCIF endpoint on the card. This results in a SCIF_DISCNCT
+ *message being sent to host SCIF. SCIF_DISCNCT message processing on the
+ *host SCIF sets the host COSM SCIF endpoint state to DISCONNECTED and 
wakes
+ *up the host COSM thread blocked in scif_poll(..) resulting in
+ *scif_poll(..)  returning POLLHUP.
+ * 5. On the card, scif_peer_release_dev is next called which results in an
+ *SCIF_EXIT message being sent to the host and after receiving the
+ *SCIF_EXIT_ACK from the host the peer device teardown on the card is
+ *complete.
+ * 6. As part of the SCIF_EXIT message processing on the host, host sends a
+ *SCIF_REMOVE_NODE to itself corresponding to the card being removed. This
+ *starts a similar SCIF peer device teardown sequence on the host
+ *corresponding to the card being shut down.
+ *
+ * Card reset
+ * --
+ * The case of interest here is when the card has not been previously shut down
+ * since most of the steps below are skipped in that case:
+
+ * 1. cosm_stop(..) invokes hw_ops->stop(..) method of the base PCIe driver
+ *which unregisters the SCIF HW device resulting in scif_remove(..) being
+ *called on the host.
+ * 2. scif_remove(..) calls scif_disconnect_node(..) which results in a
+ *SCIF_EXIT message being sent to the card.
+ * 3. The card executes scif_stop() as part of SCIF_EXIT message
+ *processing. This results in the COSM endpoint on the 

[PATCH char-misc-next 04/19] misc: mic: Coprocessor State Management (COSM) driver

[Ashutosh Dixit]

The COSM driver allows boot, shutdown and reset of Intel MIC devices
via sysfs. This functionality was previously present in the Intel MIC
host driver but has now been taken out into a separate driver so that
it can be shared between multiple generations of Intel MIC products.
The sysfs kernel ABI used by the COSM driver is the same as that
defined originally for the MIC host driver in
Documentation/ABI/testing/sysfs-class-mic.txt.

The COSM driver also contains support for dumping the MIC card log_buf
and doing a "force reset" for the card via debugfs. The OSPM support
present in the MIC host driver has now largely been moved to user
space and only a small required OSPM functionality is now present in
the driver.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Dasaratharaman Chandramouli 

Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/cosm/Makefile   |  10 +
 drivers/misc/mic/cosm/cosm_debugfs.c | 155 
 drivers/misc/mic/cosm/cosm_main.c| 388 +
 drivers/misc/mic/cosm/cosm_main.h|  70 ++
 drivers/misc/mic/cosm/cosm_sysfs.c   | 461 +++
 5 files changed, 1084 insertions(+)
 create mode 100644 drivers/misc/mic/cosm/Makefile
 create mode 100644 drivers/misc/mic/cosm/cosm_debugfs.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.c
 create mode 100644 drivers/misc/mic/cosm/cosm_main.h
 create mode 100644 drivers/misc/mic/cosm/cosm_sysfs.c

diff --git a/drivers/misc/mic/cosm/Makefile b/drivers/misc/mic/cosm/Makefile
new file mode 100644
index 000..b85d4d4
--- /dev/null
+++ b/drivers/misc/mic/cosm/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile - Intel MIC Coprocessor State Management (COSM) Driver
+# Copyright(c) 2015, Intel Corporation.
+#
+obj-$(CONFIG_MIC_COSM) += mic_cosm.o
+
+mic_cosm-objs := cosm_main.o
+mic_cosm-objs += cosm_debugfs.o
+mic_cosm-objs += cosm_sysfs.o
+mic_cosm-objs += cosm_scif_server.o
diff --git a/drivers/misc/mic/cosm/cosm_debugfs.c 
b/drivers/misc/mic/cosm/cosm_debugfs.c
new file mode 100644
index 000..30d953d
--- /dev/null
+++ b/drivers/misc/mic/cosm/cosm_debugfs.c
@@ -0,0 +1,155 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Coprocessor State Management (COSM) Driver
+ *
+ */
+
+#include 
+#include 
+#include "cosm_main.h"
+
+/* Debugfs parent dir */
+static struct dentry *cosm_dbg;
+
+/**
+ * cosm_log_buf_show - Display MIC kernel log buffer
+ *
+ * log_buf addr/len is read from System.map by user space
+ * and populated in sysfs entries.
+ */
+static int cosm_log_buf_show(struct seq_file *s, void *unused)
+{
+   void __iomem *log_buf_va;
+   int __iomem *log_buf_len_va;
+   struct cosm_device *cdev = s->private;
+   void *kva;
+   int size;
+   u64 aper_offset;
+
+   if (!cdev || !cdev->log_buf_addr || !cdev->log_buf_len)
+   goto done;
+
+   mutex_lock(>cosm_mutex);
+   switch (cdev->state) {
+   case MIC_BOOTING:
+   case MIC_ONLINE:
+   case MIC_SHUTTING_DOWN:
+   break;
+   default:
+   goto unlock;
+   }
+
+   /*
+* Card kernel will never be relocated and any kernel text/data mapping
+* can be translated to phys address by subtracting __START_KERNEL_map.
+*/
+   aper_offset = (u64)cdev->log_buf_len - __START_KERNEL_map;
+   log_buf_len_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+   aper_offset = (u64)cdev->log_buf_addr - __START_KERNEL_map;
+   log_buf_va = cdev->hw_ops->aper(cdev)->va + aper_offset;
+
+   size = ioread32(log_buf_len_va);
+   kva = kmalloc(size, GFP_KERNEL);
+   if (!kva)
+   goto unlock;
+
+   memcpy_fromio(kva, log_buf_va, size);
+   seq_write(s, kva, size);
+   kfree(kva);
+unlock:
+   mutex_unlock(>cosm_mutex);
+done:
+   return 0;
+}
+
+static int cosm_log_buf_open(struct inode *inode, struct file *file)
+{
+   return single_open(file, cosm_log_buf_show, inode->i_private);
+}
+
+static const struct file_operations log_buf_ops = {
+   .owner   = THIS_MODULE,
+   .open= cosm_log_buf_open,
+   .read= seq_read,
+   .llseek  = seq_lseek,
+   .release = single_release
+};
+
+/**
+ * cosm_force_reset_show - Force MIC reset
+ *
+ * Invokes the f

[PATCH char-misc-next 03/19] misc: mic: MIC COSM bus

[Ashutosh Dixit]

The MIC COSM bus allows the co-processor state management (COSM)
functionality to be shared between multiple generations of Intel MIC
products. The COSM driver registers itself on the COSM bus. The base
PCIe drivers implement the bus ops and register COSM devices on the
bus, resulting in the COSM driver being probed with the COSM devices.
COSM bus ops, e.g. start, stop, ready, reset, therefore abstract out
common functionality from its specific implementation for individual
generations of MIC products.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/Kconfig|  22 ++-
 drivers/misc/mic/bus/Makefile   |   1 +
 drivers/misc/mic/bus/cosm_bus.c | 141 
 drivers/misc/mic/bus/cosm_bus.h | 134 ++
 4 files changed, 296 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/mic/bus/cosm_bus.c
 create mode 100644 drivers/misc/mic/bus/cosm_bus.h

diff --git a/drivers/misc/mic/Kconfig b/drivers/misc/mic/Kconfig
index e9f2f56..1488232 100644
--- a/drivers/misc/mic/Kconfig
+++ b/drivers/misc/mic/Kconfig
@@ -36,7 +36,7 @@ comment "Intel MIC Host Driver"
 
 config INTEL_MIC_HOST
tristate "Intel MIC Host Driver"
-   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && PCI && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VHOST_RING
help
  This enables Host Driver support for the Intel Many Integrated
@@ -56,7 +56,7 @@ comment "Intel MIC Card Driver"
 
 config INTEL_MIC_CARD
tristate "Intel MIC Card Driver"
-   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS
+   depends on 64BIT && X86 && INTEL_MIC_BUS && SCIF_BUS && MIC_COSM
select VIRTIO
help
  This enables card driver support for the Intel Many Integrated
@@ -88,3 +88,21 @@ config SCIF
  More information about the Intel MIC family as well as the Linux
  OS and tools for MIC to use with this driver are available from
  <http://software.intel.com/en-us/mic-developer>.
+
+comment "Intel MIC Coprocessor State Management (COSM) Drivers"
+
+config MIC_COSM
+   tristate "Intel MIC Coprocessor State Management (COSM) Drivers"
+   depends on 64BIT && PCI && X86 && SCIF
+   help
+ This enables COSM driver support for the Intel Many
+ Integrated Core (MIC) family of PCIe form factor coprocessor
+ devices. COSM drivers implement functions such as boot,
+ shutdown, reset and reboot of MIC devices.
+
+ If you are building a host kernel with an Intel MIC device then
+ say M (recommended) or Y, else say N. If unsure say N.
+
+ More information about the Intel MIC family as well as the Linux
+ OS and tools for MIC to use with this driver are available from
+ <http://software.intel.com/en-us/mic-developer>.
diff --git a/drivers/misc/mic/bus/Makefile b/drivers/misc/mic/bus/Makefile
index 1ed37e2..761842b 100644
--- a/drivers/misc/mic/bus/Makefile
+++ b/drivers/misc/mic/bus/Makefile
@@ -4,3 +4,4 @@
 #
 obj-$(CONFIG_INTEL_MIC_BUS) += mic_bus.o
 obj-$(CONFIG_SCIF_BUS) += scif_bus.o
+obj-$(CONFIG_MIC_COSM) += cosm_bus.o
diff --git a/drivers/misc/mic/bus/cosm_bus.c b/drivers/misc/mic/bus/cosm_bus.c
new file mode 100644
index 000..1e36830
--- /dev/null
+++ b/drivers/misc/mic/bus/cosm_bus.c
@@ -0,0 +1,141 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC COSM Bus Driver
+ */
+#include 
+#include 
+#include 
+#include "cosm_bus.h"
+
+/* Unique numbering for cosm devices. */
+static DEFINE_IDA(cosm_index_ida);
+
+static int cosm_dev_probe(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+   return drv->probe(dev);
+}
+
+static int cosm_dev_remove(struct device *d)
+{
+   struct cosm_device *dev = dev_to_cosm(d);
+   struct cosm_driver *drv = drv_to_cosm(dev->dev.driver);
+
+   drv->remove(dev);
+   return 0;
+}
+
+static struct bus_type cosm_bus = {
+

[PATCH char-misc-next 02/19] misc: mic: Add support for kernel mode SCIF clients

[Ashutosh Dixit]

Add support for registration/de-registration of kernel mode SCIF
clients. SCIF clients are probed with new and existing SCIF peer
devices. Similarly the client remove method is called when SCIF
peer devices are removed.

Changes to SCIF peer device framework necessitated by supporting
kernel mode SCIF clients are also included in this patch.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/scif/scif_api.c  |  44 +
 drivers/misc/mic/scif/scif_main.c |  88 --
 drivers/misc/mic/scif/scif_main.h |   5 +-
 drivers/misc/mic/scif/scif_nm.c   |  10 +--
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +-
 drivers/misc/mic/scif/scif_peer_bus.c | 165 +-
 drivers/misc/mic/scif/scif_peer_bus.h |  42 +
 include/linux/scif.h  |  58 
 8 files changed, 256 insertions(+), 221 deletions(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index bf2d70f..900c86d 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -1430,3 +1430,47 @@ int scif_get_node_ids(u16 *nodes, int len, u16 *self)
return online;
 }
 EXPORT_SYMBOL_GPL(scif_get_node_ids);
+
+static int scif_add_client_dev(struct device *dev, struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->probe)
+   client->probe(spdev);
+   return 0;
+}
+
+static int scif_remove_client_dev(struct device *dev,
+ struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client->remove)
+   client->remove(spdev);
+   return 0;
+}
+
+void scif_client_unregister(struct scif_client *client)
+{
+   subsys_interface_unregister(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_unregister);
+
+int scif_client_register(struct scif_client *client)
+{
+   struct subsys_interface *si = >si;
+
+   si->name = client->name;
+   si->subsys = _peer_bus;
+   si->add_dev = scif_add_client_dev;
+   si->remove_dev = scif_remove_client_dev;
+
+   return subsys_interface_register(>si);
+}
+EXPORT_SYMBOL_GPL(scif_client_register);
diff --git a/drivers/misc/mic/scif/scif_main.c 
b/drivers/misc/mic/scif/scif_main.c
index 6ce851f..f90bd06 100644
--- a/drivers/misc/mic/scif/scif_main.c
+++ b/drivers/misc/mic/scif/scif_main.c
@@ -80,35 +80,6 @@ irqreturn_t scif_intr_handler(int irq, void *data)
return IRQ_HANDLED;
 }
 
-static int scif_peer_probe(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   mutex_lock(_info.conflock);
-   scif_info.total++;
-   scif_info.maxid = max_t(u32, spdev->dnode, scif_info.maxid);
-   mutex_unlock(_info.conflock);
-   rcu_assign_pointer(scifdev->spdev, spdev);
-
-   /* In the future SCIF kernel client devices will be added here */
-   return 0;
-}
-
-static void scif_peer_remove(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = _dev[spdev->dnode];
-
-   /* In the future SCIF kernel client devices will be removed here */
-   spdev = rcu_dereference(scifdev->spdev);
-   if (spdev)
-   RCU_INIT_POINTER(scifdev->spdev, NULL);
-   synchronize_rcu();
-
-   mutex_lock(_info.conflock);
-   scif_info.total--;
-   mutex_unlock(_info.conflock);
-}
-
 static void scif_qp_setup_handler(struct work_struct *work)
 {
struct scif_dev *scifdev = container_of(work, struct scif_dev,
@@ -139,20 +110,13 @@ static void scif_qp_setup_handler(struct work_struct 
*work)
}
 }
 
-static int scif_setup_scifdev(struct scif_hw_dev *sdev)
+static int scif_setup_scifdev(void)
 {
+   /* We support a maximum of 129 SCIF nodes including the mgmt node */
+#define MAX_SCIF_NODES 129
int i;
-   u8 num_nodes;
+   u8 num_nodes = MAX_SCIF_NODES;
 
-   if (sdev->snode) {
-   struct mic_bootparam __iomem *bp = sdev->rdp;
-
-   num_nodes = ioread8(>tot_nodes);
-   } else {
-   struct mic_bootparam *bp = sdev->dp;
-
-   num_nodes = bp->tot_nodes;
-   }
scif_dev = kcalloc(num_nodes, sizeof(*scif_dev), GFP_KERNEL);
if (!scif_dev)
return -ENOMEM;
@@ -163,7 +127,7 @@ static int scif_setup_scifdev(struct scif_hw_dev *sdev)
scifdev->exit = OP_IDLE;
init_waitqueue_head(>disconn_wq);
mutex_init(>lock);
-   INIT_WORK(>init_msg_work, scif

[PATCH char-misc-next 01/19] misc: mic: SCIF poll

[Ashutosh Dixit]

SCIF poll allows both user and kernel mode clients to wait on
events on a SCIF endpoint. These events include availability of
space or data in the SCIF ring buffer, availability of connection
requests on a listening endpoint and completion of connections
when using async connects.

Reviewed-by: Nikhil Rao 
Reviewed-by: Sudeep Dutt 
Signed-off-by: Ashutosh Dixit 
---
 drivers/misc/mic/scif/scif_api.c  | 158 +-
 drivers/misc/mic/scif/scif_epd.h  |  22 ++
 drivers/misc/mic/scif/scif_fd.c   |   9 +++
 drivers/misc/mic/scif/scif_main.h |   2 +
 include/linux/scif.h  |  74 ++
 5 files changed, 264 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index f39d313..bf2d70f 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -37,9 +37,21 @@ enum conn_async_state {
ASYNC_CONN_FLUSH_WORK   /* async work flush in progress  */
 };
 
+/*
+ * File operations for anonymous inode file associated with a SCIF endpoint,
+ * used in kernel mode SCIF poll. Kernel mode SCIF poll calls portions of the
+ * poll API in the kernel and these take in a struct file *. Since a struct
+ * file is not available to kernel mode SCIF, it uses an anonymous file for
+ * this purpose.
+ */
+const struct file_operations scif_anon_fops = {
+   .owner = THIS_MODULE,
+};
+
 scif_epd_t scif_open(void)
 {
struct scif_endpt *ep;
+   int err;
 
might_sleep();
ep = kzalloc(sizeof(*ep), GFP_KERNEL);
@@ -50,6 +62,10 @@ scif_epd_t scif_open(void)
if (!ep->qp_info.qp)
goto err_qp_alloc;
 
+   err = scif_anon_inode_getfile(ep);
+   if (err)
+   goto err_anon_inode;
+
spin_lock_init(>lock);
mutex_init(>sendlock);
mutex_init(>recvlock);
@@ -59,6 +75,8 @@ scif_epd_t scif_open(void)
"SCIFAPI open: ep %p success\n", ep);
return ep;
 
+err_anon_inode:
+   kfree(ep->qp_info.qp);
 err_qp_alloc:
kfree(ep);
 err_ep_alloc:
@@ -279,6 +297,7 @@ int scif_close(scif_epd_t epd)
}
}
scif_put_port(ep->port.port);
+   scif_anon_inode_fput(ep);
scif_teardown_ep(ep);
scif_add_epd_to_zombie_list(ep, !SCIF_EPLOCK_HELD);
return 0;
@@ -558,8 +577,10 @@ void scif_conn_handler(struct work_struct *work)
list_del(>conn_list);
}
spin_unlock(_info.nb_connect_lock);
-   if (ep)
+   if (ep) {
ep->conn_err = scif_conn_func(ep);
+   wake_up_interruptible(>conn_pend_wq);
+   }
} while (ep);
 }
 
@@ -660,6 +681,7 @@ int __scif_connect(scif_epd_t epd, struct scif_port_id 
*dst, bool non_block)
ep->remote_dev = _dev[dst->node];
ep->qp_info.qp->magic = SCIFEP_MAGIC;
if (ep->conn_async_state == ASYNC_CONN_INPROGRESS) {
+   init_waitqueue_head(>conn_pend_wq);
spin_lock(_info.nb_connect_lock);
list_add_tail(>conn_list, _info.nb_connect_list);
spin_unlock(_info.nb_connect_lock);
@@ -788,6 +810,10 @@ retry_connection:
goto scif_accept_error_qpalloc;
}
 
+   err = scif_anon_inode_getfile(cep);
+   if (err)
+   goto scif_accept_error_anon_inode;
+
cep->qp_info.qp->magic = SCIFEP_MAGIC;
spdev = scif_get_peer_dev(cep->remote_dev);
if (IS_ERR(spdev)) {
@@ -858,6 +884,8 @@ retry:
spin_unlock(>lock);
return 0;
 scif_accept_error_map:
+   scif_anon_inode_fput(cep);
+scif_accept_error_anon_inode:
scif_teardown_ep(cep);
 scif_accept_error_qpalloc:
kfree(cep);
@@ -1247,6 +1275,134 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags)
 }
 EXPORT_SYMBOL_GPL(scif_recv);
 
+static inline void _scif_poll_wait(struct file *f, wait_queue_head_t *wq,
+  poll_table *p, struct scif_endpt *ep)
+{
+   /*
+* Because poll_wait makes a GFP_KERNEL allocation, give up the lock
+* and regrab it afterwards. Because the endpoint state might have
+* changed while the lock was given up, the state must be checked
+* again after re-acquiring the lock. The code in __scif_pollfd(..)
+* does this.
+*/
+   spin_unlock(>lock);
+   poll_wait(f, wq, p);
+   spin_lock(>lock);
+}
+
+unsigned int
+__scif_pollfd(struct file *f, poll_table *wait, struct scif_endpt *ep)
+{
+   unsigned int mask = 0;
+
+   dev_dbg(scif_info.mdev.this_device,
+   "SCIFAPI pollfd: ep %p %s\n", ep, scif_ep_states[ep->state]);
+
+   spin_lock(>lock);
+
+   /* Endpoint is waiting for a non-blocking connect to complete */
+   if (ep->conn_asy

[PATCH char-misc-next 00/19] misc: mic: Enable COSM and remaining SCIF functionality

[Ashutosh Dixit]

The Symmetric Communication Interface (SCIF) API was accepted upstream
during the v4.2 merge window @ https://lkml.org/lkml/2015/3/30/865.
This patch series completes the implementation of the SCIF API and
also introduces the MIC Coprocessor State Management (COSM) drivers.

The first patch introduces SCIF poll which allows user and kernel mode
clients to block for events on a SCIF endpoint. The next patch
introduces support for SCIF kernel mode clients. SCIF clients are
probed when new SCIF nodes come online. Similarly the client remove
method is called as SCIF nodes go offline. The SCIF client framework
makes use of the SCIF peer bus infrastructure submitted previously.

The next seven patches are a re-factoring of functionality for boot,
shutdown and reset of MIC cards. This functionality, called COSM, was
previously present in the MIC host driver but has now been re-factored
into a separate driver so that it can be shared across multiple
generations of MIC products. The MIC sysfs interface documented
earlier has been updated. Further, the COSM host driver communicates
with a COSM client on the card using SCIF. COSM is therefore the first
kernel mode SCIF client and demonstrates the use of a subset of the
SCIF API.

Patch 10 converts the iova implementation used by INTEL_IOMMU into a
library without any implementation changes so that it can be reused by
SCIF to generate registered offsets.

Patch 11 onward completes the SCIF implementation by enabling the
following SCIF Remote Memory Access (RMA) functionality:
a. Memory registration to pin and unpin pages
b. Remote Memory mapping for low latency CPU accesses
c. Remote DMA (RDMA) for high bandwidth DMA transfers
d. Fence mechanisms to synchronize RDMAs

Documentation/mic/scif_overview.txt contains more information about
SCIF. This patch series is divided into 19 patches as follows:

1) SCIF poll

2) Support for kernel mode SCIF clients

3) MIC COSM bus

4) Coprocessor State Management (COSM) driver

5) COSM SCIF server

6) COSM client driver

7) Remove COSM functionality from the MIC host driver

8) Remove COSM functionality from the MIC card driver

9) Update MIC host daemon with COSM changes

10) Convert iova.c into a library

11) Add support to program MIC x100 status descriptors

12) SCIF RMA header file and IOCTL changes

13) SCIF RMA header file

14) SCIF memory registration and unregistration

15) SCIF RMA list operations

16) SCIF remote memory map/unmap interface

17) SCIF DMA and CPU copy interface

18) SCIF fence

19) SCIF RMA nodeqp and minor miscellaneous changes

These patches have also been scanned by Fengguang Wu's 0-day
infrastructure and no issues have been reported. The IOVA patch (patch
10) is being sent to char-misc because of the dependency between that
patch and SCIF patches 12 through 19.

Ashutosh Dixit (9):
  misc: mic: SCIF poll
  misc: mic: Add support for kernel mode SCIF clients
  misc: mic: MIC COSM bus
  misc: mic: Coprocessor State Management (COSM) driver
  misc: mic: COSM SCIF server
  misc: mic: COSM client driver
  misc: mic: Remove COSM functionality from the MIC host driver
  misc: mic: Remove COSM functionality from the MIC card driver
  misc: mic: Update MIC host daemon with COSM changes

Harish Chegondi (1):
  lib: convert iova.c into a library

Siva Yerramreddy (1):
  dma: Add support to program MIC x100 status descriptiors

Sudeep Dutt (8):
  misc: mic: SCIF RMA header file and IOCTL changes
  misc: mic: SCIF RMA header file
  misc: mic: SCIF memory registration and unregistration
  misc: mic: SCIF RMA list operations
  misc: mic: SCIF remote memory map/unmap interface
  misc: mic: SCIF DMA and CPU copy interface
  misc: mic: SCIF fence
  misc: mic: SCIF RMA nodeqp and minor miscellaneous changes

 Documentation/ABI/testing/sysfs-class-mic.txt   |   29 +-
 Documentation/mic/mic_overview.txt  |   31 +-
 Documentation/mic/mpssd/mpss|4 +-
 Documentation/mic/mpssd/mpssd.c |  362 +++--
 Documentation/mic/mpssd/mpssd.h |1 +
 drivers/dma/mic_x100_dma.c  |   44 +-
 drivers/iommu/Kconfig   |5 +-
 drivers/iommu/Makefile  |1 -
 drivers/misc/mic/Kconfig|   23 +-
 drivers/misc/mic/Makefile   |2 +
 drivers/misc/mic/bus/Makefile   |1 +
 drivers/misc/mic/bus/cosm_bus.c |  141 ++
 drivers/misc/mic/bus/cosm_bus.h |  134 ++
 drivers/misc/mic/bus/mic_bus.c  |   22 +-
 drivers/misc/mic/bus/scif_bus.c |7 +-
 drivers/misc/mic/bus/scif_bus.h |6 +-
 drivers/misc/mic/card/mic_device.c  |   88 +-
 drivers/misc/mic/card/mic_x100.c|2 +-
 drivers/misc/mic/common/mic_dev.h   |   13 +
 drivers/misc/mic/cosm/Makefile  |   10 +
 drivers/misc/mic/cosm/cosm_debugfs.c|  155

[PATCH char-misc-next 07/19] misc: mic: Remove COSM functionality from the MIC host driver

[Ashutosh Dixit]

Since COSM functionality is now moved into a separate COSM driver
drivers, this patch removes this functionality from the base MIC host
driver. The MIC host driver now implements cosm_hw_ops and registers a
COSM device which allows the COSM driver to trigger
boot/shutdown/reset of the MIC devices via the cosm_hw_ops.

Reviewed-by: Nikhil Rao nikhil@intel.com
Reviewed-by: Sudeep Dutt sudeep.d...@intel.com
Signed-off-by: Dasaratharaman Chandramouli 
dasaratharaman.chandramo...@intel.com
Signed-off-by: Ashutosh Dixit ashutosh.di...@intel.com
---
 drivers/misc/mic/Makefile   |   2 -
 drivers/misc/mic/host/Makefile  |   1 -
 drivers/misc/mic/host/mic_boot.c| 317 +++--
 drivers/misc/mic/host/mic_debugfs.c | 114 +
 drivers/misc/mic/host/mic_device.h  |  88 +--
 drivers/misc/mic/host/mic_fops.c|   4 +-
 drivers/misc/mic/host/mic_intr.c|  46 ++--
 drivers/misc/mic/host/mic_main.c| 223 +++---
 drivers/misc/mic/host/mic_smpt.c|  30 ++-
 drivers/misc/mic/host/mic_sysfs.c   | 459 
 drivers/misc/mic/host/mic_virtio.c  |  17 +-
 drivers/misc/mic/host/mic_virtio.h  |   2 +-
 drivers/misc/mic/host/mic_x100.c|  46 ++--
 13 files changed, 204 insertions(+), 1145 deletions(-)
 delete mode 100644 drivers/misc/mic/host/mic_sysfs.c

diff --git a/drivers/misc/mic/Makefile b/drivers/misc/mic/Makefile
index a74042c..a2e75f4 100644
--- a/drivers/misc/mic/Makefile
+++ b/drivers/misc/mic/Makefile
@@ -2,7 +2,5 @@
 # Makefile - Intel MIC Linux driver.
 # Copyright(c) 2013, Intel Corporation.
 #
-obj-$(CONFIG_INTEL_MIC_HOST) += host/
-obj-$(CONFIG_INTEL_MIC_CARD) += card/
 obj-y += bus/
 obj-$(CONFIG_SCIF) += scif/
diff --git a/drivers/misc/mic/host/Makefile b/drivers/misc/mic/host/Makefile
index c2197f9..004d3db 100644
--- a/drivers/misc/mic/host/Makefile
+++ b/drivers/misc/mic/host/Makefile
@@ -5,7 +5,6 @@
 obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
 mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
-mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_intr.o
 mic_host-objs += mic_boot.o
diff --git a/drivers/misc/mic/host/mic_boot.c b/drivers/misc/mic/host/mic_boot.c
index e5f6a5e..7845564 100644
--- a/drivers/misc/mic/host/mic_boot.c
+++ b/drivers/misc/mic/host/mic_boot.c
@@ -22,9 +22,9 @@
 #include linux/firmware.h
 #include linux/pci.h
 #include linux/kmod.h
-
 #include linux/mic_common.h
 #include linux/mic_bus.h
+#include ../bus/scif_bus.h
 #include ../common/mic_dev.h
 #include mic_device.h
 #include mic_smpt.h
@@ -99,7 +99,7 @@ static int __mic_dma_map_sg(struct device *dev, struct 
scatterlist *sg,
int i, j, ret;
dma_addr_t da;
 
-   ret = dma_map_sg(mdev-sdev-parent, sg, nents, dir);
+   ret = dma_map_sg(mdev-pdev-dev, sg, nents, dir);
if (ret = 0)
return 0;
 
@@ -115,7 +115,7 @@ err:
mic_unmap(mdev, sg_dma_address(s), s-length);
sg_dma_address(s) = mic_to_dma_addr(mdev, sg_dma_address(s));
}
-   dma_unmap_sg(mdev-sdev-parent, sg, nents, dir);
+   dma_unmap_sg(mdev-pdev-dev, sg, nents, dir);
return 0;
 }
 
@@ -135,7 +135,7 @@ static void __mic_dma_unmap_sg(struct device *dev,
mic_unmap(mdev, sg_dma_address(s), s-length);
sg_dma_address(s) = da;
}
-   dma_unmap_sg(mdev-sdev-parent, sg, nents, dir);
+   dma_unmap_sg(mdev-pdev-dev, sg, nents, dir);
 }
 
 static struct dma_map_ops __mic_dma_ops = {
@@ -270,48 +270,13 @@ static struct mbus_hw_ops mbus_hw_ops = {
.ack_interrupt = _mic_ack_interrupt,
 };
 
-/**
- * mic_reset - Reset the MIC device.
- * @mdev: pointer to mic_device instance
- */
-static void mic_reset(struct mic_device *mdev)
-{
-   int i;
-
-#define MIC_RESET_TO (45)
-
-   reinit_completion(mdev-reset_wait);
-   mdev-ops-reset_fw_ready(mdev);
-   mdev-ops-reset(mdev);
-
-   for (i = 0; i  MIC_RESET_TO; i++) {
-   if (mdev-ops-is_fw_ready(mdev))
-   goto done;
-   /*
-* Resets typically take 10s of seconds to complete.
-* Since an MMIO read is required to check if the
-* firmware is ready or not, a 1 second delay works nicely.
-*/
-   msleep(1000);
-   }
-   mic_set_state(mdev, MIC_RESET_FAILED);
-done:
-   complete_all(mdev-reset_wait);
-}
-
 /* Initialize the MIC bootparams */
 void mic_bootparam_init(struct mic_device *mdev)
 {
struct mic_bootparam *bootparam = mdev-dp;
 
bootparam-magic = cpu_to_le32(MIC_MAGIC);
-   bootparam-c2h_shutdown_db = mdev-shutdown_db;
-   bootparam-h2c_shutdown_db = -1;
bootparam-h2c_config_db = -1;
-   bootparam-shutdown_status = 0;
-   bootparam-shutdown_card = 0;
-   /* Total nodes = number of MICs + 1 for self node */
-   bootparam-tot_nodes = atomic_read(g_num_mics) + 1

[PATCH char-misc-next 17/19] misc: mic: SCIF DMA and CPU copy interface

[Ashutosh Dixit]

From: Sudeep Dutt sudeep.d...@intel.com

SCIF allows users to read from or write to registered remote memory
via CPU copies or DMA. The API verifies that both local and remote
windows are valid before initiating the CPU or DMA transfers. SCIF has
optimized algorithms for handling byte aligned as well as cache line
aligned DMA engines. A registration cache is maintained to avoid the
overhead of pinning pages repeatedly if buffers are reused. The
registration cache is invalidated upon receipt of MMU notifier
callbacks.  SCIF windows are destroyed and the pages are unpinned only
once all prior DMAs initiated using that window are drained. Users can
request synchronous DMA operations as well as tail byte ordering if
required. CPU copies are always performed synchronously.

Reviewed-by: Ashutosh Dixit ashutosh.di...@intel.com
Reviewed-by: Nikhil Rao nikhil@intel.com
Signed-off-by: Sudeep Dutt sudeep.d...@intel.com
---
 drivers/misc/mic/scif/scif_dma.c | 1979 ++
 1 file changed, 1979 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_dma.c

diff --git a/drivers/misc/mic/scif/scif_dma.c b/drivers/misc/mic/scif/scif_dma.c
new file mode 100644
index 000..bf750b3
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_dma.c
@@ -0,0 +1,1979 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+#include scif_main.h
+#include scif_map.h
+
+/*
+ * struct scif_dma_comp_cb - SCIF DMA completion callback
+ *
+ * @dma_completion_func: DMA completion callback
+ * @cb_cookie: DMA completion callback cookie
+ * @temp_buf: Temporary buffer
+ * @temp_buf_to_free: Temporary buffer to be freed
+ * @is_cache: Is a kmem_cache allocated buffer
+ * @dst_offset: Destination registration offset
+ * @dst_window: Destination registration window
+ * @len: Length of the temp buffer
+ * @temp_phys: DMA address of the temp buffer
+ * @sdev: The SCIF device
+ * @header_padding: padding for cache line alignment
+ */
+struct scif_dma_comp_cb {
+   void (*dma_completion_func)(void *cookie);
+   void *cb_cookie;
+   u8 *temp_buf;
+   u8 *temp_buf_to_free;
+   bool is_cache;
+   s64 dst_offset;
+   struct scif_window *dst_window;
+   size_t len;
+   dma_addr_t temp_phys;
+   struct scif_dev *sdev;
+   int header_padding;
+};
+
+/**
+ * struct scif_copy_work - Work for DMA copy
+ *
+ * @src_offset: Starting source offset
+ * @dst_offset: Starting destination offset
+ * @src_window: Starting src registered window
+ * @dst_window: Starting dst registered window
+ * @loopback: true if this is a loopback DMA transfer
+ * @len: Length of the transfer
+ * @comp_cb: DMA copy completion callback
+ * @remote_dev: The remote SCIF peer device
+ * @fence_type: polling or interrupt based
+ * @ordered: is this a tail byte ordered DMA transfer
+ */
+struct scif_copy_work {
+   s64 src_offset;
+   s64 dst_offset;
+   struct scif_window *src_window;
+   struct scif_window *dst_window;
+   int loopback;
+   size_t len;
+   struct scif_dma_comp_cb   *comp_cb;
+   struct scif_dev *remote_dev;
+   int fence_type;
+   bool ordered;
+};
+
+#ifndef list_entry_next
+#define list_entry_next(pos, member) \
+   list_entry(pos-member.next, typeof(*pos), member)
+#endif
+
+/**
+ * scif_reserve_dma_chan:
+ * @ep: Endpoint Descriptor.
+ *
+ * This routine reserves a DMA channel for a particular
+ * endpoint. All DMA transfers for an endpoint are always
+ * programmed on the same DMA channel.
+ */
+int scif_reserve_dma_chan(struct scif_endpt *ep)
+{
+   int err = 0;
+   struct scif_dev *scifdev;
+   struct scif_hw_dev *sdev;
+   struct dma_chan *chan;
+
+   /* Loopback DMAs are not supported on the management node */
+   if (!scif_info.nodeid  scifdev_self(ep-remote_dev))
+   return 0;
+   if (scif_info.nodeid)
+   scifdev = scif_dev[0];
+   else
+   scifdev = ep-remote_dev;
+   sdev = scifdev-sdev;
+   if (!sdev-num_dma_ch)
+   return -ENODEV;
+   chan = sdev-dma_ch[scifdev-dma_ch_idx];
+   scifdev-dma_ch_idx = (scifdev-dma_ch_idx + 1) % sdev-num_dma_ch;
+   mutex_lock(ep-rma_info.rma_lock);
+   ep-rma_info.dma_chan = chan;
+   mutex_unlock(ep-rma_info.rma_lock);
+   return err;
+}
+
+#ifdef CONFIG_MMU_NOTIFIER
+/**
+ * scif_rma_destroy_tcw:
+ *
+ * This routine destroys temporary cached windows
+ */
+static
+void

[PATCH char-misc-next 18/19] misc: mic: SCIF fence

[Ashutosh Dixit]

From: Sudeep Dutt sudeep.d...@intel.com

This patch implements the fence APIs required to synchronize
DMAs. SCIF provides an interface to return a mark for all DMAs
programmed at the instant the API was called. Users can then wait on
the mark provided previously by blocking inside the kernel. Upon
receipt of a DMA completion interrupt the waiting thread is woken
up. There is also an interface to signal DMA completion by polling for
a location to be updated via a signal cookie to avoid the interrupt
overhead in the mark/wait interface. SCIF allows programming fences on
both the local and the remote node for both the mark/wait or the fence
signal APIs.

Reviewed-by: Ashutosh Dixit ashutosh.di...@intel.com
Reviewed-by: Nikhil Rao nikhil@intel.com
Signed-off-by: Sudeep Dutt sudeep.d...@intel.com
---
 drivers/misc/mic/scif/scif_fence.c | 773 +
 1 file changed, 773 insertions(+)
 create mode 100644 drivers/misc/mic/scif/scif_fence.c

diff --git a/drivers/misc/mic/scif/scif_fence.c 
b/drivers/misc/mic/scif/scif_fence.c
new file mode 100644
index 000..1620aaa
--- /dev/null
+++ b/drivers/misc/mic/scif/scif_fence.c
@@ -0,0 +1,773 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * Intel SCIF driver.
+ *
+ */
+
+#include scif_main.h
+
+/**
+ * scif_recv_mark: Handle SCIF_MARK request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested a mark.
+ */
+void scif_recv_mark(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg-payload[0];
+   int mark, err;
+
+   err = _scif_fence_mark(ep, mark);
+   if (err)
+   msg-uop = SCIF_MARK_NACK;
+   else
+   msg-uop = SCIF_MARK_ACK;
+   msg-payload[0] = ep-remote_ep;
+   msg-payload[2] = mark;
+   scif_nodeqp_send(ep-remote_dev, msg);
+}
+
+/**
+ * scif_recv_mark_resp: Handle SCIF_MARK_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_MARK message.
+ */
+void scif_recv_mark_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg-payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg-payload[1];
+
+   mutex_lock(ep-rma_info.rma_lock);
+   if (SCIF_MARK_ACK == msg-uop) {
+   fence_req-state = OP_COMPLETED;
+   fence_req-dma_mark = (int)msg-payload[2];
+   } else {
+   fence_req-state = OP_FAILED;
+   }
+   mutex_unlock(ep-rma_info.rma_lock);
+   complete(fence_req-comp);
+}
+
+/**
+ * scif_recv_wait: Handle SCIF_WAIT request
+ * @msg:   Interrupt message
+ *
+ * The peer has requested waiting on a fence.
+ */
+void scif_recv_wait(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg-payload[0];
+   struct scif_remote_fence_info *fence;
+
+   /*
+* Allocate structure for remote fence information and
+* send a NACK if the allocation failed. The peer will
+* return ENOMEM upon receiving a NACK.
+*/
+   fence = kmalloc(sizeof(*fence), GFP_KERNEL);
+   if (!fence) {
+   msg-payload[0] = ep-remote_ep;
+   msg-uop = SCIF_WAIT_NACK;
+   scif_nodeqp_send(ep-remote_dev, msg);
+   return;
+   }
+
+   /* Prepare the fence request */
+   memcpy(fence-msg, msg, sizeof(struct scifmsg));
+   INIT_LIST_HEAD(fence-list);
+
+   /* Insert to the global remote fence request list */
+   mutex_lock(scif_info.fencelock);
+   atomic_inc(ep-rma_info.fence_refcount);
+   list_add_tail(fence-list, scif_info.fence);
+   mutex_unlock(scif_info.fencelock);
+
+   schedule_work(scif_info.misc_work);
+}
+
+/**
+ * scif_recv_wait_resp: Handle SCIF_WAIT_(N)ACK messages.
+ * @msg:   Interrupt message
+ *
+ * The peer has responded to a SCIF_WAIT message.
+ */
+void scif_recv_wait_resp(struct scif_dev *scifdev, struct scifmsg *msg)
+{
+   struct scif_endpt *ep = (struct scif_endpt *)msg-payload[0];
+   struct scif_fence_info *fence_req =
+   (struct scif_fence_info *)msg-payload[1];
+
+   mutex_lock(ep-rma_info.rma_lock);
+   if (SCIF_WAIT_ACK == msg-uop)
+   fence_req-state = OP_COMPLETED;
+   else
+   fence_req-state = OP_FAILED;
+   mutex_unlock(ep-rma_info.rma_lock);
+   complete(fence_req-comp

[PATCH char-misc-next 02/19] misc: mic: Add support for kernel mode SCIF clients

[Ashutosh Dixit]

Add support for registration/de-registration of kernel mode SCIF
clients. SCIF clients are probed with new and existing SCIF peer
devices. Similarly the client remove method is called when SCIF
peer devices are removed.

Changes to SCIF peer device framework necessitated by supporting
kernel mode SCIF clients are also included in this patch.

Reviewed-by: Nikhil Rao nikhil@intel.com
Reviewed-by: Sudeep Dutt sudeep.d...@intel.com
Signed-off-by: Ashutosh Dixit ashutosh.di...@intel.com
---
 drivers/misc/mic/scif/scif_api.c  |  44 +
 drivers/misc/mic/scif/scif_main.c |  88 --
 drivers/misc/mic/scif/scif_main.h |   5 +-
 drivers/misc/mic/scif/scif_nm.c   |  10 +--
 drivers/misc/mic/scif/scif_nodeqp.c   |  65 +-
 drivers/misc/mic/scif/scif_peer_bus.c | 165 +-
 drivers/misc/mic/scif/scif_peer_bus.h |  42 +
 include/linux/scif.h  |  58 
 8 files changed, 256 insertions(+), 221 deletions(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index bf2d70f..900c86d 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -1430,3 +1430,47 @@ int scif_get_node_ids(u16 *nodes, int len, u16 *self)
return online;
 }
 EXPORT_SYMBOL_GPL(scif_get_node_ids);
+
+static int scif_add_client_dev(struct device *dev, struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client-probe)
+   client-probe(spdev);
+   return 0;
+}
+
+static int scif_remove_client_dev(struct device *dev,
+ struct subsys_interface *si)
+{
+   struct scif_client *client =
+   container_of(si, struct scif_client, si);
+   struct scif_peer_dev *spdev =
+   container_of(dev, struct scif_peer_dev, dev);
+
+   if (client-remove)
+   client-remove(spdev);
+   return 0;
+}
+
+void scif_client_unregister(struct scif_client *client)
+{
+   subsys_interface_unregister(client-si);
+}
+EXPORT_SYMBOL_GPL(scif_client_unregister);
+
+int scif_client_register(struct scif_client *client)
+{
+   struct subsys_interface *si = client-si;
+
+   si-name = client-name;
+   si-subsys = scif_peer_bus;
+   si-add_dev = scif_add_client_dev;
+   si-remove_dev = scif_remove_client_dev;
+
+   return subsys_interface_register(client-si);
+}
+EXPORT_SYMBOL_GPL(scif_client_register);
diff --git a/drivers/misc/mic/scif/scif_main.c 
b/drivers/misc/mic/scif/scif_main.c
index 6ce851f..f90bd06 100644
--- a/drivers/misc/mic/scif/scif_main.c
+++ b/drivers/misc/mic/scif/scif_main.c
@@ -80,35 +80,6 @@ irqreturn_t scif_intr_handler(int irq, void *data)
return IRQ_HANDLED;
 }
 
-static int scif_peer_probe(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = scif_dev[spdev-dnode];
-
-   mutex_lock(scif_info.conflock);
-   scif_info.total++;
-   scif_info.maxid = max_t(u32, spdev-dnode, scif_info.maxid);
-   mutex_unlock(scif_info.conflock);
-   rcu_assign_pointer(scifdev-spdev, spdev);
-
-   /* In the future SCIF kernel client devices will be added here */
-   return 0;
-}
-
-static void scif_peer_remove(struct scif_peer_dev *spdev)
-{
-   struct scif_dev *scifdev = scif_dev[spdev-dnode];
-
-   /* In the future SCIF kernel client devices will be removed here */
-   spdev = rcu_dereference(scifdev-spdev);
-   if (spdev)
-   RCU_INIT_POINTER(scifdev-spdev, NULL);
-   synchronize_rcu();
-
-   mutex_lock(scif_info.conflock);
-   scif_info.total--;
-   mutex_unlock(scif_info.conflock);
-}
-
 static void scif_qp_setup_handler(struct work_struct *work)
 {
struct scif_dev *scifdev = container_of(work, struct scif_dev,
@@ -139,20 +110,13 @@ static void scif_qp_setup_handler(struct work_struct 
*work)
}
 }
 
-static int scif_setup_scifdev(struct scif_hw_dev *sdev)
+static int scif_setup_scifdev(void)
 {
+   /* We support a maximum of 129 SCIF nodes including the mgmt node */
+#define MAX_SCIF_NODES 129
int i;
-   u8 num_nodes;
+   u8 num_nodes = MAX_SCIF_NODES;
 
-   if (sdev-snode) {
-   struct mic_bootparam __iomem *bp = sdev-rdp;
-
-   num_nodes = ioread8(bp-tot_nodes);
-   } else {
-   struct mic_bootparam *bp = sdev-dp;
-
-   num_nodes = bp-tot_nodes;
-   }
scif_dev = kcalloc(num_nodes, sizeof(*scif_dev), GFP_KERNEL);
if (!scif_dev)
return -ENOMEM;
@@ -163,7 +127,7 @@ static int scif_setup_scifdev(struct scif_hw_dev *sdev)
scifdev-exit = OP_IDLE;
init_waitqueue_head(scifdev-disconn_wq);
mutex_init(scifdev-lock);
-   INIT_WORK

[PATCH char-misc-next 01/19] misc: mic: SCIF poll

[Ashutosh Dixit]

SCIF poll allows both user and kernel mode clients to wait on
events on a SCIF endpoint. These events include availability of
space or data in the SCIF ring buffer, availability of connection
requests on a listening endpoint and completion of connections
when using async connects.

Reviewed-by: Nikhil Rao nikhil@intel.com
Reviewed-by: Sudeep Dutt sudeep.d...@intel.com
Signed-off-by: Ashutosh Dixit ashutosh.di...@intel.com
---
 drivers/misc/mic/scif/scif_api.c  | 158 +-
 drivers/misc/mic/scif/scif_epd.h  |  22 ++
 drivers/misc/mic/scif/scif_fd.c   |   9 +++
 drivers/misc/mic/scif/scif_main.h |   2 +
 include/linux/scif.h  |  74 ++
 5 files changed, 264 insertions(+), 1 deletion(-)

diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index f39d313..bf2d70f 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -37,9 +37,21 @@ enum conn_async_state {
ASYNC_CONN_FLUSH_WORK   /* async work flush in progress  */
 };
 
+/*
+ * File operations for anonymous inode file associated with a SCIF endpoint,
+ * used in kernel mode SCIF poll. Kernel mode SCIF poll calls portions of the
+ * poll API in the kernel and these take in a struct file *. Since a struct
+ * file is not available to kernel mode SCIF, it uses an anonymous file for
+ * this purpose.
+ */
+const struct file_operations scif_anon_fops = {
+   .owner = THIS_MODULE,
+};
+
 scif_epd_t scif_open(void)
 {
struct scif_endpt *ep;
+   int err;
 
might_sleep();
ep = kzalloc(sizeof(*ep), GFP_KERNEL);
@@ -50,6 +62,10 @@ scif_epd_t scif_open(void)
if (!ep-qp_info.qp)
goto err_qp_alloc;
 
+   err = scif_anon_inode_getfile(ep);
+   if (err)
+   goto err_anon_inode;
+
spin_lock_init(ep-lock);
mutex_init(ep-sendlock);
mutex_init(ep-recvlock);
@@ -59,6 +75,8 @@ scif_epd_t scif_open(void)
SCIFAPI open: ep %p success\n, ep);
return ep;
 
+err_anon_inode:
+   kfree(ep-qp_info.qp);
 err_qp_alloc:
kfree(ep);
 err_ep_alloc:
@@ -279,6 +297,7 @@ int scif_close(scif_epd_t epd)
}
}
scif_put_port(ep-port.port);
+   scif_anon_inode_fput(ep);
scif_teardown_ep(ep);
scif_add_epd_to_zombie_list(ep, !SCIF_EPLOCK_HELD);
return 0;
@@ -558,8 +577,10 @@ void scif_conn_handler(struct work_struct *work)
list_del(ep-conn_list);
}
spin_unlock(scif_info.nb_connect_lock);
-   if (ep)
+   if (ep) {
ep-conn_err = scif_conn_func(ep);
+   wake_up_interruptible(ep-conn_pend_wq);
+   }
} while (ep);
 }
 
@@ -660,6 +681,7 @@ int __scif_connect(scif_epd_t epd, struct scif_port_id 
*dst, bool non_block)
ep-remote_dev = scif_dev[dst-node];
ep-qp_info.qp-magic = SCIFEP_MAGIC;
if (ep-conn_async_state == ASYNC_CONN_INPROGRESS) {
+   init_waitqueue_head(ep-conn_pend_wq);
spin_lock(scif_info.nb_connect_lock);
list_add_tail(ep-conn_list, scif_info.nb_connect_list);
spin_unlock(scif_info.nb_connect_lock);
@@ -788,6 +810,10 @@ retry_connection:
goto scif_accept_error_qpalloc;
}
 
+   err = scif_anon_inode_getfile(cep);
+   if (err)
+   goto scif_accept_error_anon_inode;
+
cep-qp_info.qp-magic = SCIFEP_MAGIC;
spdev = scif_get_peer_dev(cep-remote_dev);
if (IS_ERR(spdev)) {
@@ -858,6 +884,8 @@ retry:
spin_unlock(cep-lock);
return 0;
 scif_accept_error_map:
+   scif_anon_inode_fput(cep);
+scif_accept_error_anon_inode:
scif_teardown_ep(cep);
 scif_accept_error_qpalloc:
kfree(cep);
@@ -1247,6 +1275,134 @@ int scif_recv(scif_epd_t epd, void *msg, int len, int 
flags)
 }
 EXPORT_SYMBOL_GPL(scif_recv);
 
+static inline void _scif_poll_wait(struct file *f, wait_queue_head_t *wq,
+  poll_table *p, struct scif_endpt *ep)
+{
+   /*
+* Because poll_wait makes a GFP_KERNEL allocation, give up the lock
+* and regrab it afterwards. Because the endpoint state might have
+* changed while the lock was given up, the state must be checked
+* again after re-acquiring the lock. The code in __scif_pollfd(..)
+* does this.
+*/
+   spin_unlock(ep-lock);
+   poll_wait(f, wq, p);
+   spin_lock(ep-lock);
+}
+
+unsigned int
+__scif_pollfd(struct file *f, poll_table *wait, struct scif_endpt *ep)
+{
+   unsigned int mask = 0;
+
+   dev_dbg(scif_info.mdev.this_device,
+   SCIFAPI pollfd: ep %p %s\n, ep, scif_ep_states[ep-state]);
+
+   spin_lock(ep-lock);
+
+   /* Endpoint is waiting for a non-blocking connect to complete */
+   if (ep-conn_async_state