[PATCH v2 00/36] Add rdma verbs transport library

[Dennis Dalessandro]

The following series implements rdmavt. This is the rdma verbs transport
software library which will help to solve the problem of code duplication
between hardware drivers when it comes to a verbs implementation.

Rdmavt is basically just another verbs provider and lives in the Infiniband tree
in a new sw directory. It provides a software implementation of the Infiniband
verbs API. More details can be found in the following threads:

http://www.spinics.net/lists/linux-rdma/msg29064.html
http://www.spinics.net/lists/linux-rdma/msg29922.html

This patch series is based on what appears to be the latest branch on Doug's
kernel.org repo (k.o/for-4.4-rc). I can rebase onto something else if needed.

Changes since v1:
Removed driver specific version
Fixed license text to remove copyright and put on top
Return 0 in rvt_map_sg instead of BAD_DMA_AGGRESS
Remove #include of dma.h from dma.c
Update comment about protection domain limit
Remove comment on alternative design for private data
Rename CDR macro to CHECK_DRIVER_OVERRIDE
Change all the stubs to return EOPNOTSUPP
Fix comment style for rvt_query_port
Fix typo in subject
Rename rdi.lk_table to rdi.lkey_table
Rename rvt_sge.m => rvt_sge.cur_map (Sean)
Rename rvt_sge.n => rvt_sge.cur_seg (Sean)
Remove rvt_reg_phys_mr
Drop support for commit 38071a461f0a ("IB/qib: Support the new memory
   registration API")

---

Dennis Dalessandro (34):
  IB/rdmavt: Create module framework and handle driver registration
  IB/rdmavt: Consolidate dma ops in rdmavt.
  IB/rdmavt: Add protection domain to rdmavt.
  IB/rdmavt: Add ib core device attributes to rvt driver params list
  IB/rdmavt: Macroize override checks during driver registration
  IB/rdmavt: Add query and modify device stubs
  IB/rdmavt: Add query and modify port stubs
  IB/rdmavt: Add pkey query stub
  IB/rdmavt: Add query gid stub
  IB/rdmavt: Alloc and dealloc ucontexts
  IB/rdmavt: Add queue pair function stubs
  IB/rdmavt: Add address handle stubs
  IB/rdmavt: Add memory region stubs
  IB/rdmavt: Add SRQ stubs
  IB/rdmavt: Add multicast stubs
  IB/rdmavt: Add process MAD stub
  IB/rdmavt: Add mmap stub
  IB/rdmavt: Add get port immutable stub
  IB/rdmavt: Add completion queue function stubs
  IB/rdmavt: Add post send and recv stubs
  IB/rdmavt: Move MR datastructures into rvt
  IB/rdmavt: Add queue pair data structure to rdmavt
  IB/rdmavt: Move driver helper functions to a common structure
  IB/rdmavt: Add device specific info prints
  IB/rdmavt: Add the start of capability flags
  IB/rdmavt: Move memory registration into rdmavt
  IB/rdmavt: Do not use rvt prints which rely on driver too early
  IB/rdmavt: Move SRQ data structure into rdmavt
  IB/rdmavt: Add an ibport data structure to rdmavt
  IB/rdmavt: Add driver notification for new AH
  IB/rdmavt: Break rdma_vt main include header file up
  IB/rdmavt: Initialize and teardown of qpn table
  IB/rdmavt: Add mmap related functions
  IB/rdmavt: Add pkey support

Kamal Heib (2):
  IB/rdmavt: Add common LID defines to rdmavt
  IB/rdmavt: Add AH to rdmavt


 MAINTAINERS   |6 
 drivers/infiniband/Kconfig|2 
 drivers/infiniband/Makefile   |1 
 drivers/infiniband/sw/Makefile|1 
 drivers/infiniband/sw/rdmavt/Kconfig  |6 
 drivers/infiniband/sw/rdmavt/Makefile |   10 
 drivers/infiniband/sw/rdmavt/ah.c |  169 +++
 drivers/infiniband/sw/rdmavt/ah.h |   59 ++
 drivers/infiniband/sw/rdmavt/cq.c |  113 +
 drivers/infiniband/sw/rdmavt/cq.h |   62 ++
 drivers/infiniband/sw/rdmavt/dma.c|  182 +++
 drivers/infiniband/sw/rdmavt/dma.h|   53 ++
 drivers/infiniband/sw/rdmavt/mad.c|   85 +++
 drivers/infiniband/sw/rdmavt/mad.h|   59 ++
 drivers/infiniband/sw/rdmavt/mcast.c  |   58 ++
 drivers/infiniband/sw/rdmavt/mcast.h  |   56 ++
 drivers/infiniband/sw/rdmavt/mmap.c   |  198 
 drivers/infiniband/sw/rdmavt/mmap.h   |   55 ++
 drivers/infiniband/sw/rdmavt/mr.c |  824 +
 drivers/infiniband/sw/rdmavt/mr.h |   95 
 drivers/infiniband/sw/rdmavt/pd.c |  103 
 drivers/infiniband/sw/rdmavt/pd.h |   58 ++
 drivers/infiniband/sw/rdmavt/qp.c |  379 +++
 drivers/infiniband/sw/rdmavt/qp.h |   69 +++
 drivers/infiniband/sw/rdmavt/srq.c|   86 +++
 drivers/infiniband/sw/rdmavt/srq.h|   61 ++
 drivers/infiniband/sw/rdmavt/vt.c |  377 +++
 drivers/infiniband/sw/rdmavt/vt.h |   90 
 include/rdma/ib_verbs.h   |1 
 include/rdma/rdma_vt.h|  333 +
 include/rdma/rdmavt_mr.h  |  130 +
 include/rdma/rdmavt_qp.h  |  295 
 32 files changed, 4076 insertions(+), 0 deletions(-)
 create mode 100644 

[PATCH v2 05/36] IB/rdmavt: Macroize override checks during driver registration

[Dennis Dalessandro]

Add a macro to cut down on writing the same lines over and over again for
what will be a large number of functions that will be supported.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change CDR macro to CHECK_DRIVER_OVERRIDE to be more clear

 drivers/infiniband/sw/rdmavt/vt.c |   22 +-
 1 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index b65cde4..8bd25c3 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -64,29 +64,25 @@ static void rvt_cleanup(void)
 }
 module_exit(rvt_cleanup);
 
+/*
+ * Check driver override. If driver passes a value use it, otherwise we use our
+ * own value.
+ */
+#define CHECK_DRIVER_OVERRIDE(rdi, x) \
+   rdi->ibdev.x = rdi->ibdev.x ? : rvt_ ##x
+
 int rvt_register_device(struct rvt_dev_info *rdi)
 {
if (!rdi)
return -EINVAL;
 
-   /*
-* Drivers have the option to override anything in the ibdev that they
-* want to specifically handle. VT needs to check for things it supports
-* and if the driver wants to handle that functionality let it. We may
-* come up with a better mechanism that simplifies the code at some
-* point.
-*/
-
/* DMA Operations */
rdi->ibdev.dma_ops =
rdi->ibdev.dma_ops ? : _default_dma_mapping_ops;
 
/* Protection Domain */
-   rdi->ibdev.alloc_pd =
-   rdi->ibdev.alloc_pd ? : rvt_alloc_pd;
-   rdi->ibdev.dealloc_pd =
-   rdi->ibdev.dealloc_pd ? : rvt_dealloc_pd;
-
+   CHECK_DRIVER_OVERRIDE(rdi, alloc_pd);
+   CHECK_DRIVER_OVERRIDE(rdi, dealloc_pd);
spin_lock_init(>n_pds_lock);
rdi->n_pds_allocated = 0;
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 10/36] IB/rdmavt: Alloc and dealloc ucontexts

[Dennis Dalessandro]

Add the stubs to allocate and deallocate user contexts. This will be
handled completely by rvt.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

drivers/infiniband/sw/rdmavt/vt.c |   22 ++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index e95f197..cd19429 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -178,6 +178,26 @@ static int rvt_query_gid(struct ib_device *ibdev, u8 port,
return -EOPNOTSUPP;
 }
 
+/**
+ * rvt_alloc_ucontext - Allocate a user context
+ * @ibdev: Vers IB dev
+ * @data: User data allocated
+ */
+static struct ib_ucontext *rvt_alloc_ucontext(struct ib_device *ibdev,
+ struct ib_udata *udata)
+{
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ *rvt_dealloc_ucontext - Free a user context
+ *@context - Free this
+ */
+static int rvt_dealloc_ucontext(struct ib_ucontext *context)
+{
+   return -EOPNOTSUPP;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -197,6 +217,8 @@ int rvt_register_device(struct rvt_dev_info *rdi)
CHECK_DRIVER_OVERRIDE(rdi, modify_port);
CHECK_DRIVER_OVERRIDE(rdi, query_pkey);
CHECK_DRIVER_OVERRIDE(rdi, query_gid);
+   CHECK_DRIVER_OVERRIDE(rdi, alloc_ucontext);
+   CHECK_DRIVER_OVERRIDE(rdi, dealloc_ucontext);
 
/* DMA Operations */
rdi->ibdev.dma_ops =

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 09/36] IB/rdmavt: Add query gid stub

[Dennis Dalessandro]

The handler for query gid operates along the same lines as the query pkey
handler. The driver will take care to keep the guid table updated.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |   22 ++
 1 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index fc5372d..e95f197 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -157,6 +157,27 @@ static int rvt_query_pkey(struct ib_device *ibdev, u8 
port, u16 index,
return 0;
 }
 
+/**
+ * rvt_query_gid - Return a gid from the table
+ * @ibdev: Verbs IB dev
+ * @port: Port number
+ * @index: = Index in table
+ * @gid: Gid to return
+ *
+ * Returns 0 on success
+ */
+static int rvt_query_gid(struct ib_device *ibdev, u8 port,
+int index, union ib_gid *gid)
+{
+   /*
+* Driver is responsible for updating the guid table. Which will be used
+* to craft the return value. This will work similar to how query_pkey()
+* is being done.
+*/
+
+   return -EOPNOTSUPP;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -175,6 +196,7 @@ int rvt_register_device(struct rvt_dev_info *rdi)
CHECK_DRIVER_OVERRIDE(rdi, query_port);
CHECK_DRIVER_OVERRIDE(rdi, modify_port);
CHECK_DRIVER_OVERRIDE(rdi, query_pkey);
+   CHECK_DRIVER_OVERRIDE(rdi, query_gid);
 
/* DMA Operations */
rdi->ibdev.dma_ops =

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 08/36] IB/rdmavt: Add pkey query stub

[Dennis Dalessandro]

The pkey table will reside in the rvt structure but it will be modified
only when the driver requests then rvt will simply read the value to return
in the query.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |   21 +
 include/rdma/rdma_vt.h|3 ++-
 2 files changed, 23 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 5ac241c..fc5372d 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -137,6 +137,26 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 
port,
return -EOPNOTSUPP;
 }
 
+/**
+ * rvt_query_pkey - Return a pkey from the table at a given index
+ * @ibdev: Verbs IB dev
+ * @port: Port number
+ * @intex: Index into pkey table
+ *
+ * Returns 0 on failure pkey otherwise
+ */
+static int rvt_query_pkey(struct ib_device *ibdev, u8 port, u16 index,
+ u16 *pkey)
+{
+   /*
+* Driver will be responsible for keeping rvt_dev_info.pkey_table up to
+* date. This function will just return that value. There is no need to
+* lock, if a stale value is read and sent to the user so be it there is
+* no way to protect against that anyway.
+*/
+   return 0;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -154,6 +174,7 @@ int rvt_register_device(struct rvt_dev_info *rdi)
CHECK_DRIVER_OVERRIDE(rdi, modify_device);
CHECK_DRIVER_OVERRIDE(rdi, query_port);
CHECK_DRIVER_OVERRIDE(rdi, modify_port);
+   CHECK_DRIVER_OVERRIDE(rdi, query_pkey);
 
/* DMA Operations */
rdi->ibdev.dma_ops =
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 2990e03..bf072a4 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -114,12 +114,13 @@ struct rvt_dev_info {
 * The driver will also be responsible for filling in certain members of
 * dparms.props
 */
-
struct ib_device ibdev;
 
/* Driver specific properties */
struct rvt_driver_params dparms;
 
+   /* PKey Table goes here */
+
/*
 * The work to create port files in /sys/class Infiniband is different
 * depending on the driver. This should not be extracted away and

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 13/36] IB/rdmavt: Add memory region stubs

[Dennis Dalessandro]

Adds the function stubs for allocating, and registering memory regions, as
well as deregistering them.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/Makefile |2 
 drivers/infiniband/sw/rdmavt/mr.c |  182 +
 drivers/infiniband/sw/rdmavt/mr.h |   72 +
 drivers/infiniband/sw/rdmavt/vt.c |   11 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 267 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/mr.c
 create mode 100644 drivers/infiniband/sw/rdmavt/mr.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 628c684..084ee6a 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o pd.o qp.o
+rdmavt-y := vt.o ah.o dma.o mr.o pd.o qp.o
diff --git a/drivers/infiniband/sw/rdmavt/mr.c 
b/drivers/infiniband/sw/rdmavt/mr.c
new file mode 100644
index 000..c8e5cfe
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mr.c
@@ -0,0 +1,182 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include 
+#include "mr.h"
+
+/**
+ * rvt_get_dma_mr - get a DMA memory region
+ * @pd: protection domain for this memory region
+ * @acc: access flags
+ *
+ * Returns the memory region on success, otherwise returns an errno.
+ * Note that all DMA addresses should be created via the
+ * struct ib_dma_mapping_ops functions (see dma.c).
+ */
+struct ib_mr *rvt_get_dma_mr(struct ib_pd *pd, int acc)
+{
+   /*
+* Alloc mr and init it.
+* Alloc lkey.
+*/
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * rvt_reg_phys_mr - register a physical memory region
+ * @pd: protection domain for this memory region
+ * @buffer_list: pointer to the list of physical buffers to register
+ * @num_phys_buf: the number of physical buffers to register
+ * @iova_start: the starting address passed over IB which maps to this MR
+ *
+ * Returns the memory region on success, otherwise returns an errno.
+ */
+struct ib_mr *rvt_reg_phys_mr(struct ib_pd *pd,
+ struct ib_phys_buf *buffer_list,
+ int num_phys_buf, int acc, u64 *iova_start)
+{
+   /* Alloc mr and build it up */
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * rvt_reg_user_mr - register a userspace memory region
+ * @pd: protection domain for this memory region
+ * @start: starting userspace address
+ * @length: length of region to register
+ * @mr_access_flags: access flags for 

[PATCH v2 12/36] IB/rdmavt: Add address handle stubs

[Dennis Dalessandro]

Adds the stubs for create, destroy, modify, and query of the
address handle.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

drivers/infiniband/sw/rdmavt/Makefile |3 -
 drivers/infiniband/sw/rdmavt/ah.c |   76 +
 drivers/infiniband/sw/rdmavt/ah.h |   59 ++
 drivers/infiniband/sw/rdmavt/vt.c |6 +++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 143 insertions(+), 2 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/ah.c
 create mode 100644 drivers/infiniband/sw/rdmavt/ah.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 9a9a095..628c684 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,5 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o dma.o pd.o qp.o
-
+rdmavt-y := vt.o ah.o dma.o pd.o qp.o
diff --git a/drivers/infiniband/sw/rdmavt/ah.c 
b/drivers/infiniband/sw/rdmavt/ah.c
new file mode 100644
index 000..d368955
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/ah.c
@@ -0,0 +1,76 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "ah.h"
+
+/**
+ * rvt_create_ah - create an address handle
+ * @pd: the protection domain
+ * @ah_attr: the attributes of the AH
+ *
+ * This may be called from interrupt context.
+ */
+struct ib_ah *rvt_create_ah(struct ib_pd *pd,
+   struct ib_ah_attr *ah_attr)
+{
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+int rvt_destroy_ah(struct ib_ah *ibah)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_modify_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
+{
+   return -EOPNOTSUPP;
+}
diff --git a/drivers/infiniband/sw/rdmavt/ah.h 
b/drivers/infiniband/sw/rdmavt/ah.h
new file mode 100644
index 000..8cd7ea7
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/ah.h
@@ -0,0 +1,59 @@
+#ifndef DEF_RVTAH_H
+#define DEF_RVTAH_H
+
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or 

[PATCH v2 20/36] IB/rdmavt: Add post send and recv stubs

[Dennis Dalessandro]

This adds the post sned and recv function stubs.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Fix typo in subject
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/qp.c |   62 +
 drivers/infiniband/sw/rdmavt/qp.h |7 
 drivers/infiniband/sw/rdmavt/vt.c |3 ++
 3 files changed, 71 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/qp.c 
b/drivers/infiniband/sw/rdmavt/qp.c
index a59f28d..23a5f68 100644
--- a/drivers/infiniband/sw/rdmavt/qp.c
+++ b/drivers/infiniband/sw/rdmavt/qp.c
@@ -118,3 +118,65 @@ int rvt_query_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
 {
return -EOPNOTSUPP;
 }
+
+/**
+ * rvt_post_receive - post a receive on a QP
+ * @ibqp: the QP to post the receive on
+ * @wr: the WR to post
+ * @bad_wr: the first bad WR is put here
+ *
+ * This may be called from interrupt context.
+ */
+int rvt_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+ struct ib_recv_wr **bad_wr)
+{
+   /*
+* When a packet arrives the driver needs to call up to rvt to process
+* the packet. The UD, RC, UC processing will be done in rvt, however
+* the driver should be able to override this if it so choses. Perhaps a
+* set of function pointers set up at registration time.
+*/
+
+   return -EOPNOTSUPP;
+}
+
+/**
+ * rvt_post_send - post a send on a QP
+ * @ibqp: the QP to post the send on
+ * @wr: the list of work requests to post
+ * @bad_wr: the first bad WR is put here
+ *
+ * This may be called from interrupt context.
+ */
+int rvt_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+ struct ib_send_wr **bad_wr)
+{
+   /*
+* VT-DRIVER-API: do_send()
+* Driver needs to have a do_send() call which is a single entry point
+* to take an already formed packet and throw it out on the wire. Once
+* the packet is sent the driver needs to make an upcall to rvt so the
+* completion queue can be notified and/or any other outstanding
+* work/book keeping can be finished.
+*
+* Note that there should also be a way for rvt to protect itself
+* against hangs in the driver layer. If a send doesn't actually
+* complete in a timely manor rvt needs to return an error event.
+*/
+
+   return -EOPNOTSUPP;
+}
+
+/**
+ * rvt_post_srq_receive - post a receive on a shared receive queue
+ * @ibsrq: the SRQ to post the receive on
+ * @wr: the list of work requests to post
+ * @bad_wr: A pointer to the first WR to cause a problem is put here
+ *
+ * This may be called from interrupt context.
+ */
+int rvt_post_srq_recv(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
+ struct ib_recv_wr **bad_wr)
+{
+   return -EOPNOTSUPP;
+}
diff --git a/drivers/infiniband/sw/rdmavt/qp.h 
b/drivers/infiniband/sw/rdmavt/qp.h
index 43acba0..10bc636 100644
--- a/drivers/infiniband/sw/rdmavt/qp.h
+++ b/drivers/infiniband/sw/rdmavt/qp.h
@@ -63,5 +63,10 @@ int rvt_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
 int rvt_destroy_qp(struct ib_qp *ibqp);
 int rvt_query_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 int attr_mask, struct ib_qp_init_attr *init_attr);
-
+int rvt_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr,
+ struct ib_recv_wr **bad_wr);
+int rvt_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
+ struct ib_send_wr **bad_wr);
+int rvt_post_srq_recv(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
+ struct ib_recv_wr **bad_wr);
 #endif  /* DEF_RVTQP_H */
diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 33986e3..0cf80d1 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -232,6 +232,9 @@ int rvt_register_device(struct rvt_dev_info *rdi)
CHECK_DRIVER_OVERRIDE(rdi, modify_qp);
CHECK_DRIVER_OVERRIDE(rdi, destroy_qp);
CHECK_DRIVER_OVERRIDE(rdi, query_qp);
+   CHECK_DRIVER_OVERRIDE(rdi, post_send);
+   CHECK_DRIVER_OVERRIDE(rdi, post_recv);
+   CHECK_DRIVER_OVERRIDE(rdi, post_srq_recv);
 
/* Address Handle */
CHECK_DRIVER_OVERRIDE(rdi, create_ah);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 21/36] IB/rdmavt: Move MR datastructures into rvt

[Dennis Dalessandro]

This patch adds the MR datastructures based on hfi1 into rvt. For now the
data structures are defined in include/rdma/rdma_vt.h but once all MR
functionality has been moved from the drivers into rvt these should move to
rdmavt/mr.h

Reviewed-by: Ira Weiny 
Reviewed-by: Dean Luick 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Rename rdi.lk_table => rdi.lkey_table

 include/rdma/rdma_vt.h |   53 
 1 files changed, 53 insertions(+), 0 deletions(-)

diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index bf072a4..f232e39 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -56,6 +56,56 @@
 #include "ib_verbs.h"
 
 /*
+ * For Memory Regions. This stuff should probably be moved into rdmavt/mr.h 
once
+ * drivers no longer need access to the MR directly.
+ */
+
+/*
+ * A segment is a linear region of low physical memory.
+ * Used by the verbs layer.
+ */
+struct rvt_seg {
+   void *vaddr;
+   size_t length;
+};
+
+/* The number of rvt_segs that fit in a page. */
+#define RVT_SEGSZ (PAGE_SIZE / sizeof(struct rvt_seg))
+
+struct rvt_segarray {
+   struct rvt_seg segs[RVT_SEGSZ];
+};
+
+struct rvt_mregion {
+   struct ib_pd *pd;   /* shares refcnt of ibmr.pd */
+   u64 user_base;  /* User's address for this region */
+   u64 iova;   /* IB start address of this region */
+   size_t length;
+   u32 lkey;
+   u32 offset; /* offset (bytes) to start of region */
+   int access_flags;
+   u32 max_segs;   /* number of rvt_segs in all the arrays */
+   u32 mapsz;  /* size of the map array */
+   u8  page_shift; /* 0 - non unform/non powerof2 sizes */
+   u8  lkey_published; /* in global table */
+   struct completion comp; /* complete when refcount goes to zero */
+   atomic_t refcount;
+   struct rvt_segarray *map[0];/* the segments */
+};
+
+#define RVT_MAX_LKEY_TABLE_BITS 23
+
+struct rvt_lkey_table {
+   spinlock_t lock; /* protect changes in this struct */
+   u32 next;   /* next unused index (speeds search) */
+   u32 gen;/* generation count */
+   u32 max;/* size of the table */
+   struct rvt_mregion __rcu **table;
+};
+
+/* End Memmory Region */
+
+/*
  * Things that are driver specific, module parameters in hfi1 and qib
  */
 struct rvt_driver_params {
@@ -119,6 +169,9 @@ struct rvt_dev_info {
/* Driver specific properties */
struct rvt_driver_params dparms;
 
+   struct rvt_mregion __rcu *dma_mr;
+   struct rvt_lkey_table lkey_table;
+
/* PKey Table goes here */
 
/*

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 23/36] IB/rdmavt: Move driver helper functions to a common structure

[Dennis Dalessandro]

Drivers are going to need to provide multiple functions for rdmavt to
call in to. We already have one, so go ahead and push this into a
data structure designated for driver supplied functions.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |6 +-
 include/rdma/rdma_vt.h|   22 +++---
 2 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 0cf80d1..e4881ca 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -281,8 +281,12 @@ int rvt_register_device(struct rvt_dev_info *rdi)
spin_lock_init(>n_pds_lock);
rdi->n_pds_allocated = 0;
 
+   /* Validate that drivers have provided the right functions */
+   if (!rdi->driver_f.port_callback)
+   return -EINVAL;
+
/* We are now good to announce we exist */
-   return ib_register_device(>ibdev, rdi->port_callback);
+   return ib_register_device(>ibdev, rdi->driver_f.port_callback);
 }
 EXPORT_SYMBOL(rvt_register_device);
 
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 9baa7f0..e0beedc 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -383,6 +383,19 @@ struct rvt_driver_params {
 */
 };
 
+/*
+ * Functions that drivers are required to support
+ */
+struct rvt_driver_provided {
+   /*
+* The work to create port files in /sys/class Infiniband is different
+* depending on the driver. This should not be extracted away and
+* instead drivers are responsible for setting the correct callback for
+* this.
+*/
+   int (*port_callback)(struct ib_device *, u8, struct kobject *);
+};
+
 /* Protection domain */
 struct rvt_pd {
struct ib_pd ibpd;
@@ -407,13 +420,8 @@ struct rvt_dev_info {
 
/* PKey Table goes here */
 
-   /*
-* The work to create port files in /sys/class Infiniband is different
-* depending on the driver. This should not be extracted away and
-* instead drivers are responsible for setting the correct callback for
-* this.
-*/
-   int (*port_callback)(struct ib_device *, u8, struct kobject *);
+   /* Driver specific helper functions */
+   struct rvt_driver_provided driver_f;
 
/* Internal use */
int n_pds_allocated;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 10/10] IB/qib: Remove srq from qib

[Dennis Dalessandro]

Remove srq from qib now that it has been moved into rdmavt.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Reviewed-by: Harish Chegondi 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib_qp.c|2 +-
 drivers/infiniband/hw/qib/qib_ruc.c   |4 ++--
 drivers/infiniband/hw/qib/qib_srq.c   |   10 +-
 drivers/infiniband/hw/qib/qib_verbs.h |   13 -
 4 files changed, 8 insertions(+), 21 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_qp.c 
b/drivers/infiniband/hw/qib/qib_qp.c
index f3964ef..79907d1 100644
--- a/drivers/infiniband/hw/qib/qib_qp.c
+++ b/drivers/infiniband/hw/qib/qib_qp.c
@@ -1036,7 +1036,7 @@ struct ib_qp *qib_create_qp(struct ib_pd *ibpd,
sz = sizeof(*qp);
sg_list_sz = 0;
if (init_attr->srq) {
-   struct qib_srq *srq = to_isrq(init_attr->srq);
+   struct rvt_srq *srq = ibsrq_to_rvtsrq(init_attr->srq);
 
if (srq->rq.max_sge > 1)
sg_list_sz = sizeof(*qp->r_sg_list) *
diff --git a/drivers/infiniband/hw/qib/qib_ruc.c 
b/drivers/infiniband/hw/qib/qib_ruc.c
index 3900459..682447e 100644
--- a/drivers/infiniband/hw/qib/qib_ruc.c
+++ b/drivers/infiniband/hw/qib/qib_ruc.c
@@ -141,14 +141,14 @@ int qib_get_rwqe(struct rvt_qp *qp, int wr_id_only)
unsigned long flags;
struct rvt_rq *rq;
struct rvt_rwq *wq;
-   struct qib_srq *srq;
+   struct rvt_srq *srq;
struct rvt_rwqe *wqe;
void (*handler)(struct ib_event *, void *);
u32 tail;
int ret;
 
if (qp->ibqp.srq) {
-   srq = to_isrq(qp->ibqp.srq);
+   srq = ibsrq_to_rvtsrq(qp->ibqp.srq);
handler = srq->ibsrq.event_handler;
rq = >rq;
} else {
diff --git a/drivers/infiniband/hw/qib/qib_srq.c 
b/drivers/infiniband/hw/qib/qib_srq.c
index 8547263..e9dfa30 100644
--- a/drivers/infiniband/hw/qib/qib_srq.c
+++ b/drivers/infiniband/hw/qib/qib_srq.c
@@ -48,7 +48,7 @@
 int qib_post_srq_receive(struct ib_srq *ibsrq, struct ib_recv_wr *wr,
 struct ib_recv_wr **bad_wr)
 {
-   struct qib_srq *srq = to_isrq(ibsrq);
+   struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct rvt_rwq *wq;
unsigned long flags;
int ret;
@@ -103,7 +103,7 @@ struct ib_srq *qib_create_srq(struct ib_pd *ibpd,
  struct ib_udata *udata)
 {
struct qib_ibdev *dev = to_idev(ibpd->device);
-   struct qib_srq *srq;
+   struct rvt_srq *srq;
u32 sz;
struct ib_srq *ret;
 
@@ -212,7 +212,7 @@ int qib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr 
*attr,
   enum ib_srq_attr_mask attr_mask,
   struct ib_udata *udata)
 {
-   struct qib_srq *srq = to_isrq(ibsrq);
+   struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct rvt_rwq *wq;
int ret = 0;
 
@@ -350,7 +350,7 @@ bail:
 
 int qib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
 {
-   struct qib_srq *srq = to_isrq(ibsrq);
+   struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
 
attr->max_wr = srq->rq.size - 1;
attr->max_sge = srq->rq.max_sge;
@@ -364,7 +364,7 @@ int qib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr 
*attr)
  */
 int qib_destroy_srq(struct ib_srq *ibsrq)
 {
-   struct qib_srq *srq = to_isrq(ibsrq);
+   struct rvt_srq *srq = ibsrq_to_rvtsrq(ibsrq);
struct qib_ibdev *dev = to_idev(ibsrq->device);
 
spin_lock(>n_srqs_lock);
diff --git a/drivers/infiniband/hw/qib/qib_verbs.h 
b/drivers/infiniband/hw/qib/qib_verbs.h
index a9d7f0a..00dd2ad 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.h
+++ b/drivers/infiniband/hw/qib/qib_verbs.h
@@ -251,14 +251,6 @@ struct qib_cq {
struct rvt_mmap_info *ip;
 };
 
-struct qib_srq {
-   struct ib_srq ibsrq;
-   struct rvt_rq rq;
-   struct rvt_mmap_info *ip;
-   /* send signal when number of RWQEs < limit */
-   u32 limit;
-};
-
 /*
  * qib specific data structure that will be hidden from rvt after the queue 
pair
  * is made common.
@@ -539,11 +531,6 @@ static inline struct qib_cq *to_icq(struct ib_cq *ibcq)
return container_of(ibcq, struct qib_cq, ibcq);
 }
 
-static inline struct qib_srq *to_isrq(struct ib_srq *ibsrq)
-{
-   return container_of(ibsrq, struct qib_srq, ibsrq);
-}
-
 static inline struct rvt_qp *to_iqp(struct ib_qp *ibqp)
 {
return container_of(ibqp, struct rvt_qp, ibqp);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 09/10] IB/qib: Use address handle in rdmavt and remove from qib

[Dennis Dalessandro]

Original patch from Kamal Heib , split
apart from original.

Remove AH from qib and use rdmavt version.

Reviewed-by: Ira Weiny 
Signed-off-by: Kamal Heib 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib_iba7322.c |2 
 drivers/infiniband/hw/qib/qib_mad.c |2 
 drivers/infiniband/hw/qib/qib_qp.c  |6 +
 drivers/infiniband/hw/qib/qib_ruc.c |2 
 drivers/infiniband/hw/qib/qib_ud.c  |4 -
 drivers/infiniband/hw/qib/qib_verbs.c   |  126 ++-
 drivers/infiniband/hw/qib/qib_verbs.h   |   20 -
 7 files changed, 19 insertions(+), 143 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_iba7322.c 
b/drivers/infiniband/hw/qib/qib_iba7322.c
index 6c8ff10..1fbe308 100644
--- a/drivers/infiniband/hw/qib/qib_iba7322.c
+++ b/drivers/infiniband/hw/qib/qib_iba7322.c
@@ -5515,7 +5515,7 @@ static void try_7322_ipg(struct qib_pportdata *ppd)
ret = PTR_ERR(ah);
else {
send_buf->ah = ah;
-   ibp->smi_ah = to_iah(ah);
+   ibp->smi_ah = ibah_to_rvtah(ah);
ret = 0;
}
} else {
diff --git a/drivers/infiniband/hw/qib/qib_mad.c 
b/drivers/infiniband/hw/qib/qib_mad.c
index 70fc1b2..43f8c49 100644
--- a/drivers/infiniband/hw/qib/qib_mad.c
+++ b/drivers/infiniband/hw/qib/qib_mad.c
@@ -109,7 +109,7 @@ static void qib_send_trap(struct qib_ibport *ibp, void 
*data, unsigned len)
ret = PTR_ERR(ah);
else {
send_buf->ah = ah;
-   ibp->sm_ah = to_iah(ah);
+   ibp->sm_ah = ibah_to_rvtah(ah);
ret = 0;
}
} else
diff --git a/drivers/infiniband/hw/qib/qib_qp.c 
b/drivers/infiniband/hw/qib/qib_qp.c
index 143924d..f3964ef 100644
--- a/drivers/infiniband/hw/qib/qib_qp.c
+++ b/drivers/infiniband/hw/qib/qib_qp.c
@@ -438,7 +438,7 @@ static void clear_mr_refs(struct rvt_qp *qp, int clr_sends)
if (qp->ibqp.qp_type == IB_QPT_UD ||
qp->ibqp.qp_type == IB_QPT_SMI ||
qp->ibqp.qp_type == IB_QPT_GSI)
-   atomic_dec(_iah(wqe->ud_wr.ah)->refcount);
+   
atomic_dec(_to_rvtah(wqe->ud_wr.ah)->refcount);
if (++qp->s_last >= qp->s_size)
qp->s_last = 0;
}
@@ -595,7 +595,7 @@ int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
if (attr_mask & IB_QP_AV) {
if (attr->ah_attr.dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
goto inval;
-   if (qib_check_ah(qp->ibqp.device, >ah_attr))
+   if (rvt_check_ah(qp->ibqp.device, >ah_attr))
goto inval;
}
 
@@ -603,7 +603,7 @@ int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
if (attr->alt_ah_attr.dlid >=
be16_to_cpu(IB_MULTICAST_LID_BASE))
goto inval;
-   if (qib_check_ah(qp->ibqp.device, >alt_ah_attr))
+   if (rvt_check_ah(qp->ibqp.device, >alt_ah_attr))
goto inval;
if (attr->alt_pkey_index >= qib_get_npkeys(dd_from_dev(dev)))
goto inval;
diff --git a/drivers/infiniband/hw/qib/qib_ruc.c 
b/drivers/infiniband/hw/qib/qib_ruc.c
index 02e79a8..3900459 100644
--- a/drivers/infiniband/hw/qib/qib_ruc.c
+++ b/drivers/infiniband/hw/qib/qib_ruc.c
@@ -789,7 +789,7 @@ void qib_send_complete(struct rvt_qp *qp, struct rvt_swqe 
*wqe,
if (qp->ibqp.qp_type == IB_QPT_UD ||
qp->ibqp.qp_type == IB_QPT_SMI ||
qp->ibqp.qp_type == IB_QPT_GSI)
-   atomic_dec(_iah(wqe->ud_wr.ah)->refcount);
+   atomic_dec(_to_rvtah(wqe->ud_wr.ah)->refcount);
 
/* See ch. 11.2.4.1 and 10.7.3.1 */
if (!(qp->s_flags & QIB_S_SIGNAL_REQ_WR) ||
diff --git a/drivers/infiniband/hw/qib/qib_ud.c 
b/drivers/infiniband/hw/qib/qib_ud.c
index 1d9d037..6dc20ca 100644
--- a/drivers/infiniband/hw/qib/qib_ud.c
+++ b/drivers/infiniband/hw/qib/qib_ud.c
@@ -77,7 +77,7 @@ static void qib_ud_loopback(struct rvt_qp *sqp, struct 
rvt_swqe *swqe)
goto drop;
}
 
-   ah_attr = _iah(swqe->ud_wr.ah)->attr;
+   ah_attr = _to_rvtah(swqe->ud_wr.ah)->attr;
ppd = ppd_from_ibp(ibp);
 
if (qp->ibqp.qp_num > 1) {
@@ -279,7 +279,7 @@ int qib_make_ud_req(struct rvt_qp *qp)
/* Construct the header. */
ibp = to_iport(qp->ibqp.device, qp->port_num);
ppd = ppd_from_ibp(ibp);
-   ah_attr = _iah(wqe->ud_wr.ah)->attr;
+   ah_attr = 

[PATCH v2 08/10] IB/qib: Remove qp and mr functionality from qib

[Dennis Dalessandro]

Remove qp and mr support from qib and use rdmavt. These two changes
cannot be reasonably be split apart into separate patches because they
depend on eachother in multple places. This paves the way to remove
even more functions in subsequent patches.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/Makefile  |4 
 drivers/infiniband/hw/qib/qib.h |   14 -
 drivers/infiniband/hw/qib/qib_cq.c  |2 
 drivers/infiniband/hw/qib/qib_driver.c  |4 
 drivers/infiniband/hw/qib/qib_keys.c|  184 +
 drivers/infiniband/hw/qib/qib_mmap.c|   18 -
 drivers/infiniband/hw/qib/qib_mr.c  |  539 ---
 drivers/infiniband/hw/qib/qib_qp.c  |   72 ++--
 drivers/infiniband/hw/qib/qib_rc.c  |   98 ++---
 drivers/infiniband/hw/qib/qib_ruc.c |   58 +--
 drivers/infiniband/hw/qib/qib_sdma.c|8 
 drivers/infiniband/hw/qib/qib_srq.c |   28 +
 drivers/infiniband/hw/qib/qib_uc.c  |8 
 drivers/infiniband/hw/qib/qib_ud.c  |   16 -
 drivers/infiniband/hw/qib/qib_verbs.c   |  175 -
 drivers/infiniband/hw/qib/qib_verbs.h   |  436 +++---
 drivers/infiniband/hw/qib/qib_verbs_mcast.c |8 
 17 files changed, 308 insertions(+), 1364 deletions(-)
 delete mode 100644 drivers/infiniband/hw/qib/qib_mr.c

diff --git a/drivers/infiniband/hw/qib/Makefile 
b/drivers/infiniband/hw/qib/Makefile
index 8a8f892..75140f5 100644
--- a/drivers/infiniband/hw/qib/Makefile
+++ b/drivers/infiniband/hw/qib/Makefile
@@ -1,8 +1,8 @@
 obj-$(CONFIG_INFINIBAND_QIB) += ib_qib.o
 
 ib_qib-y := qib_cq.o qib_diag.o qib_driver.o qib_eeprom.o \
-   qib_file_ops.o qib_fs.o qib_init.o qib_intr.o qib_keys.o \
-   qib_mad.o qib_mmap.o qib_mr.o qib_pcie.o qib_pio_copy.o \
+   qib_file_ops.o qib_fs.o qib_init.o qib_intr.o \
+   qib_mad.o qib_mmap.o qib_pcie.o qib_pio_copy.o \
qib_qp.o qib_qsfp.o qib_rc.o qib_ruc.o qib_sdma.o qib_srq.o \
qib_sysfs.o qib_twsi.o qib_tx.o qib_uc.o qib_ud.o \
qib_user_pages.o qib_user_sdma.o qib_verbs_mcast.o qib_iba7220.o \
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index e610eaf..309b6f3 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -231,7 +231,7 @@ struct qib_ctxtdata {
/* ctxt rcvhdrq head offset */
u32 head;
/* lookaside fields */
-   struct qib_qp *lookaside_qp;
+   struct rvt_qp *lookaside_qp;
u32 lookaside_qpn;
/* QPs waiting for context processing */
struct list_head qp_wait_list;
@@ -241,7 +241,7 @@ struct qib_ctxtdata {
 #endif
 };
 
-struct qib_sge_state;
+struct rvt_sge_state;
 
 struct qib_sdma_txreq {
int flags;
@@ -259,14 +259,14 @@ struct qib_sdma_desc {
 
 struct qib_verbs_txreq {
struct qib_sdma_txreq   txreq;
-   struct qib_qp   *qp;
-   struct qib_swqe *wqe;
+   struct rvt_qp   *qp;
+   struct rvt_swqe *wqe;
u32 dwords;
u16 hdr_dwords;
u16 hdr_inx;
struct qib_pio_header   *align_buf;
-   struct qib_mregion  *mr;
-   struct qib_sge_state*ss;
+   struct rvt_mregion  *mr;
+   struct rvt_sge_state*ss;
 };
 
 #define QIB_SDMA_TXREQ_F_USELARGEBUF  0x1
@@ -1324,7 +1324,7 @@ void __qib_sdma_intr(struct qib_pportdata *);
 void qib_sdma_intr(struct qib_pportdata *);
 void qib_user_sdma_send_desc(struct qib_pportdata *dd,
struct list_head *pktlist);
-int qib_sdma_verbs_send(struct qib_pportdata *, struct qib_sge_state *,
+int qib_sdma_verbs_send(struct qib_pportdata *, struct rvt_sge_state *,
u32, struct qib_verbs_txreq *);
 /* ppd->sdma_lock should be locked before calling this. */
 int qib_sdma_make_progress(struct qib_pportdata *dd);
diff --git a/drivers/infiniband/hw/qib/qib_cq.c 
b/drivers/infiniband/hw/qib/qib_cq.c
index 2b45d0b..c1ea21e 100644
--- a/drivers/infiniband/hw/qib/qib_cq.c
+++ b/drivers/infiniband/hw/qib/qib_cq.c
@@ -466,7 +466,7 @@ int qib_resize_cq(struct ib_cq *ibcq, int cqe, struct 
ib_udata *udata)
 
if (cq->ip) {
struct qib_ibdev *dev = to_idev(ibcq->device);
-   struct qib_mmap_info *ip = cq->ip;
+   struct rvt_mmap_info *ip = cq->ip;
 
qib_update_mmap_info(dev, ip, sz, wc);
 
diff --git a/drivers/infiniband/hw/qib/qib_driver.c 
b/drivers/infiniband/hw/qib/qib_driver.c
index ae5a725..eafdee9 100644
--- a/drivers/infiniband/hw/qib/qib_driver.c
+++ b/drivers/infiniband/hw/qib/qib_driver.c
@@ -322,7 +322,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct 
qib_pportdata *ppd,
struct 

[PATCH v2 06/10] IB/qib: Remove driver specific members from qib qp type

[Dennis Dalessandro]

In preparation for moving the queue pair data structure to rdmavt the
members of the driver specific queue pairs which are not common need to be
pushed off to a private driver structure. This structure will be available
in the queue pair once moved to rdmavt as a void pointer. This patch while
not adding a lot of value in and of itself is a prerequisite to move the
queue pair out of the drivers and into rdmavt.

The driver specific, private queue pair data structure should condense as
more of the send side code moves to rdmavt.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib_qp.c|   75 -
 drivers/infiniband/hw/qib/qib_rc.c|7 ++-
 drivers/infiniband/hw/qib/qib_ruc.c   |   18 +---
 drivers/infiniband/hw/qib/qib_sdma.c  |   17 +--
 drivers/infiniband/hw/qib/qib_uc.c|7 ++-
 drivers/infiniband/hw/qib/qib_ud.c|   22 +-
 drivers/infiniband/hw/qib/qib_verbs.c |   61 +--
 drivers/infiniband/hw/qib/qib_verbs.h |   26 ---
 8 files changed, 145 insertions(+), 88 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_qp.c 
b/drivers/infiniband/hw/qib/qib_qp.c
index 116d323..7e570be 100644
--- a/drivers/infiniband/hw/qib/qib_qp.c
+++ b/drivers/infiniband/hw/qib/qib_qp.c
@@ -370,10 +370,11 @@ struct qib_qp *qib_lookup_qpn(struct qib_ibport *ibp, u32 
qpn)
  */
 static void qib_reset_qp(struct qib_qp *qp, enum ib_qp_type type)
 {
+   struct qib_qp_priv *priv = qp->priv;
qp->remote_qpn = 0;
qp->qkey = 0;
qp->qp_access_flags = 0;
-   atomic_set(>s_dma_busy, 0);
+   atomic_set(>s_dma_busy, 0);
qp->s_flags &= QIB_S_SIGNAL_REQ_WR;
qp->s_hdrwords = 0;
qp->s_wqe = NULL;
@@ -473,6 +474,7 @@ static void clear_mr_refs(struct qib_qp *qp, int clr_sends)
  */
 int qib_error_qp(struct qib_qp *qp, enum ib_wc_status err)
 {
+   struct qib_qp_priv *priv = qp->priv;
struct qib_ibdev *dev = to_idev(qp->ibqp.device);
struct ib_wc wc;
int ret = 0;
@@ -491,9 +493,9 @@ int qib_error_qp(struct qib_qp *qp, enum ib_wc_status err)
qp->s_flags &= ~QIB_S_ANY_WAIT_SEND;
 
spin_lock(>pending_lock);
-   if (!list_empty(>iowait) && !(qp->s_flags & QIB_S_BUSY)) {
+   if (!list_empty(>iowait) && !(qp->s_flags & QIB_S_BUSY)) {
qp->s_flags &= ~QIB_S_ANY_WAIT_IO;
-   list_del_init(>iowait);
+   list_del_init(>iowait);
}
spin_unlock(>pending_lock);
 
@@ -503,9 +505,9 @@ int qib_error_qp(struct qib_qp *qp, enum ib_wc_status err)
qib_put_mr(qp->s_rdma_mr);
qp->s_rdma_mr = NULL;
}
-   if (qp->s_tx) {
-   qib_put_txreq(qp->s_tx);
-   qp->s_tx = NULL;
+   if (priv->s_tx) {
+   qib_put_txreq(priv->s_tx);
+   priv->s_tx = NULL;
}
}
 
@@ -571,6 +573,7 @@ int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
 {
struct qib_ibdev *dev = to_idev(ibqp->device);
struct qib_qp *qp = to_iqp(ibqp);
+   struct qib_qp_priv *priv = qp->priv;
enum ib_qp_state cur_state, new_state;
struct ib_event ev;
int lastwqe = 0;
@@ -698,19 +701,20 @@ int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
if (qp->state != IB_QPS_RESET) {
qp->state = IB_QPS_RESET;
spin_lock(>pending_lock);
-   if (!list_empty(>iowait))
-   list_del_init(>iowait);
+   if (!list_empty(>iowait))
+   list_del_init(>iowait);
spin_unlock(>pending_lock);
qp->s_flags &= ~(QIB_S_TIMER | QIB_S_ANY_WAIT);
spin_unlock(>s_lock);
spin_unlock_irq(>r_lock);
/* Stop the sending work queue and retry timer */
-   cancel_work_sync(>s_work);
+   cancel_work_sync(>s_work);
del_timer_sync(>s_timer);
-   wait_event(qp->wait_dma, !atomic_read(>s_dma_busy));
-   if (qp->s_tx) {
-   qib_put_txreq(qp->s_tx);
-   qp->s_tx = NULL;
+   wait_event(priv->wait_dma,
+  !atomic_read(>s_dma_busy));
+   if (priv->s_tx) {
+   qib_put_txreq(priv->s_tx);
+   priv->s_tx = NULL;
}
remove_qp(dev, qp);
wait_event(qp->wait, 

[PATCH v2 04/10] IB/qib: Remove most uses of QIB_PERMISSIVE_LID and QIB_MULTICAST_LID_BASE

[Dennis Dalessandro]

This patch removes most of the uses of QIB_PERMISSIBVE_LID and
QIB_MULTICAST_LID_BASE in favor of the recently added IB_* versions.
There are still minor uses in AH functions as well as the QIB_* defines
but those will be removed in a follow on patch.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib_driver.c |2 +-
 drivers/infiniband/hw/qib/qib_mad.c|4 ++--
 drivers/infiniband/hw/qib/qib_qp.c |5 +++--
 drivers/infiniband/hw/qib/qib_ud.c |   11 ++-
 drivers/infiniband/hw/qib/qib_verbs.c  |2 +-
 5 files changed, 13 insertions(+), 11 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_driver.c 
b/drivers/infiniband/hw/qib/qib_driver.c
index f58fdc3..57b19ea 100644
--- a/drivers/infiniband/hw/qib/qib_driver.c
+++ b/drivers/infiniband/hw/qib/qib_driver.c
@@ -319,7 +319,7 @@ static u32 qib_rcv_hdrerr(struct qib_ctxtdata *rcd, struct 
qib_pportdata *ppd,
if (tlen < 24)
goto drop;
 
-   if (lid < QIB_MULTICAST_LID_BASE) {
+   if (lid < be16_to_cpu(IB_MULTICAST_LID_BASE)) {
lid &= ~((1 << ppd->lmc) - 1);
if (unlikely(lid != ppd->lid))
goto drop;
diff --git a/drivers/infiniband/hw/qib/qib_mad.c 
b/drivers/infiniband/hw/qib/qib_mad.c
index c65d3aa..70fc1b2 100644
--- a/drivers/infiniband/hw/qib/qib_mad.c
+++ b/drivers/infiniband/hw/qib/qib_mad.c
@@ -691,7 +691,7 @@ static int subn_set_portinfo(struct ib_smp *smp, struct 
ib_device *ibdev,
 
lid = be16_to_cpu(pip->lid);
/* Must be a valid unicast LID address. */
-   if (lid == 0 || lid >= QIB_MULTICAST_LID_BASE)
+   if (lid == 0 || lid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
smp->status |= IB_SMP_INVALID_FIELD;
else if (ppd->lid != lid || ppd->lmc != (pip->mkeyprot_resv_lmc & 7)) {
if (ppd->lid != lid)
@@ -706,7 +706,7 @@ static int subn_set_portinfo(struct ib_smp *smp, struct 
ib_device *ibdev,
smlid = be16_to_cpu(pip->sm_lid);
msl = pip->neighbormtu_mastersmsl & 0xF;
/* Must be a valid unicast LID address. */
-   if (smlid == 0 || smlid >= QIB_MULTICAST_LID_BASE)
+   if (smlid == 0 || smlid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
smp->status |= IB_SMP_INVALID_FIELD;
else if (smlid != ibp->sm_lid || msl != ibp->sm_sl) {
spin_lock_irqsave(>lock, flags);
diff --git a/drivers/infiniband/hw/qib/qib_qp.c 
b/drivers/infiniband/hw/qib/qib_qp.c
index 40f85bb..e6544dc 100644
--- a/drivers/infiniband/hw/qib/qib_qp.c
+++ b/drivers/infiniband/hw/qib/qib_qp.c
@@ -589,14 +589,15 @@ int qib_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr 
*attr,
goto inval;
 
if (attr_mask & IB_QP_AV) {
-   if (attr->ah_attr.dlid >= QIB_MULTICAST_LID_BASE)
+   if (attr->ah_attr.dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE))
goto inval;
if (qib_check_ah(qp->ibqp.device, >ah_attr))
goto inval;
}
 
if (attr_mask & IB_QP_ALT_PATH) {
-   if (attr->alt_ah_attr.dlid >= QIB_MULTICAST_LID_BASE)
+   if (attr->alt_ah_attr.dlid >=
+   be16_to_cpu(IB_MULTICAST_LID_BASE))
goto inval;
if (qib_check_ah(qp->ibqp.device, >alt_ah_attr))
goto inval;
diff --git a/drivers/infiniband/hw/qib/qib_ud.c 
b/drivers/infiniband/hw/qib/qib_ud.c
index 59193f6..32fc80c 100644
--- a/drivers/infiniband/hw/qib/qib_ud.c
+++ b/drivers/infiniband/hw/qib/qib_ud.c
@@ -32,6 +32,7 @@
  */
 
 #include 
+#include 
 
 #include "qib.h"
 #include "qib_mad.h"
@@ -278,8 +279,8 @@ int qib_make_ud_req(struct qib_qp *qp)
ibp = to_iport(qp->ibqp.device, qp->port_num);
ppd = ppd_from_ibp(ibp);
ah_attr = _iah(wqe->ud_wr.ah)->attr;
-   if (ah_attr->dlid >= QIB_MULTICAST_LID_BASE) {
-   if (ah_attr->dlid != QIB_PERMISSIVE_LID)
+   if (ah_attr->dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE)) {
+   if (ah_attr->dlid != be16_to_cpu(IB_LID_PERMISSIVE))
this_cpu_inc(ibp->pmastats->n_multicast_xmit);
else
this_cpu_inc(ibp->pmastats->n_unicast_xmit);
@@ -368,8 +369,8 @@ int qib_make_ud_req(struct qib_qp *qp)
/*
 * Use the multicast QP if the destination LID is a multicast LID.
 */
-   ohdr->bth[1] = ah_attr->dlid >= QIB_MULTICAST_LID_BASE &&
-   ah_attr->dlid != QIB_PERMISSIVE_LID ?
+   ohdr->bth[1] = ah_attr->dlid >= be16_to_cpu(IB_MULTICAST_LID_BASE) &&
+   ah_attr->dlid != be16_to_cpu(IB_LID_PERMISSIVE) ?
cpu_to_be32(QIB_MULTICAST_QPN) :

[PATCH v2 01/10] IB/qib: Begin to use rdmavt for verbs

[Dennis Dalessandro]

This patch begins to make use of rdmavt by registering with it and
providing access to the header files. This is just the beginning of
rdmavt support in qib.

Most functionality is still being done in the driver, set flags so that
rdmavt will let qib continue to handle mr, qp, and cq init.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/Kconfig |2 +-
 drivers/infiniband/hw/qib/qib.h   |1 +
 drivers/infiniband/hw/qib/qib_init.c  |5 +++--
 drivers/infiniband/hw/qib/qib_intr.c  |2 +-
 drivers/infiniband/hw/qib/qib_mad.c   |5 +++--
 drivers/infiniband/hw/qib/qib_sysfs.c |   20 ++--
 drivers/infiniband/hw/qib/qib_verbs.c |   19 ++-
 drivers/infiniband/hw/qib/qib_verbs.h |8 ++--
 8 files changed, 39 insertions(+), 23 deletions(-)

diff --git a/drivers/infiniband/hw/qib/Kconfig 
b/drivers/infiniband/hw/qib/Kconfig
index 495be09..e0fdb92 100644
--- a/drivers/infiniband/hw/qib/Kconfig
+++ b/drivers/infiniband/hw/qib/Kconfig
@@ -1,6 +1,6 @@
 config INFINIBAND_QIB
tristate "Intel PCIe HCA support"
-   depends on 64BIT
+   depends on 64BIT && INFINIBAND_RDMAVT
---help---
This is a low-level driver for Intel PCIe QLE InfiniBand host
channel adapters.  This driver does not support the Intel
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index 7df16f7..0e68e1f 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -52,6 +52,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "qib_common.h"
 #include "qib_verbs.h"
diff --git a/drivers/infiniband/hw/qib/qib_init.c 
b/drivers/infiniband/hw/qib/qib_init.c
index 4ff340f..47190f1 100644
--- a/drivers/infiniband/hw/qib/qib_init.c
+++ b/drivers/infiniband/hw/qib/qib_init.c
@@ -42,6 +42,7 @@
 #ifdef CONFIG_INFINIBAND_QIB_DCA
 #include 
 #endif
+#include 
 
 #include "qib.h"
 #include "qib_common.h"
@@ -1081,7 +1082,7 @@ void qib_free_devdata(struct qib_devdata *dd)
qib_dbg_ibdev_exit(>verbs_dev);
 #endif
free_percpu(dd->int_counter);
-   ib_dealloc_device(>verbs_dev.ibdev);
+   ib_dealloc_device(>verbs_dev.rdi.ibdev);
 }
 
 u64 qib_int_counter(struct qib_devdata *dd)
@@ -1171,7 +1172,7 @@ struct qib_devdata *qib_alloc_devdata(struct pci_dev 
*pdev, size_t extra)
 bail:
if (!list_empty(>list))
list_del_init(>list);
-   ib_dealloc_device(>verbs_dev.ibdev);
+   ib_dealloc_device(>verbs_dev.rdi.ibdev);
return ERR_PTR(ret);
 }
 
diff --git a/drivers/infiniband/hw/qib/qib_intr.c 
b/drivers/infiniband/hw/qib/qib_intr.c
index 086616d..a014fd4 100644
--- a/drivers/infiniband/hw/qib/qib_intr.c
+++ b/drivers/infiniband/hw/qib/qib_intr.c
@@ -74,7 +74,7 @@ static void signal_ib_event(struct qib_pportdata *ppd, enum 
ib_event_type ev)
struct ib_event event;
struct qib_devdata *dd = ppd->dd;
 
-   event.device = >verbs_dev.ibdev;
+   event.device = >verbs_dev.rdi.ibdev;
event.element.port_num = ppd->port;
event.event = ev;
ib_dispatch_event();
diff --git a/drivers/infiniband/hw/qib/qib_mad.c 
b/drivers/infiniband/hw/qib/qib_mad.c
index 9625e7c..c65d3aa 100644
--- a/drivers/infiniband/hw/qib/qib_mad.c
+++ b/drivers/infiniband/hw/qib/qib_mad.c
@@ -1028,7 +1028,7 @@ static int set_pkeys(struct qib_devdata *dd, u8 port, u16 
*pkeys)
(void) dd->f_set_ib_cfg(ppd, QIB_IB_CFG_PKEYS, 0);
 
event.event = IB_EVENT_PKEY_CHANGE;
-   event.device = >verbs_dev.ibdev;
+   event.device = >verbs_dev.rdi.ibdev;
event.element.port_num = port;
ib_dispatch_event();
}
@@ -2483,7 +2483,8 @@ int qib_create_agents(struct qib_ibdev *dev)
 
for (p = 0; p < dd->num_pports; p++) {
ibp = >pport[p].ibport_data;
-   agent = ib_register_mad_agent(>ibdev, p + 1, IB_QPT_SMI,
+   agent = ib_register_mad_agent(>rdi.ibdev, p + 1,
+ IB_QPT_SMI,
  NULL, 0, send_handler,
  NULL, NULL, 0);
if (IS_ERR(agent)) {
diff --git a/drivers/infiniband/hw/qib/qib_sysfs.c 
b/drivers/infiniband/hw/qib/qib_sysfs.c
index 81f56cd..72a160e 100644
--- a/drivers/infiniband/hw/qib/qib_sysfs.c
+++ b/drivers/infiniband/hw/qib/qib_sysfs.c
@@ -502,7 +502,7 @@ static ssize_t show_rev(struct device *device, struct 
device_attribute *attr,
char *buf)
 {
struct qib_ibdev *dev =
-   container_of(device, struct qib_ibdev, ibdev.dev);
+   container_of(device, struct qib_ibdev, rdi.ibdev.dev);
 
return sprintf(buf, "%x\n", dd_from_dev(dev)->minrev);
 }
@@ -511,7 +511,7 @@ static 

[PATCH v2 00/10] Begin to use rdmavt for qib

[Dennis Dalessandro]

The following series implements the beginning support of rdmavt in the qib
driver. The memory registration and protection domain functionality has been
moved out of the driver as well as critical qp data structures. More patches
will follow to incrementally remove code from qib and use the rdmavt
implementation.

This patch builds on the v2 "Add rdma verbs transport library" patch.

Changes since v1:
Rebased onto what appears to be the latest branch on Doug's kernel.org repo
 (k.o/for-4.4-rc). I can rebase onto something else if needed.

---

Dennis Dalessandro (10):
  IB/qib: Begin to use rdmavt for verbs
  IB/qib: Remove dma.c and use rdmavt version of dma functions
  IB/qib: Use rdmavt protection domain
  IB/qib: Remove most uses of QIB_PERMISSIVE_LID and QIB_MULTICAST_LID_BASE
  IB/qib: Use rdmavt lid defines in qib
  IB/qib: Remove driver specific members from qib qp type
  IB/qib: Add device specific info prints
  IB/qib: Remove qp and mr functionality from qib
  IB/qib: Use address handle in rdmavt and remove from qib
  IB/qib: Remove srq from qib


 drivers/infiniband/hw/qib/Kconfig   |2 
 drivers/infiniband/hw/qib/Makefile  |6 
 drivers/infiniband/hw/qib/qib.h |   17 +
 drivers/infiniband/hw/qib/qib_common.h  |2 
 drivers/infiniband/hw/qib/qib_cq.c  |2 
 drivers/infiniband/hw/qib/qib_driver.c  |   22 +
 drivers/infiniband/hw/qib/qib_iba7322.c |2 
 drivers/infiniband/hw/qib/qib_init.c|5 
 drivers/infiniband/hw/qib/qib_intr.c|2 
 drivers/infiniband/hw/qib/qib_keys.c|  186 +
 drivers/infiniband/hw/qib/qib_mad.c |   11 -
 drivers/infiniband/hw/qib/qib_mmap.c|   18 -
 drivers/infiniband/hw/qib/qib_mr.c  |  539 ---
 drivers/infiniband/hw/qib/qib_qp.c  |  161 +---
 drivers/infiniband/hw/qib/qib_rc.c  |  105 +++--
 drivers/infiniband/hw/qib/qib_ruc.c |   84 ++--
 drivers/infiniband/hw/qib/qib_sdma.c|   25 +
 drivers/infiniband/hw/qib/qib_srq.c |   38 +-
 drivers/infiniband/hw/qib/qib_sysfs.c   |   20 +
 drivers/infiniband/hw/qib/qib_uc.c  |   15 -
 drivers/infiniband/hw/qib/qib_ud.c  |   53 +--
 drivers/infiniband/hw/qib/qib_verbs.c   |  448 +++---
 drivers/infiniband/hw/qib/qib_verbs.h   |  488 +++-
 drivers/infiniband/hw/qib/qib_verbs_mcast.c |8 
 24 files changed, 546 insertions(+), 1713 deletions(-)
 delete mode 100644 drivers/infiniband/hw/qib/qib_mr.c

-- 
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 07/10] IB/qib: Add device specific info prints

[Dennis Dalessandro]

Implement get_card_name and get_pci_dev helper functions for rdmavt
for qib.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib.h|2 ++
 drivers/infiniband/hw/qib/qib_driver.c |   16 
 drivers/infiniband/hw/qib/qib_verbs.c  |2 ++
 3 files changed, 20 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index 0e68e1f..e610eaf 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -1455,6 +1455,8 @@ u64 qib_sps_ints(void);
 dma_addr_t qib_map_page(struct pci_dev *, struct page *, unsigned long,
  size_t, int);
 const char *qib_get_unit_name(int unit);
+const char *qib_get_card_name(struct rvt_dev_info *rdi);
+struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi);
 
 /*
  * Flush write combining store buffers (if present) and perform a write
diff --git a/drivers/infiniband/hw/qib/qib_driver.c 
b/drivers/infiniband/hw/qib/qib_driver.c
index 57b19ea..ae5a725 100644
--- a/drivers/infiniband/hw/qib/qib_driver.c
+++ b/drivers/infiniband/hw/qib/qib_driver.c
@@ -90,6 +90,22 @@ const char *qib_get_unit_name(int unit)
return iname;
 }
 
+const char *qib_get_card_name(struct rvt_dev_info *rdi)
+{
+   struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
+   struct qib_devdata *dd = container_of(ibdev,
+ struct qib_devdata, verbs_dev);
+   return qib_get_unit_name(dd->unit);
+}
+
+struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi)
+{
+   struct qib_ibdev *ibdev = container_of(rdi, struct qib_ibdev, rdi);
+   struct qib_devdata *dd = container_of(ibdev,
+ struct qib_devdata, verbs_dev);
+   return dd->pcidev;
+}
+
 /*
  * Return count of units with at least one port ACTIVE.
  */
diff --git a/drivers/infiniband/hw/qib/qib_verbs.c 
b/drivers/infiniband/hw/qib/qib_verbs.c
index 32c2811..b7e03ae 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.c
+++ b/drivers/infiniband/hw/qib/qib_verbs.c
@@ -2240,6 +2240,8 @@ int qib_register_ib_device(struct qib_devdata *dd)
 * Fill in rvt info object.
 */
dd->verbs_dev.rdi.driver_f.port_callback = qib_create_port_files;
+   dd->verbs_dev.rdi.driver_f.get_card_name = qib_get_card_name;
+   dd->verbs_dev.rdi.driver_f.get_pci_dev = qib_get_pci_dev;
dd->verbs_dev.rdi.dparms.props.max_pd = ib_qib_max_pds;
dd->verbs_dev.rdi.flags = (RVT_FLAG_MR_INIT_DRIVER |
   RVT_FLAG_QP_INIT_DRIVER |

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 03/10] IB/qib: Use rdmavt protection domain

[Dennis Dalessandro]

Remove protection domain datastructure from qib and use rdmavts version.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/qib_keys.c  |6 ++-
 drivers/infiniband/hw/qib/qib_mr.c|2 +
 drivers/infiniband/hw/qib/qib_ruc.c   |4 +-
 drivers/infiniband/hw/qib/qib_verbs.c |   66 +++--
 drivers/infiniband/hw/qib/qib_verbs.h |   13 +--
 5 files changed, 12 insertions(+), 79 deletions(-)

diff --git a/drivers/infiniband/hw/qib/qib_keys.c 
b/drivers/infiniband/hw/qib/qib_keys.c
index d725c56..04fa272 100644
--- a/drivers/infiniband/hw/qib/qib_keys.c
+++ b/drivers/infiniband/hw/qib/qib_keys.c
@@ -152,7 +152,7 @@ out:
  * Check the IB SGE for validity and initialize our internal version
  * of it.
  */
-int qib_lkey_ok(struct qib_lkey_table *rkt, struct qib_pd *pd,
+int qib_lkey_ok(struct qib_lkey_table *rkt, struct rvt_pd *pd,
struct qib_sge *isge, struct ib_sge *sge, int acc)
 {
struct qib_mregion *mr;
@@ -263,7 +263,7 @@ int qib_rkey_ok(struct qib_qp *qp, struct qib_sge *sge,
 */
rcu_read_lock();
if (rkey == 0) {
-   struct qib_pd *pd = to_ipd(qp->ibqp.pd);
+   struct rvt_pd *pd = ibpd_to_rvtpd(qp->ibqp.pd);
struct qib_ibdev *dev = to_idev(pd->ibpd.device);
 
if (pd->user)
@@ -341,7 +341,7 @@ bail:
 int qib_reg_mr(struct qib_qp *qp, struct ib_reg_wr *wr)
 {
struct qib_lkey_table *rkt = _idev(qp->ibqp.device)->lk_table;
-   struct qib_pd *pd = to_ipd(qp->ibqp.pd);
+   struct rvt_pd *pd = ibpd_to_rvtpd(qp->ibqp.pd);
struct qib_mr *mr = to_imr(wr->mr);
struct qib_mregion *mrg;
u32 key = wr->key;
diff --git a/drivers/infiniband/hw/qib/qib_mr.c 
b/drivers/infiniband/hw/qib/qib_mr.c
index 294f5c7..7fe4159 100644
--- a/drivers/infiniband/hw/qib/qib_mr.c
+++ b/drivers/infiniband/hw/qib/qib_mr.c
@@ -99,7 +99,7 @@ struct ib_mr *qib_get_dma_mr(struct ib_pd *pd, int acc)
struct ib_mr *ret;
int rval;
 
-   if (to_ipd(pd)->user) {
+   if (ibpd_to_rvtpd(pd)->user) {
ret = ERR_PTR(-EPERM);
goto bail;
}
diff --git a/drivers/infiniband/hw/qib/qib_ruc.c 
b/drivers/infiniband/hw/qib/qib_ruc.c
index b1aa21b..425c8c2 100644
--- a/drivers/infiniband/hw/qib/qib_ruc.c
+++ b/drivers/infiniband/hw/qib/qib_ruc.c
@@ -84,11 +84,11 @@ static int qib_init_sge(struct qib_qp *qp, struct qib_rwqe 
*wqe)
int i, j, ret;
struct ib_wc wc;
struct qib_lkey_table *rkt;
-   struct qib_pd *pd;
+   struct rvt_pd *pd;
struct qib_sge_state *ss;
 
rkt = _idev(qp->ibqp.device)->lk_table;
-   pd = to_ipd(qp->ibqp.srq ? qp->ibqp.srq->pd : qp->ibqp.pd);
+   pd = ibpd_to_rvtpd(qp->ibqp.srq ? qp->ibqp.srq->pd : qp->ibqp.pd);
ss = >r_sge;
ss->sg_list = qp->r_sg_list;
qp->r_len = 0;
diff --git a/drivers/infiniband/hw/qib/qib_verbs.c 
b/drivers/infiniband/hw/qib/qib_verbs.c
index fb4f045..7711d64 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.c
+++ b/drivers/infiniband/hw/qib/qib_verbs.c
@@ -346,7 +346,7 @@ static int qib_post_one_send(struct qib_qp *qp, struct 
ib_send_wr *wr,
int ret;
unsigned long flags;
struct qib_lkey_table *rkt;
-   struct qib_pd *pd;
+   struct rvt_pd *pd;
 
spin_lock_irqsave(>s_lock, flags);
 
@@ -396,7 +396,7 @@ static int qib_post_one_send(struct qib_qp *qp, struct 
ib_send_wr *wr,
}
 
rkt = _idev(qp->ibqp.device)->lk_table;
-   pd = to_ipd(qp->ibqp.pd);
+   pd = ibpd_to_rvtpd(qp->ibqp.pd);
wqe = get_swqe_ptr(qp, qp->s_head);
 
if (qp->ibqp.qp_type != IB_QPT_UC &&
@@ -1599,7 +1599,7 @@ static int qib_query_device(struct ib_device *ibdev, 
struct ib_device_attr *prop
props->max_mr = dev->lk_table.max;
props->max_fmr = dev->lk_table.max;
props->max_map_per_fmr = 32767;
-   props->max_pd = ib_qib_max_pds;
+   props->max_pd = dev->rdi.dparms.props.max_pd;
props->max_qp_rd_atom = QIB_MAX_RDMA_ATOMIC;
props->max_qp_init_rd_atom = 255;
/* props->max_res_rd_atom */
@@ -1751,61 +1751,6 @@ static int qib_query_gid(struct ib_device *ibdev, u8 
port,
return ret;
 }
 
-static struct ib_pd *qib_alloc_pd(struct ib_device *ibdev,
- struct ib_ucontext *context,
- struct ib_udata *udata)
-{
-   struct qib_ibdev *dev = to_idev(ibdev);
-   struct qib_pd *pd;
-   struct ib_pd *ret;
-
-   /*
-* This is actually totally arbitrary.  Some correctness tests
-* assume there's a maximum number of PDs that can be allocated.
-* We don't actually have this limit, but we fail the test if
-* we allow allocations of more than we report for this 

[PATCH v2 02/10] IB/qib: Remove dma.c and use rdmavt version of dma functions

[Dennis Dalessandro]

This patch removes the qib_dma.c file and uses the version which has been
added to rdmavt.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/hw/qib/Makefile|2 +-
 drivers/infiniband/hw/qib/qib_verbs.c |3 ++-
 drivers/infiniband/hw/qib/qib_verbs.h |2 --
 3 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/qib/Makefile 
b/drivers/infiniband/hw/qib/Makefile
index 57f8103..8a8f892 100644
--- a/drivers/infiniband/hw/qib/Makefile
+++ b/drivers/infiniband/hw/qib/Makefile
@@ -1,6 +1,6 @@
 obj-$(CONFIG_INFINIBAND_QIB) += ib_qib.o
 
-ib_qib-y := qib_cq.o qib_diag.o qib_dma.o qib_driver.o qib_eeprom.o \
+ib_qib-y := qib_cq.o qib_diag.o qib_driver.o qib_eeprom.o \
qib_file_ops.o qib_fs.o qib_init.o qib_intr.o qib_keys.o \
qib_mad.o qib_mmap.o qib_mr.o qib_pcie.o qib_pio_copy.o \
qib_qp.o qib_qsfp.o qib_rc.o qib_ruc.o qib_sdma.o qib_srq.o \
diff --git a/drivers/infiniband/hw/qib/qib_verbs.c 
b/drivers/infiniband/hw/qib/qib_verbs.c
index 9c721aa..fb4f045 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.c
+++ b/drivers/infiniband/hw/qib/qib_verbs.c
@@ -41,6 +41,7 @@
 #include 
 #include 
 #include 
+#include 
 
 #include "qib.h"
 #include "qib_common.h"
@@ -2269,7 +2270,7 @@ int qib_register_ib_device(struct qib_devdata *dd)
ibdev->detach_mcast = qib_multicast_detach;
ibdev->process_mad = qib_process_mad;
ibdev->mmap = qib_mmap;
-   ibdev->dma_ops = _dma_mapping_ops;
+   ibdev->dma_ops = NULL;
ibdev->get_port_immutable = qib_port_immutable;
 
snprintf(ibdev->node_desc, sizeof(ibdev->node_desc),
diff --git a/drivers/infiniband/hw/qib/qib_verbs.h 
b/drivers/infiniband/hw/qib/qib_verbs.h
index 1e43fda..8487914 100644
--- a/drivers/infiniband/hw/qib/qib_verbs.h
+++ b/drivers/infiniband/hw/qib/qib_verbs.h
@@ -1186,6 +1186,4 @@ extern unsigned int ib_qib_max_srq_wrs;
 
 extern const u32 ib_qib_rnr_table[];
 
-extern struct ib_dma_mapping_ops qib_dma_mapping_ops;
-
 #endif  /* QIB_VERBS_H */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PULL REQUEST] Please pull rdma.git

[Doug Ledford]

Hi Linus,

This should be the last pull request for the 4.4 cycle.  There are three
fixes here.  The first two were very small in terms of number of lines,
the third is more lines of change than I like this late in the cycle,
but there are positive test results from Avagotech and from my own test
setup with the target hardware, and given the problem was a 100% failure
case, I sent it through.

The following changes since commit ab5cdc31630c7596d81ca8fbe7d695f10666f39b:

  IB/mlx5: Postpone remove_keys under knowledge of coming preemption
(2015-12-08 16:55:31 -0500)

are available in the git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git
tags/for-linus

for you to fetch changes up to f41647ef06536199d3366530da050b411546979d:

  RDMA/be2net: Remove open and close entry points (2015-12-28 11:45:55
-0500)


Three late 4.4-rc fixes

- A previous patch updated the mlx4 driver to use vmalloc when there was
  not enough memory to get a contiguous region large enough for our needs,
  so we need kvfree() whenever we free that item.  We missed one place,
  so fix that now.
- A previous patch added code to match incoming packets against a specific
  device, but failed to compensate for devices that have both InfiniBand
  and Ethernet ports.  Fix that.
- Under certain vlan conditions, the ocrdma driver would fail to bring up
  any vlan interfaces and would print out a circular locking failure.  Fix
  that.


Devesh Sharma (4):
  RDMA/ocrdma: Fix vlan-id assignment in qp parameters
  RDMA/ocrdma: Dispatch only port event when port state changes
  RDMA/ocrdma: Depend on async link events from CNA
  RDMA/be2net: Remove open and close entry points

Matan Barak (1):
  IB/cma: cma_match_net_dev needs to take into account port_num

Wengang Wang (1):
  IB/mlx4: Replace kfree with kvfree in mlx4_ib_destroy_srq

 drivers/infiniband/core/cma.c   | 16 
 drivers/infiniband/hw/mlx4/srq.c|  2 +-
 drivers/infiniband/hw/ocrdma/ocrdma.h   | 10 +
 drivers/infiniband/hw/ocrdma/ocrdma_hw.c| 49 -
 drivers/infiniband/hw/ocrdma/ocrdma_hw.h|  4 +-
 drivers/infiniband/hw/ocrdma/ocrdma_main.c  | 57
+
 drivers/infiniband/hw/ocrdma/ocrdma_sli.h   | 49 +++--
 drivers/infiniband/hw/ocrdma/ocrdma_verbs.c |  2 +-
 drivers/net/ethernet/emulex/benet/be.h  |  2 -
 drivers/net/ethernet/emulex/benet/be_main.c |  4 --
 drivers/net/ethernet/emulex/benet/be_roce.c | 36 --
 drivers/net/ethernet/emulex/benet/be_roce.h |  4 +-
 12 files changed, 134 insertions(+), 101 deletions(-)

-- 
Doug Ledford 
  GPG KeyID: 0E572FDD




signature.asc
Description: OpenPGP digital signature

[PATCH v2 02/36] IB/rdmavt: Consolidate dma ops in rdmavt.

[Dennis Dalessandro]

This patch adds dma functions to rdmavt. The source is hfi1's version of
dma.c which will be removed by a subsequent hfi1 patch.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
return 0 in rvt_map_sg instead of BAD_DMA_ADDRESS
remove include of dma.h from dma.c

 drivers/infiniband/sw/rdmavt/Makefile |2 
 drivers/infiniband/sw/rdmavt/dma.c|  182 +
 drivers/infiniband/sw/rdmavt/dma.h|   53 ++
 drivers/infiniband/sw/rdmavt/vt.c |   10 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 247 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/dma.c
 create mode 100644 drivers/infiniband/sw/rdmavt/dma.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 98a664d..134d2d0 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o
+rdmavt-y := vt.o dma.o
diff --git a/drivers/infiniband/sw/rdmavt/dma.c 
b/drivers/infiniband/sw/rdmavt/dma.c
new file mode 100644
index 000..8fd7955
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/dma.c
@@ -0,0 +1,182 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+#include 
+#include 
+#include 
+
+#define BAD_DMA_ADDRESS ((u64)0)
+
+/*
+ * The following functions implement driver specific replacements
+ * for the ib_dma_*() functions.
+ *
+ * These functions return kernel virtual addresses instead of
+ * device bus addresses since the driver uses the CPU to copy
+ * data instead of using hardware DMA.
+ */
+
+static int rvt_mapping_error(struct ib_device *dev, u64 dma_addr)
+{
+   return dma_addr == BAD_DMA_ADDRESS;
+}
+
+static u64 rvt_dma_map_single(struct ib_device *dev, void *cpu_addr,
+ size_t size, enum dma_data_direction direction)
+{
+   if (WARN_ON(!valid_dma_direction(direction)))
+   return BAD_DMA_ADDRESS;
+
+   return (u64)cpu_addr;
+}
+
+static void rvt_dma_unmap_single(struct ib_device *dev, u64 addr, size_t size,
+enum dma_data_direction direction)
+{
+   /* This is a stub, nothing to be done here */
+}
+
+static u64 rvt_dma_map_page(struct ib_device *dev, struct page *page,
+   unsigned long offset, size_t size,
+   enum dma_data_direction direction)
+{
+   u64 addr;
+
+   if (WARN_ON(!valid_dma_direction(direction)))
+   return BAD_DMA_ADDRESS;
+
+   if (offset + size > PAGE_SIZE)
+   return BAD_DMA_ADDRESS;
+
+   addr = 

[PATCH v2 01/36] IB/rdmavt: Create module framework and handle driver registration

[Dennis Dalessandro]

This patch introduces the basics for a new module called rdma_vt. This new
driver is a software implementation of the InfiniBand verbs and aims to
replace the multiple implementations that exist and duplicate each others'
code.

While the call to actually register the device with the IB core happens in
rdma_vt, most of the work is still done in the drivers themselves. This
will be changing in a follow on patch this is just laying the groundwork
for this infrastructure.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
remove rdmavt specific driver version
move copyright above license text

 MAINTAINERS   |6 ++
 drivers/infiniband/Kconfig|2 +
 drivers/infiniband/Makefile   |1 
 drivers/infiniband/sw/Makefile|1 
 drivers/infiniband/sw/rdmavt/Kconfig  |6 ++
 drivers/infiniband/sw/rdmavt/Makefile |   10 
 drivers/infiniband/sw/rdmavt/vt.c |   83 +
 drivers/infiniband/sw/rdmavt/vt.h |   53 +
 include/rdma/rdma_vt.h|   70 
 9 files changed, 232 insertions(+), 0 deletions(-)
 create mode 100644 drivers/infiniband/sw/Makefile
 create mode 100644 drivers/infiniband/sw/rdmavt/Kconfig
 create mode 100644 drivers/infiniband/sw/rdmavt/Makefile
 create mode 100644 drivers/infiniband/sw/rdmavt/vt.c
 create mode 100644 drivers/infiniband/sw/rdmavt/vt.h
 create mode 100644 include/rdma/rdma_vt.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 69c8a9c..37fc192 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8880,6 +8880,12 @@ L:   rds-de...@oss.oracle.com (moderated for 
non-subscribers)
 S: Supported
 F: net/rds/
 
+RDMAVT - RDMA verbs software
+M: Dennis Dalessandro 
+L: linux-rdma@vger.kernel.org
+S: Supported
+F: drivers/infiniband/sw/rdmavt
+
 READ-COPY UPDATE (RCU)
 M: "Paul E. McKenney" 
 M: Josh Triplett 
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index aa26f3c..1b00bb8 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -72,4 +72,6 @@ source "drivers/infiniband/ulp/srpt/Kconfig"
 source "drivers/infiniband/ulp/iser/Kconfig"
 source "drivers/infiniband/ulp/isert/Kconfig"
 
+source "drivers/infiniband/sw/rdmavt/Kconfig"
+
 endif # INFINIBAND
diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile
index dc21836..fad0b44 100644
--- a/drivers/infiniband/Makefile
+++ b/drivers/infiniband/Makefile
@@ -1,3 +1,4 @@
 obj-$(CONFIG_INFINIBAND)   += core/
 obj-$(CONFIG_INFINIBAND)   += hw/
 obj-$(CONFIG_INFINIBAND)   += ulp/
+obj-$(CONFIG_INFINIBAND)   += sw/
diff --git a/drivers/infiniband/sw/Makefile b/drivers/infiniband/sw/Makefile
new file mode 100644
index 000..988b6a0
--- /dev/null
+++ b/drivers/infiniband/sw/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_INFINIBAND_RDMAVT)+= rdmavt/
diff --git a/drivers/infiniband/sw/rdmavt/Kconfig 
b/drivers/infiniband/sw/rdmavt/Kconfig
new file mode 100644
index 000..11aa6a3
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/Kconfig
@@ -0,0 +1,6 @@
+config INFINIBAND_RDMAVT
+   tristate "RDMA verbs transport library"
+   depends on 64BIT
+   default m
+   ---help---
+   This is a common software verbs provider for RDMA networks.
diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
new file mode 100644
index 000..98a664d
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -0,0 +1,10 @@
+#
+# rdmavt driver
+#
+#
+#
+# Called from the kernel module build system.
+#
+obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
+
+rdmavt-y := vt.o
diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
new file mode 100644
index 000..aa325db
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -0,0 +1,83 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - 

[PATCH v2 11/36] IB/rdmavt: Add queue pair function stubs

[Dennis Dalessandro]

Adds the stubs for create, modify, destroy and query functions for queue
pairs.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

drivers/infiniband/sw/rdmavt/Makefile |2 -
 drivers/infiniband/sw/rdmavt/qp.c |  120 +
 drivers/infiniband/sw/rdmavt/qp.h |   67 ++
 drivers/infiniband/sw/rdmavt/vt.c |6 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 195 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/qp.c
 create mode 100644 drivers/infiniband/sw/rdmavt/qp.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index c6751bb..9a9a095 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,5 +7,5 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o dma.o pd.o
+rdmavt-y := vt.o dma.o pd.o qp.o
 
diff --git a/drivers/infiniband/sw/rdmavt/qp.c 
b/drivers/infiniband/sw/rdmavt/qp.c
new file mode 100644
index 000..a59f28d
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/qp.c
@@ -0,0 +1,120 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "qp.h"
+
+/**
+ * rvt_create_qp - create a queue pair for a device
+ * @ibpd: the protection domain who's device we create the queue pair for
+ * @init_attr: the attributes of the queue pair
+ * @udata: user data for libibverbs.so
+ *
+ * Returns the queue pair on success, otherwise returns an errno.
+ *
+ * Called by the ib_create_qp() core verbs function.
+ */
+struct ib_qp *rvt_create_qp(struct ib_pd *ibpd,
+   struct ib_qp_init_attr *init_attr,
+   struct ib_udata *udata)
+{
+   /*
+* Queue pair creation is mostly an rvt issue. However, drivers have
+* their own unique idea of what queue pare numbers mean. For instance
+* there is a reserved range for PSM.
+*
+* VI-DRIVER-API: make_qpn()
+* Returns a valid QPN for verbs to use
+*/
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * qib_modify_qp - modify the attributes of a queue pair
+ * @ibqp: the queue pair who's attributes we're modifying
+ * @attr: the new attributes
+ * @attr_mask: the mask of attributes to modify
+ * @udata: user data for libibverbs.so
+ *
+ * Returns 0 on success, otherwise returns an errno.
+ */
+int rvt_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
+ int attr_mask, struct ib_udata *udata)
+{
+   /*
+* VT-DRIVER-API: qp_mtu()
+* OPA devices have a per VL MTU the driver has a mapping of IB SL to SC
+ 

[PATCH v2 19/36] IB/rdmavt: Add completion queue function stubs

[Dennis Dalessandro]

Create stubs for completion queue creation, polling,
resizing, calling for notification, and destroying.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/Makefile |2 -
 drivers/infiniband/sw/rdmavt/cq.c |  113 +
 drivers/infiniband/sw/rdmavt/cq.h |   62 ++
 drivers/infiniband/sw/rdmavt/vt.c |7 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 184 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/cq.c
 create mode 100644 drivers/infiniband/sw/rdmavt/cq.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 6f530d1..00f0188 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o
+rdmavt-y := vt.o ah.o cq.o dma.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o
diff --git a/drivers/infiniband/sw/rdmavt/cq.c 
b/drivers/infiniband/sw/rdmavt/cq.c
new file mode 100644
index 000..8d96194
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/cq.c
@@ -0,0 +1,113 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "cq.h"
+
+/**
+ * rvt_create_cq - create a completion queue
+ * @ibdev: the device this completion queue is attached to
+ * @attr: creation attributes
+ * @context: unused by the QLogic_IB driver
+ * @udata: user data for libibverbs.so
+ *
+ * Returns a pointer to the completion queue or negative errno values
+ * for failure.
+ *
+ * Called by ib_create_cq() in the generic verbs code.
+ */
+struct ib_cq *rvt_create_cq(struct ib_device *ibdev,
+   const struct ib_cq_init_attr *attr,
+   struct ib_ucontext *context,
+   struct ib_udata *udata)
+{
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * rvt_destroy_cq - destroy a completion queue
+ * @ibcq: the completion queue to destroy.
+ *
+ * Returns 0 for success.
+ *
+ * Called by ib_destroy_cq() in the generic verbs code.
+ */
+int rvt_destroy_cq(struct ib_cq *ibcq)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags notify_flags)
+{
+   return -EOPNOTSUPP;
+}
+
+/**
+ * rvt_resize_cq - change the size of the CQ
+ * @ibcq: the completion queue
+ *
+ * Returns 0 for success.
+ */
+int rvt_resize_cq(struct ib_cq *ibcq, int cqe, struct ib_udata *udata)
+{
+   return -EOPNOTSUPP;
+}
+
+/**
+ * rvt_poll_cq - poll for work completion entries
+ * 

[PATCH v2 29/36] IB/rdmavt: Add AH to rdmavt

[Dennis Dalessandro]

From: Kamal Heib 

Original patch is from Kamal Heib . It has
been split into three separate patches. This one for rdmavt,
a follow on for qib, and one for hfi1.

Create datastructure for address handle and implement the
create/destroy/modify/query of address handle for rdmavt.

Reviewed-by: Ira Weiny 
Signed-off-by: Kamal Heib 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/ah.c |   98 +++--
 drivers/infiniband/sw/rdmavt/vt.c |5 ++
 include/rdma/rdma_vt.h|   17 ++
 3 files changed, 115 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/ah.c 
b/drivers/infiniband/sw/rdmavt/ah.c
index d368955..2519db9 100644
--- a/drivers/infiniband/sw/rdmavt/ah.c
+++ b/drivers/infiniband/sw/rdmavt/ah.c
@@ -45,7 +45,49 @@
  *
  */
 
+#include 
 #include "ah.h"
+#include "vt.h" /* for prints */
+
+/**
+ * rvt_check_ah - validate the attributes of AH
+ * @ibdev: the ib device
+ * @ah_attr: the attributes of the AH
+ */
+int rvt_check_ah(struct ib_device *ibdev,
+struct ib_ah_attr *ah_attr)
+{
+   int err;
+   struct ib_port_attr port_attr;
+   struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
+   enum rdma_link_layer link = rdma_port_get_link_layer(ibdev,
+ah_attr->port_num);
+
+   err = ib_query_port(ibdev, ah_attr->port_num, _attr);
+   if (err)
+   return -EINVAL;
+   if (ah_attr->port_num < 1 ||
+   ah_attr->port_num > ibdev->phys_port_cnt)
+   return -EINVAL;
+   if (ah_attr->static_rate != IB_RATE_PORT_CURRENT &&
+   ib_rate_to_mbps(ah_attr->static_rate) < 0)
+   return -EINVAL;
+   if ((ah_attr->ah_flags & IB_AH_GRH) &&
+   ah_attr->grh.sgid_index >= port_attr.gid_tbl_len)
+   return -EINVAL;
+   if (link != IB_LINK_LAYER_ETHERNET) {
+   if (ah_attr->dlid == 0)
+   return -EINVAL;
+   if (ah_attr->dlid >= RVT_MULTICAST_LID_BASE &&
+   ah_attr->dlid != RVT_PERMISSIVE_LID &&
+   !(ah_attr->ah_flags & IB_AH_GRH))
+   return -EINVAL;
+   }
+   if (rdi->driver_f.check_ah(ibdev, ah_attr))
+   return -EINVAL;
+   return 0;
+}
+EXPORT_SYMBOL(rvt_check_ah);
 
 /**
  * rvt_create_ah - create an address handle
@@ -57,20 +99,68 @@
 struct ib_ah *rvt_create_ah(struct ib_pd *pd,
struct ib_ah_attr *ah_attr)
 {
-   return ERR_PTR(-EOPNOTSUPP);
+   struct rvt_ah *ah;
+   struct rvt_dev_info *dev = ib_to_rvt(pd->device);
+   unsigned long flags;
+
+   if (rvt_check_ah(pd->device, ah_attr))
+   return ERR_PTR(-EINVAL);
+
+   ah = kmalloc(sizeof(*ah), GFP_ATOMIC);
+   if (!ah)
+   return ERR_PTR(-ENOMEM);
+
+   spin_lock_irqsave(>n_ahs_lock, flags);
+   if (dev->n_ahs_allocated == dev->dparms.props.max_ah) {
+   spin_unlock(>n_ahs_lock);
+   kfree(ah);
+   return ERR_PTR(-ENOMEM);
+   }
+
+   dev->n_ahs_allocated++;
+   spin_unlock_irqrestore(>n_ahs_lock, flags);
+
+   ah->attr = *ah_attr;
+   atomic_set(>refcount, 0);
+
+   return >ibah;
 }
 
 int rvt_destroy_ah(struct ib_ah *ibah)
 {
-   return -EOPNOTSUPP;
+   struct rvt_dev_info *dev = ib_to_rvt(ibah->device);
+   struct rvt_ah *ah = ibah_to_rvtah(ibah);
+   unsigned long flags;
+
+   if (atomic_read(>refcount) != 0)
+   return -EBUSY;
+
+   spin_lock_irqsave(>n_ahs_lock, flags);
+   dev->n_ahs_allocated--;
+   spin_unlock_irqrestore(>n_ahs_lock, flags);
+
+   kfree(ah);
+
+   return 0;
 }
 
 int rvt_modify_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
 {
-   return -EOPNOTSUPP;
+   struct rvt_ah *ah = ibah_to_rvtah(ibah);
+
+   if (rvt_check_ah(ibah->device, ah_attr))
+   return -EINVAL;
+
+   ah->attr = *ah_attr;
+
+   return 0;
 }
 
 int rvt_query_ah(struct ib_ah *ibah, struct ib_ah_attr *ah_attr)
 {
-   return -EOPNOTSUPP;
+   struct rvt_ah *ah = ibah_to_rvtah(ibah);
+
+   *ah_attr = ah->attr;
+
+   return 0;
 }
diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index e92af9c..7dab0ca 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -221,7 +221,8 @@ int rvt_register_device(struct rvt_dev_info *rdi)
 
if ((!rdi->driver_f.port_callback) ||
(!rdi->driver_f.get_card_name) ||
-   (!rdi->driver_f.get_pci_dev)) {
+   (!rdi->driver_f.get_pci_dev) ||
+   (!rdi->driver_f.check_ah)) {
return -EINVAL;
}
 
@@ -252,6 +253,8 @@ int rvt_register_device(struct rvt_dev_info *rdi)

[PATCH v2 27/36] IB/rdmavt: Do not use rvt prints which rely on driver too early

[Dennis Dalessandro]

Trying to print debug and error messages with the rdmavt helpers will not
work out so well if the drivers have not provided the get_card and get
pci functions. Use the normal pr_error instead until we can check this.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 516c810..e92af9c 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -266,7 +266,7 @@ int rvt_register_device(struct rvt_dev_info *rdi)
/* Mem Region */
ret = rvt_driver_mr_init(rdi);
if (ret) {
-   rvt_pr_err(rdi, "Error in driver MR init.\n");
+   pr_err("Error in driver MR init.\n");
goto bail_no_mr;
}
 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 28/36] IB/rdmavt: Add common LID defines to rdmavt

[Dennis Dalessandro]

From: Kamal Heib 

Original patch is from Kamal Heib . It has
been split into separate patches.

This patch adds RVT_PERMISSIVE_LID and RVT_MULTICAST_LID_BASE
to rdmavt.

Reviewed-by: Ira Weiny 
Signed-off-by: Kamal Heib 
Signed-off-by: Dennis Dalessandro 
---
 include/rdma/rdma_vt.h |3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 9a47957..dbb45bc 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -55,6 +55,9 @@
 
 #include "ib_verbs.h"
 
+#define RVT_MULTICAST_LID_BASE 0xC000
+#define RVT_PERMISSIVE_LID 0x
+
 /*
  * For some of the IBTA objects there will likely be some
  * initializations required. We need flags to determine whether it is OK

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mad: Ensure fairness in ib_mad_completion_handler

[Eli Cohen]

On Thu, Dec 10, 2015 at 04:52:30PM -0500, ira.we...@intel.com wrote:
> From: Dean Luick 
> 
>  
> @@ -2555,6 +2567,7 @@ static void ib_mad_completion_handler(struct 
> work_struct *work)
>  {
>   struct ib_mad_port_private *port_priv;
>   struct ib_wc wc;
> + int count = 0;
>  
>   port_priv = container_of(work, struct ib_mad_port_private, work);
>   ib_req_notify_cq(port_priv->cq, IB_CQ_NEXT_COMP);

I think you shoudld push the call to ib_req_notify_cq outside the
while loop. You don't need to arm the CQ if you re-queued the work.
Only when you have drained the CQ should you re-arm.

> @@ -2574,6 +2587,11 @@ static void ib_mad_completion_handler(struct 
> work_struct *work)
>   }
>   } else
>   mad_error_handler(port_priv, );
> +
> + if (++count > MAD_COMPLETION_PROC_LIMIT) {
> + queue_work(port_priv->wq, _priv->work);
> + break;
> + }
>   }
>  }
>  
> -- 
> 1.8.2
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 03/36] IB/rdmavt: Add protection domain to rdmavt.

[Dennis Dalessandro]

Add datastructure for and allocation/deallocation of protection domains for
RDMAVT.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---

Changes since v1:
update comment about protection domain limit

 drivers/infiniband/sw/rdmavt/Makefile |3 +
 drivers/infiniband/sw/rdmavt/pd.c |  103 +
 drivers/infiniband/sw/rdmavt/pd.h |   58 +++
 drivers/infiniband/sw/rdmavt/vt.c |   12 
 drivers/infiniband/sw/rdmavt/vt.h |1 
 include/rdma/rdma_vt.h|   34 ++-
 6 files changed, 206 insertions(+), 5 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/pd.c
 create mode 100644 drivers/infiniband/sw/rdmavt/pd.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 134d2d0..c6751bb 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,5 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o dma.o
+rdmavt-y := vt.o dma.o pd.o
+
diff --git a/drivers/infiniband/sw/rdmavt/pd.c 
b/drivers/infiniband/sw/rdmavt/pd.c
new file mode 100644
index 000..e6e153f
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/pd.c
@@ -0,0 +1,103 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include 
+#include "pd.h"
+
+struct ib_pd *rvt_alloc_pd(struct ib_device *ibdev,
+  struct ib_ucontext *context,
+  struct ib_udata *udata)
+{
+   struct rvt_dev_info *dev = ib_to_rvt(ibdev);
+   struct rvt_pd *pd;
+   struct ib_pd *ret;
+
+   pd = kmalloc(sizeof(*pd), GFP_KERNEL);
+   if (!pd) {
+   ret = ERR_PTR(-ENOMEM);
+   goto bail;
+   }
+   /*
+* While we could continue allocating protecetion domains, being
+* constrained only by system resources. The IBTA spec defines that
+* there is a max_pd limit that can be set and we need to check for
+* that.
+*/
+
+   spin_lock(>n_pds_lock);
+   if (dev->n_pds_allocated == dev->dparms.max_pds) {
+   spin_unlock(>n_pds_lock);
+   kfree(pd);
+   ret = ERR_PTR(-ENOMEM);
+   goto bail;
+   }
+
+   dev->n_pds_allocated++;
+   spin_unlock(>n_pds_lock);
+
+   /* ib_alloc_pd() will initialize pd->ibpd. */
+   pd->user = udata ? 1 : 0;
+
+   ret = >ibpd;
+
+bail:
+   return ret;
+}
+
+int rvt_dealloc_pd(struct ib_pd *ibpd)
+{
+   struct rvt_pd *pd = ibpd_to_rvtpd(ibpd);
+   struct rvt_dev_info *dev = ib_to_rvt(ibpd->device);
+
+   spin_lock(>n_pds_lock);
+   dev->n_pds_allocated--;
+   

[PATCH v2 04/36] IB/rdmavt: Add ib core device attributes to rvt driver params list

[Dennis Dalessandro]

Instead of trying to handle each parameter separately, add ib_device_attr
to rvt_driver_params. This means drivers will fill this in and pass to the
rvt registration function.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
remove comment on alternative design for private data

 drivers/infiniband/sw/rdmavt/pd.c |2 +
 include/rdma/rdma_vt.h|   57 -
 2 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/pd.c 
b/drivers/infiniband/sw/rdmavt/pd.c
index e6e153f..f8dba88 100644
--- a/drivers/infiniband/sw/rdmavt/pd.c
+++ b/drivers/infiniband/sw/rdmavt/pd.c
@@ -69,7 +69,7 @@ struct ib_pd *rvt_alloc_pd(struct ib_device *ibdev,
 */
 
spin_lock(>n_pds_lock);
-   if (dev->n_pds_allocated == dev->dparms.max_pds) {
+   if (dev->n_pds_allocated == dev->dparms.props.max_pd) {
spin_unlock(>n_pds_lock);
kfree(pd);
ret = ERR_PTR(-ENOMEM);
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 6bf5fd4..2990e03 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -59,7 +59,45 @@
  * Things that are driver specific, module parameters in hfi1 and qib
  */
 struct rvt_driver_params {
-   int max_pds;
+   /*
+* driver required fields:
+*  node_guid
+*  phys_port_cnt
+*  dma_device
+*  owner
+* driver optional fields (rvt will provide generic value if blank):
+*  name
+*  node_desc
+* rvt fields, driver value ignored:
+*  uverbs_abi_ver
+*  node_type
+*  num_comp_vectors
+*  uverbs_cmd_mask
+*/
+   struct ib_device_attr props;
+
+   /*
+* Drivers will need to support a number of notifications to rvt in
+* accordance with certain events. This structure should contain a mask
+* of the supported events. Such events that the rvt may need to know
+* about include:
+* port errors
+* port active
+* lid change
+* sm change
+* client reregister
+* pkey change
+*
+* There may also be other events that the rvt layers needs to know
+* about this is not an exhaustive list. Some events though rvt does not
+* need to rely on the driver for such as completion queue error.
+*/
+int rvt_signal_supported;
+
+   /*
+* Anything driver specific that is not covered by props
+* For instance special module parameters. Goes here.
+*/
 };
 
 /* Protection domain */
@@ -69,10 +107,25 @@ struct rvt_pd {
 };
 
 struct rvt_dev_info {
+   /*
+* Prior to calling for registration the driver will be responsible for
+* allocating space for this structure.
+*
+* The driver will also be responsible for filling in certain members of
+* dparms.props
+*/
+
struct ib_device ibdev;
 
-   /* Driver specific */
+   /* Driver specific properties */
struct rvt_driver_params dparms;
+
+   /*
+* The work to create port files in /sys/class Infiniband is different
+* depending on the driver. This should not be extracted away and
+* instead drivers are responsible for setting the correct callback for
+* this.
+*/
int (*port_callback)(struct ib_device *, u8, struct kobject *);
 
/* Internal use */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 07/36] IB/rdmavt: Add query and modify port stubs

[Dennis Dalessandro]

This patch adds the query and modify port stubs. The query will mostly
entail the driver returning everything in the ib_port_attr which will get
handed back to the verbs layer. The modify will need some API helpers in
the driver. The send_trap and post_mad_send are still issues to address.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Fix comment style for rvt_query_port
Change all stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/vt.c |   48 +
 1 files changed, 48 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index db14646..5ac241c 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -91,6 +91,52 @@ static int rvt_modify_device(struct ib_device *device,
return -EOPNOTSUPP;
 }
 
+/**
+ * rvt_query_port: Passes the query port call to the driver
+ * @ibdev: Verbs IB dev
+ * @port: port number
+ * @props: structure to hold returned properties
+ *
+ * Returns 0 on success
+ */
+static int rvt_query_port(struct ib_device *ibdev, u8 port,
+ struct ib_port_attr *props)
+{
+   /*
+* VT-DRIVER-API: query_port_state()
+* driver returns pretty much everything in ib_port_attr
+*/
+   return -EOPNOTSUPP;
+}
+
+/**
+ * rvt_modify_port
+ * @ibdev: Verbs IB dev
+ * @port: Port number
+ * @port_modify_mask: How to change the port
+ * @props: Structure to fill in
+ *
+ * Returns 0 on success
+ */
+static int rvt_modify_port(struct ib_device *ibdev, u8 port,
+  int port_modify_mask, struct ib_port_modify *props)
+{
+   /*
+* VT-DRIVER-API: set_link_state()
+* driver will set the link state using the IB enumeration
+*
+* VT-DRIVER-API: clear_qkey_violations()
+* clears driver private qkey counter
+*
+* VT-DRIVER-API: get_lid()
+* driver needs to return the LID
+*
+* TBD: send_trap() and post_mad_send() need examined to see where they
+* fit in.
+*/
+   return -EOPNOTSUPP;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -106,6 +152,8 @@ int rvt_register_device(struct rvt_dev_info *rdi)
/* Dev Ops */
CHECK_DRIVER_OVERRIDE(rdi, query_device);
CHECK_DRIVER_OVERRIDE(rdi, modify_device);
+   CHECK_DRIVER_OVERRIDE(rdi, query_port);
+   CHECK_DRIVER_OVERRIDE(rdi, modify_port);
 
/* DMA Operations */
rdi->ibdev.dma_ops =

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 06/36] IB/rdmavt: Add query and modify device stubs

[Dennis Dalessandro]

Adds the stubs which will handle the query and modify device functions. At
this time the only intention is to support changing the node desc and the
guid via these calls.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/vt.c |   31 +++
 1 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 8bd25c3..db14646 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -64,6 +64,33 @@ static void rvt_cleanup(void)
 }
 module_exit(rvt_cleanup);
 
+static int rvt_query_device(struct ib_device *ibdev,
+   struct ib_device_attr *props,
+   struct ib_udata *uhw)
+{
+   /*
+* Return rvt_dev_info.props contents
+*/
+   return -EOPNOTSUPP;
+}
+
+static int rvt_modify_device(struct ib_device *device,
+int device_modify_mask,
+struct ib_device_modify *device_modify)
+{
+   /*
+* Change dev props. Planned support is for node desc change and sys
+* guid change only. This matches hfi1 and qib behavior. Other drivers
+* that support existing modifications will need to add their support.
+*/
+
+   /*
+* VT-DRIVER-API: node_desc_change()
+* VT-DRIVER-API: sys_guid_change()
+*/
+   return -EOPNOTSUPP;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -76,6 +103,10 @@ int rvt_register_device(struct rvt_dev_info *rdi)
if (!rdi)
return -EINVAL;
 
+   /* Dev Ops */
+   CHECK_DRIVER_OVERRIDE(rdi, query_device);
+   CHECK_DRIVER_OVERRIDE(rdi, modify_device);
+
/* DMA Operations */
rdi->ibdev.dma_ops =
rdi->ibdev.dma_ops ? : _default_dma_mapping_ops;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 15/36] IB/rdmavt: Add multicast stubs

[Dennis Dalessandro]

Adds the function stubs for attach and detach multicast.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/Makefile |2 +
 drivers/infiniband/sw/rdmavt/mcast.c  |   58 +
 drivers/infiniband/sw/rdmavt/mcast.h  |   56 
 drivers/infiniband/sw/rdmavt/vt.c |4 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 +
 5 files changed, 120 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/mcast.c
 create mode 100644 drivers/infiniband/sw/rdmavt/mcast.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 204be84..d2af114 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o mr.o pd.o qp.o srq.o
+rdmavt-y := vt.o ah.o dma.o mcast.o mr.o pd.o qp.o srq.o
diff --git a/drivers/infiniband/sw/rdmavt/mcast.c 
b/drivers/infiniband/sw/rdmavt/mcast.c
new file mode 100644
index 000..5a78dc7
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mcast.c
@@ -0,0 +1,58 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "mcast.h"
+
+int rvt_attach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_detach_mcast(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
+{
+   return -EOPNOTSUPP;
+}
diff --git a/drivers/infiniband/sw/rdmavt/mcast.h 
b/drivers/infiniband/sw/rdmavt/mcast.h
new file mode 100644
index 000..21647c3
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mcast.h
@@ -0,0 +1,56 @@
+#ifndef DEF_RVTMCAST_H
+#define DEF_RVTMCAST_H
+
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ 

[PATCH v2 16/36] IB/rdmavt: Add process MAD stub

[Dennis Dalessandro]

This adds the stub for process mad. More study is needed to determine the
final MAD interaction between the driver and rvt.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/Makefile |2 -
 drivers/infiniband/sw/rdmavt/mad.c|   85 +
 drivers/infiniband/sw/rdmavt/mad.h|   59 +++
 3 files changed, 145 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/mad.c
 create mode 100644 drivers/infiniband/sw/rdmavt/mad.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index d2af114..fe65410 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o mcast.o mr.o pd.o qp.o srq.o
+rdmavt-y := vt.o ah.o dma.o mad.o mcast.o mr.o pd.o qp.o srq.o
diff --git a/drivers/infiniband/sw/rdmavt/mad.c 
b/drivers/infiniband/sw/rdmavt/mad.c
new file mode 100644
index 000..eef7029
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mad.c
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "mad.h"
+
+/**
+ * rvt_process_mad - process an incoming MAD packet
+ * @ibdev: the infiniband device this packet came in on
+ * @mad_flags: MAD flags
+ * @port: the port number this packet came in on
+ * @in_wc: the work completion entry for this packet
+ * @in_grh: the global route header for this packet
+ * @in_mad: the incoming MAD
+ * @out_mad: any outgoing MAD reply
+ *
+ * Returns IB_MAD_RESULT_SUCCESS if this is a MAD that we are not
+ * interested in processing.
+ *
+ * Note that the verbs framework has already done the MAD sanity checks,
+ * and hop count/pointer updating for IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE
+ * MADs.
+ *
+ * This is called by the ib_mad module.
+ */
+int rvt_process_mad(struct ib_device *ibdev, int mad_flags, u8 port,
+   const struct ib_wc *in_wc, const struct ib_grh *in_grh,
+   const struct ib_mad_hdr *in, size_t in_mad_size,
+   struct ib_mad_hdr *out, size_t *out_mad_size,
+   u16 *out_mad_pkey_index)
+{
+   /*
+* Drivers will need to provide a number of things. For exmaple counters
+* will need to be maintained by the driver but shoud live in the rvt
+* structure. More study will be needed to finalize the interface
+* between drivers and rvt for mad packets.
+*
+*VT-DRIVER-API: 
+*
+*/
+   return IB_MAD_RESULT_FAILURE;
+}
diff --git a/drivers/infiniband/sw/rdmavt/mad.h 

[PATCH v2 17/36] IB/rdmavt: Add mmap stub

[Dennis Dalessandro]

Adds the stub for the mmap verbs call.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/Makefile |2 +
 drivers/infiniband/sw/rdmavt/mmap.c   |   60 +
 drivers/infiniband/sw/rdmavt/mmap.h   |   55 ++
 drivers/infiniband/sw/rdmavt/vt.c |1 +
 drivers/infiniband/sw/rdmavt/vt.h |1 +
 5 files changed, 118 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/mmap.c
 create mode 100644 drivers/infiniband/sw/rdmavt/mmap.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index fe65410..6f530d1 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o mad.o mcast.o mr.o pd.o qp.o srq.o
+rdmavt-y := vt.o ah.o dma.o mad.o mcast.o mmap.o mr.o pd.o qp.o srq.o
diff --git a/drivers/infiniband/sw/rdmavt/mmap.c 
b/drivers/infiniband/sw/rdmavt/mmap.c
new file mode 100644
index 000..d09f3a0
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mmap.c
@@ -0,0 +1,60 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include 
+#include "mmap.h"
+
+/**
+ * rvt_mmap - create a new mmap region
+ * @context: the IB user context of the process making the mmap() call
+ * @vma: the VMA to be initialized
+ * Return zero if the mmap is OK. Otherwise, return an errno.
+ */
+int rvt_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
+{
+   return -EOPNOTSUPP;
+}
diff --git a/drivers/infiniband/sw/rdmavt/mmap.h 
b/drivers/infiniband/sw/rdmavt/mmap.h
new file mode 100644
index 000..94f6377
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/mmap.h
@@ -0,0 +1,55 @@
+#ifndef DEF_RDMAVTMMAP_H
+#define DEF_RDMAVTMMAP_H
+
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted 

[PATCH v2 14/36] IB/rdmavt: Add SRQ stubs

[Dennis Dalessandro]

Adds the stubs for create, modify, query, and destory for shared
request queues.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/Makefile |2 -
 drivers/infiniband/sw/rdmavt/srq.c|   86 +
 drivers/infiniband/sw/rdmavt/srq.h|   61 +++
 drivers/infiniband/sw/rdmavt/vt.c |6 ++
 drivers/infiniband/sw/rdmavt/vt.h |1 
 5 files changed, 155 insertions(+), 1 deletions(-)
 create mode 100644 drivers/infiniband/sw/rdmavt/srq.c
 create mode 100644 drivers/infiniband/sw/rdmavt/srq.h

diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
b/drivers/infiniband/sw/rdmavt/Makefile
index 084ee6a..204be84 100644
--- a/drivers/infiniband/sw/rdmavt/Makefile
+++ b/drivers/infiniband/sw/rdmavt/Makefile
@@ -7,4 +7,4 @@
 #
 obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
 
-rdmavt-y := vt.o ah.o dma.o mr.o pd.o qp.o
+rdmavt-y := vt.o ah.o dma.o mr.o pd.o qp.o srq.o
diff --git a/drivers/infiniband/sw/rdmavt/srq.c 
b/drivers/infiniband/sw/rdmavt/srq.c
new file mode 100644
index 000..bbb623a
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/srq.c
@@ -0,0 +1,86 @@
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * GPL LICENSE SUMMARY
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of version 2 of the GNU General Public License as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * BSD LICENSE
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ *
+ *  - Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ *  - Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in
+ *the documentation and/or other materials provided with the
+ *distribution.
+ *  - Neither the name of Intel Corporation nor the names of its
+ *contributors may be used to endorse or promote products derived
+ *from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ */
+
+#include "srq.h"
+
+/**
+ * rvt_create_srq - create a shared receive queue
+ * @ibpd: the protection domain of the SRQ to create
+ * @srq_init_attr: the attributes of the SRQ
+ * @udata: data from libibverbs when creating a user SRQ
+ */
+struct ib_srq *rvt_create_srq(struct ib_pd *ibpd,
+ struct ib_srq_init_attr *srq_init_attr,
+ struct ib_udata *udata)
+{
+   return ERR_PTR(-EOPNOTSUPP);
+}
+
+/**
+ * rvt_modify_srq - modify a shared receive queue
+ * @ibsrq: the SRQ to modify
+ * @attr: the new attributes of the SRQ
+ * @attr_mask: indicates which attributes to modify
+ * @udata: user data for libibverbs.so
+ */
+int rvt_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
+  enum ib_srq_attr_mask attr_mask,
+  struct ib_udata *udata)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
+{
+   return -EOPNOTSUPP;
+}
+
+int rvt_destroy_srq(struct ib_srq *ibsrq)
+{
+   return -EOPNOTSUPP;
+}
+
diff --git a/drivers/infiniband/sw/rdmavt/srq.h 
b/drivers/infiniband/sw/rdmavt/srq.h
new file mode 100644
index 000..0c3c5a7
--- /dev/null
+++ b/drivers/infiniband/sw/rdmavt/srq.h
@@ -0,0 +1,61 @@
+#ifndef DEF_RVTSRQ_H
+#define DEF_RVTSRQ_H
+
+/*
+ * Copyright(c) 2015 Intel Corporation.
+ *
+ * This file is provided under a dual BSD/GPLv2 license.  

[PATCH v2 18/36] IB/rdmavt: Add get port immutable stub

[Dennis Dalessandro]

This adds the get port immutable verbs call.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Change stubs to return EOPNOTSUPP

 drivers/infiniband/sw/rdmavt/vt.c |7 +++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index a889ec4..99fbe37 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -198,6 +198,12 @@ static int rvt_dealloc_ucontext(struct ib_ucontext 
*context)
return -EOPNOTSUPP;
 }
 
+static int rvt_get_port_immutable(struct ib_device *ibdev, u8 port_num,
+ struct ib_port_immutable *immutable)
+{
+   return -EOPNOTSUPP;
+}
+
 /*
  * Check driver override. If driver passes a value use it, otherwise we use our
  * own value.
@@ -219,6 +225,7 @@ int rvt_register_device(struct rvt_dev_info *rdi)
CHECK_DRIVER_OVERRIDE(rdi, query_gid);
CHECK_DRIVER_OVERRIDE(rdi, alloc_ucontext);
CHECK_DRIVER_OVERRIDE(rdi, dealloc_ucontext);
+   CHECK_DRIVER_OVERRIDE(rdi, get_port_immutable);
 
/* Queue Pairs */
CHECK_DRIVER_OVERRIDE(rdi, create_qp);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 25/36] IB/rdmavt: Add the start of capability flags

[Dennis Dalessandro]

Drivers will need a set of flags to dictate behavior to rdmavt. This patch
adds a placeholder and a spot for it to live, as well as a few flags
that will be used.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 include/rdma/rdma_vt.h |   12 
 1 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 4b83770..b44ac17 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -56,6 +56,16 @@
 #include "ib_verbs.h"
 
 /*
+ * For some of the IBTA objects there will likely be some
+ * initializations required. We need flags to determine whether it is OK
+ * for rdmavt to do this or not. This does not imply any functions of a
+ * partiuclar IBTA object are overridden.
+ */
+#define RVT_FLAG_MR_INIT_DRIVER BIT(1)
+#define RVT_FLAG_QP_INIT_DRIVER BIT(2)
+#define RVT_FLAG_CQ_INIT_DRIVER BIT(3)
+
+/*
  * For Memory Regions. This stuff should probably be moved into rdmavt/mr.h 
once
  * drivers no longer need access to the MR directly.
  */
@@ -429,6 +439,8 @@ struct rvt_dev_info {
/* Internal use */
int n_pds_allocated;
spinlock_t n_pds_lock; /* Protect pd allocated count */
+
+   int flags;
 };
 
 static inline struct rvt_pd *ibpd_to_rvtpd(struct ib_pd *ibpd)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 26/36] IB/rdmavt: Move memory registration into rdmavt

[Dennis Dalessandro]

Use the memory registration routines in hfi1 and move them to rdmavt.
A follow on patch will address removing the duplicated code in the
hfi1 and qib drivers.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Remove rvt_reg_phys_mr

 drivers/infiniband/sw/rdmavt/mr.c |  694 -
 drivers/infiniband/sw/rdmavt/mr.h |   23 +
 drivers/infiniband/sw/rdmavt/vt.c |   25 +
 include/rdma/rdma_vt.h|   19 +
 4 files changed, 731 insertions(+), 30 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/mr.c 
b/drivers/infiniband/sw/rdmavt/mr.c
index c8e5cfe..f1dcaf4 100644
--- a/drivers/infiniband/sw/rdmavt/mr.c
+++ b/drivers/infiniband/sw/rdmavt/mr.c
@@ -46,41 +46,298 @@
  */
 
 #include 
+#include 
+#include 
+#include 
+#include "vt.h"
 #include "mr.h"
 
+/*
+ * Do any intilization needed when a driver registers with rdmavt.
+ */
+int rvt_driver_mr_init(struct rvt_dev_info *rdi)
+{
+   unsigned int lkey_table_size = rdi->dparms.lkey_table_size;
+   unsigned lk_tab_size;
+   int i;
+
+   if (rdi->flags & RVT_FLAG_MR_INIT_DRIVER) {
+   rvt_pr_info(rdi, "Driver is doing MR init.\n");
+   return 0;
+   }
+
+   /*
+* The top hfi1_lkey_table_size bits are used to index the
+* table.  The lower 8 bits can be owned by the user (copied from
+* the LKEY).  The remaining bits act as a generation number or tag.
+*/
+   if (!lkey_table_size)
+   return -EINVAL;
+
+   spin_lock_init(>lkey_table.lock);
+
+   rdi->lkey_table.max = 1 << lkey_table_size;
+
+   /* ensure generation is at least 4 bits */
+   if (lkey_table_size > RVT_MAX_LKEY_TABLE_BITS) {
+   rvt_pr_warn(rdi, "lkey bits %u too large, reduced to %u\n",
+   lkey_table_size, RVT_MAX_LKEY_TABLE_BITS);
+   rdi->dparms.lkey_table_size = RVT_MAX_LKEY_TABLE_BITS;
+   lkey_table_size = rdi->dparms.lkey_table_size;
+   }
+   lk_tab_size = rdi->lkey_table.max * sizeof(*rdi->lkey_table.table);
+   rdi->lkey_table.table = (struct rvt_mregion __rcu **)
+  vmalloc(lk_tab_size);
+   if (!rdi->lkey_table.table)
+   return -ENOMEM;
+
+   RCU_INIT_POINTER(rdi->dma_mr, NULL);
+   for (i = 0; i < rdi->lkey_table.max; i++)
+   RCU_INIT_POINTER(rdi->lkey_table.table[i], NULL);
+
+   return 0;
+}
+
+/*
+ * called when drivers have unregistered or perhaps failed to register with us
+ */
+void rvt_mr_exit(struct rvt_dev_info *rdi)
+{
+   if (rdi->dma_mr)
+   rvt_pr_err(rdi, "DMA MR not null!\n");
+
+   vfree(rdi->lkey_table.table);
+}
+
+static void rvt_deinit_mregion(struct rvt_mregion *mr)
+{
+   int i = mr->mapsz;
+
+   mr->mapsz = 0;
+   while (i)
+   kfree(mr->map[--i]);
+}
+
+static int rvt_init_mregion(struct rvt_mregion *mr, struct ib_pd *pd,
+   int count)
+{
+   int m, i = 0;
+
+   mr->mapsz = 0;
+   m = (count + RVT_SEGSZ - 1) / RVT_SEGSZ;
+   for (; i < m; i++) {
+   mr->map[i] = kzalloc(sizeof(*mr->map[0]), GFP_KERNEL);
+   if (!mr->map[i]) {
+   rvt_deinit_mregion(mr);
+   return -ENOMEM;
+   }
+   mr->mapsz++;
+   }
+   init_completion(>comp);
+   /* count returning the ptr to user */
+   atomic_set(>refcount, 1);
+   mr->pd = pd;
+   mr->max_segs = count;
+   return 0;
+}
+
 /**
- * rvt_get_dma_mr - get a DMA memory region
- * @pd: protection domain for this memory region
- * @acc: access flags
+ * rvt_alloc_lkey - allocate an lkey
+ * @mr: memory region that this lkey protects
+ * @dma_region: 0->normal key, 1->restricted DMA key
+ *
+ * Returns 0 if successful, otherwise returns -errno.
+ *
+ * Increments mr reference count as required.
+ *
+ * Sets the lkey field mr for non-dma regions.
  *
- * Returns the memory region on success, otherwise returns an errno.
- * Note that all DMA addresses should be created via the
- * struct ib_dma_mapping_ops functions (see dma.c).
  */
-struct ib_mr *rvt_get_dma_mr(struct ib_pd *pd, int acc)
+static int rvt_alloc_lkey(struct rvt_mregion *mr, int dma_region)
 {
+   unsigned long flags;
+   u32 r;
+   u32 n;
+   int ret = 0;
+   struct rvt_dev_info *dev = ib_to_rvt(mr->pd->device);
+   struct rvt_lkey_table *rkt = >lkey_table;
+
+   rvt_get_mr(mr);
+   spin_lock_irqsave(>lock, flags);
+
+   /* special case for dma_mr lkey == 0 */
+   if (dma_region) {
+   struct rvt_mregion *tmr;
+
+   tmr = rcu_access_pointer(dev->dma_mr);
+   if (!tmr) {
+   rcu_assign_pointer(dev->dma_mr, mr);
+   

[PATCH v2 22/36] IB/rdmavt: Add queue pair data structure to rdmavt

[Dennis Dalessandro]

Add queue pair data structure as well as supporting structures to rdmavt.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
Changes since v1:
Rename rvt_sge.m => rvt_sge.cur_map
Rename rvt_sge.n => rvt_sge.cur

 drivers/infiniband/sw/rdmavt/qp.h |5 -
 include/rdma/rdma_vt.h|  233 +
 2 files changed, 233 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/qp.h 
b/drivers/infiniband/sw/rdmavt/qp.h
index 10bc636..9c2999d 100644
--- a/drivers/infiniband/sw/rdmavt/qp.h
+++ b/drivers/infiniband/sw/rdmavt/qp.h
@@ -50,11 +50,6 @@
 
 #include 
 
-struct rvt_qp {
-   struct ib_qp *ibqp;
-   /* Other stuff */
-};
-
 struct ib_qp *rvt_create_qp(struct ib_pd *ibpd,
struct ib_qp_init_attr *init_attr,
struct ib_udata *udata);
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index f232e39..9baa7f0 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -106,6 +106,239 @@ struct rvt_lkey_table {
 /* End Memmory Region */
 
 /*
+ * Things needed for the Queue Pair definition. Like the MR stuff above the
+ * following should probably get moved to qp.h once drivers stop trying to make
+ * and manipulate thier own QPs. For the few instnaces where a driver may need
+ * to look into a queue pair there should be a pointer to a driver priavte data
+ * structure that they can look at.
+ */
+
+/*
+ * These keep track of the copy progress within a memory region.
+ * Used by the verbs layer.
+ */
+struct rvt_sge {
+   struct rvt_mregion *mr;
+   void *vaddr;/* kernel virtual address of segment */
+   u32 sge_length; /* length of the SGE */
+   u32 length; /* remaining length of the segment */
+   u16 m;  /* current index: mr->map[m] */
+   u16 n;  /* current index: mr->map[m]->segs[n] */
+};
+
+/*
+ * Send work request queue entry.
+ * The size of the sg_list is determined when the QP is created and stored
+ * in qp->s_max_sge.
+ */
+struct rvt_swqe {
+   union {
+   struct ib_send_wr wr;   /* don't use wr.sg_list */
+   struct ib_ud_wr ud_wr;
+   struct ib_reg_wr reg_wr;
+   struct ib_rdma_wr rdma_wr;
+   struct ib_atomic_wr atomic_wr;
+   };
+   u32 psn;/* first packet sequence number */
+   u32 lpsn;   /* last packet sequence number */
+   u32 ssn;/* send sequence number */
+   u32 length; /* total length of data in sg_list */
+   struct rvt_sge sg_list[0];
+};
+
+/*
+ * Receive work request queue entry.
+ * The size of the sg_list is determined when the QP (or SRQ) is created
+ * and stored in qp->r_rq.max_sge (or srq->rq.max_sge).
+ */
+struct rvt_rwqe {
+   u64 wr_id;
+   u8 num_sge;
+   struct ib_sge sg_list[0];
+};
+
+/*
+ * This structure is used to contain the head pointer, tail pointer,
+ * and receive work queue entries as a single memory allocation so
+ * it can be mmap'ed into user space.
+ * Note that the wq array elements are variable size so you can't
+ * just index into the array to get the N'th element;
+ * use get_rwqe_ptr() instead.
+ */
+struct rvt_rwq {
+   u32 head;   /* new work requests posted to the head */
+   u32 tail;   /* receives pull requests from here. */
+   struct rvt_rwqe wq[0];
+};
+
+struct rvt_rq {
+   struct rvt_rwq *wq;
+   u32 size;   /* size of RWQE array */
+   u8 max_sge;
+   /* protect changes in this struct */
+   spinlock_t lock cacheline_aligned_in_smp;
+};
+
+/*
+ * This structure is used by rvt_mmap() to validate an offset
+ * when an mmap() request is made.  The vm_area_struct then uses
+ * this as its vm_private_data.
+ */
+struct rvt_mmap_info {
+   struct list_head pending_mmaps;
+   struct ib_ucontext *context;
+   void *obj;
+   __u64 offset;
+   struct kref ref;
+   unsigned size;
+};
+
+#define RVT_MAX_RDMA_ATOMIC16
+
+/*
+ * This structure holds the information that the send tasklet needs
+ * to send a RDMA read response or atomic operation.
+ */
+struct rvt_ack_entry {
+   u8 opcode;
+   u8 sent;
+   u32 psn;
+   u32 lpsn;
+   union {
+   struct rvt_sge rdma_sge;
+   u64 atomic_data;
+   };
+};
+
+struct rvt_sge_state {
+   struct rvt_sge *sg_list;  /* next SGE to be used if any */
+   struct rvt_sge sge;   /* progress state for the current SGE */
+   u32 total_len;
+   u8 num_sge;
+};
+
+/*
+ * Variables prefixed with s_ are for the requester (sender).
+ * Variables prefixed with r_ are for the responder (receiver).
+ * Variables prefixed with ack_ are for responder replies.
+ *
+ * 

[PATCH v2 24/36] IB/rdmavt: Add device specific info prints

[Dennis Dalessandro]

Follow hfi1's example for printing information about the driver and
incorporate into rdmavt. This requires two new functions to be
provided by the driver, one to get_card_name and one to get_pci_dev.

Reviewed-by: Mike Marciniszyn 
Reviewed-by: Ira Weiny 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |   13 ++---
 drivers/infiniband/sw/rdmavt/vt.h |   28 
 include/rdma/rdma_vt.h|3 +++
 3 files changed, 41 insertions(+), 3 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index e4881ca..1279c03 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -213,9 +213,18 @@ static int rvt_get_port_immutable(struct ib_device *ibdev, 
u8 port_num,
 
 int rvt_register_device(struct rvt_dev_info *rdi)
 {
+   /* Validate that drivers have provided the right information */
if (!rdi)
return -EINVAL;
 
+   if ((!rdi->driver_f.port_callback) ||
+   (!rdi->driver_f.get_card_name) ||
+   (!rdi->driver_f.get_pci_dev)) {
+   return -EINVAL;
+   }
+
+   /* Once we get past here we can use the rvt_pr macros */
+
/* Dev Ops */
CHECK_DRIVER_OVERRIDE(rdi, query_device);
CHECK_DRIVER_OVERRIDE(rdi, modify_device);
@@ -281,9 +290,7 @@ int rvt_register_device(struct rvt_dev_info *rdi)
spin_lock_init(>n_pds_lock);
rdi->n_pds_allocated = 0;
 
-   /* Validate that drivers have provided the right functions */
-   if (!rdi->driver_f.port_callback)
-   return -EINVAL;
+   rvt_pr_info(rdi, "Registration with rdmavt done.\n");
 
/* We are now good to announce we exist */
return ib_register_device(>ibdev, rdi->driver_f.port_callback);
diff --git a/drivers/infiniband/sw/rdmavt/vt.h 
b/drivers/infiniband/sw/rdmavt/vt.h
index fdb52a8..54ee05a 100644
--- a/drivers/infiniband/sw/rdmavt/vt.h
+++ b/drivers/infiniband/sw/rdmavt/vt.h
@@ -49,6 +49,7 @@
  */
 
 #include 
+#include 
 #include "dma.h"
 #include "pd.h"
 #include "qp.h"
@@ -59,4 +60,31 @@
 #include "mmap.h"
 #include "cq.h"
 
+#define rvt_pr_info(rdi, fmt, ...) \
+   __rvt_pr_info(rdi->driver_f.get_pci_dev(rdi), \
+ rdi->driver_f.get_card_name(rdi), \
+ fmt, \
+ ##__VA_ARGS__)
+
+#define rvt_pr_warn(rdi, fmt, ...) \
+   __rvt_pr_warn(rdi->driver_f.get_pci_dev(rdi), \
+ rdi->driver_f.get_card_name(rdi), \
+ fmt, \
+ ##__VA_ARGS__)
+
+#define rvt_pr_err(rdi, fmt, ...) \
+   __rvt_pr_err(rdi->driver_f.get_pci_dev(rdi), \
+rdi->driver_f.get_card_name(rdi), \
+fmt, \
+##__VA_ARGS__)
+
+#define __rvt_pr_info(pdev, name, fmt, ...) \
+   dev_info(>dev, "%s: " fmt, name, ##__VA_ARGS__)
+
+#define __rvt_pr_warn(pdev, name, fmt, ...) \
+   dev_warn(>dev, "%s: " fmt, name, ##__VA_ARGS__)
+
+#define __rvt_pr_err(pdev, name, fmt, ...) \
+   dev_err(>dev, "%s: " fmt, name, ##__VA_ARGS__)
+
 #endif  /* DEF_RDMAVT_H */
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index e0beedc..4b83770 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -386,6 +386,7 @@ struct rvt_driver_params {
 /*
  * Functions that drivers are required to support
  */
+struct rvt_dev_info;
 struct rvt_driver_provided {
/*
 * The work to create port files in /sys/class Infiniband is different
@@ -394,6 +395,8 @@ struct rvt_driver_provided {
 * this.
 */
int (*port_callback)(struct ib_device *, u8, struct kobject *);
+   const char * (*get_card_name)(struct rvt_dev_info *rdi);
+   struct pci_dev * (*get_pci_dev)(struct rvt_dev_info *rdi);
 };
 
 /* Protection domain */

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 35/36] IB/rdmavt: Add mmap related functions

[Dennis Dalessandro]

The mmap data structure was moved in a previous commit. This patch now
pulls in the related functions.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/mmap.c |  140 +++
 drivers/infiniband/sw/rdmavt/mmap.h |2 -
 drivers/infiniband/sw/rdmavt/vt.c   |1 
 include/rdma/rdma_vt.h  |   15 
 4 files changed, 156 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/mmap.c 
b/drivers/infiniband/sw/rdmavt/mmap.c
index d09f3a0..fc30ff7 100644
--- a/drivers/infiniband/sw/rdmavt/mmap.c
+++ b/drivers/infiniband/sw/rdmavt/mmap.c
@@ -46,8 +46,61 @@
  */
 
 #include 
+#include 
+#include 
+#include 
 #include "mmap.h"
 
+void rvt_mmap_init(struct rvt_dev_info *rdi)
+{
+   INIT_LIST_HEAD(>pending_mmaps);
+   spin_lock_init(>pending_lock);
+   rdi->mmap_offset = PAGE_SIZE;
+   spin_lock_init(>mmap_offset_lock);
+}
+
+/**
+ * rvt_release_mmap_info - free mmap info structure
+ * @ref: a pointer to the kref within struct rvt_mmap_info
+ */
+void rvt_release_mmap_info(struct kref *ref)
+{
+   struct rvt_mmap_info *ip =
+   container_of(ref, struct rvt_mmap_info, ref);
+   struct rvt_dev_info *rdi = ib_to_rvt(ip->context->device);
+
+   spin_lock_irq(>pending_lock);
+   list_del(>pending_mmaps);
+   spin_unlock_irq(>pending_lock);
+
+   vfree(ip->obj);
+   kfree(ip);
+}
+EXPORT_SYMBOL(rvt_release_mmap_info);
+
+/*
+ * open and close keep track of how many times the CQ is mapped,
+ * to avoid releasing it.
+ */
+static void rvt_vma_open(struct vm_area_struct *vma)
+{
+   struct rvt_mmap_info *ip = vma->vm_private_data;
+
+   kref_get(>ref);
+}
+
+static void rvt_vma_close(struct vm_area_struct *vma)
+{
+   struct rvt_mmap_info *ip = vma->vm_private_data;
+
+   kref_put(>ref, rvt_release_mmap_info);
+}
+
+static const struct vm_operations_struct rvt_vm_ops = {
+   .open = rvt_vma_open,
+   .close = rvt_vma_close,
+};
+
 /**
  * rvt_mmap - create a new mmap region
  * @context: the IB user context of the process making the mmap() call
@@ -56,5 +109,90 @@
  */
 int rvt_mmap(struct ib_ucontext *context, struct vm_area_struct *vma)
 {
-   return -EOPNOTSUPP;
+   struct rvt_dev_info *rdi = ib_to_rvt(context->device);
+   unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;
+   unsigned long size = vma->vm_end - vma->vm_start;
+   struct rvt_mmap_info *ip, *pp;
+   int ret = -EINVAL;
+
+   /*
+* Search the device's list of objects waiting for a mmap call.
+* Normally, this list is very short since a call to create a
+* CQ, QP, or SRQ is soon followed by a call to mmap().
+*/
+   spin_lock_irq(>pending_lock);
+   list_for_each_entry_safe(ip, pp, >pending_mmaps,
+pending_mmaps) {
+   /* Only the creator is allowed to mmap the object */
+   if (context != ip->context || (__u64)offset != ip->offset)
+   continue;
+   /* Don't allow a mmap larger than the object. */
+   if (size > ip->size)
+   break;
+
+   list_del_init(>pending_mmaps);
+   spin_unlock_irq(>pending_lock);
+
+   ret = remap_vmalloc_range(vma, ip->obj, 0);
+   if (ret)
+   goto done;
+   vma->vm_ops = _vm_ops;
+   vma->vm_private_data = ip;
+   rvt_vma_open(vma);
+   goto done;
+   }
+   spin_unlock_irq(>pending_lock);
+done:
+   return ret;
+}
+EXPORT_SYMBOL(rvt_mmap);
+
+/*
+ * Allocate information for hfi1_mmap
+ */
+struct rvt_mmap_info *rvt_create_mmap_info(struct rvt_dev_info *rdi,
+  u32 size,
+  struct ib_ucontext *context,
+  void *obj)
+{
+   struct rvt_mmap_info *ip;
+
+   ip = kmalloc(sizeof(*ip), GFP_KERNEL);
+   if (!ip)
+   return ip;
+
+   size = PAGE_ALIGN(size);
+
+   spin_lock_irq(>mmap_offset_lock);
+   if (rdi->mmap_offset == 0)
+   rdi->mmap_offset = PAGE_SIZE;
+   ip->offset = rdi->mmap_offset;
+   rdi->mmap_offset += size;
+   spin_unlock_irq(>mmap_offset_lock);
+
+   INIT_LIST_HEAD(>pending_mmaps);
+   ip->size = size;
+   ip->context = context;
+   ip->obj = obj;
+   kref_init(>ref);
+
+   return ip;
+}
+EXPORT_SYMBOL(rvt_create_mmap_info);
+
+void rvt_update_mmap_info(struct rvt_dev_info *rdi, struct rvt_mmap_info *ip,
+ u32 size, void *obj)
+{
+   size = PAGE_ALIGN(size);
+
+   spin_lock_irq(>mmap_offset_lock);
+   if (rdi->mmap_offset == 0)
+   rdi->mmap_offset = PAGE_SIZE;
+   

[PATCH v2 36/36] IB/rdmavt: Add pkey support

[Dennis Dalessandro]

Add pkey table in rdi per port data structure. Also bring in related pkey
functions. Drivers will still be responsible for allocating and
maintaining the pkey table. However they need to tell rdmavt where to find
the pkey table. We can not move the pkey table up into rdmavt because
drivers need to manipulate this long before registering with it.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |   46 -
 include/rdma/rdma_vt.h|   38 +++
 2 files changed, 64 insertions(+), 20 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index ab4105a..18b5f43 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -154,6 +154,17 @@ static int rvt_query_pkey(struct ib_device *ibdev, u8 
port, u16 index,
 * lock, if a stale value is read and sent to the user so be it there is
 * no way to protect against that anyway.
 */
+   struct rvt_dev_info *rdi = ib_to_rvt(ibdev);
+   int port_index;
+
+   if (index >= rvt_get_npkeys(rdi))
+   return -EINVAL;
+
+   port_index = port - 1; /* IB ports start at 1 our array at 0 */
+   if ((port_index < 0) || (port_index >= rdi->dparms.nports))
+   return -EINVAL;
+
+   *pkey = rvt_get_pkey(rdi, port_index, index);
return 0;
 }
 
@@ -227,19 +238,6 @@ int rvt_register_device(struct rvt_dev_info *rdi)
return -EINVAL;
}
 
-   if (!rdi->dparms.nports) {
-   rvt_pr_err(rdi, "Driver says it has no ports.\n");
-   return -EINVAL;
-   }
-
-   rdi->ports = kcalloc(rdi->dparms.nports,
-sizeof(struct rvt_ibport **),
-GFP_KERNEL);
-   if (!rdi->ports) {
-   rvt_pr_err(rdi, "Could not allocate port mem.\n");
-   return -ENOMEM;
-   }
-
/* Once we get past here we can use the rvt_pr macros */
rvt_mmap_init(rdi);
 
@@ -355,9 +353,25 @@ EXPORT_SYMBOL(rvt_unregister_device);
  * Keep track of a list of ports. No need to have a detach port.
  * They persist until the driver goes away.
  */
-void rvt_attach_port(struct rvt_dev_info *rdi, struct rvt_ibport *port,
-int portnum)
+int rvt_init_port(struct rvt_dev_info *rdi, struct rvt_ibport *port,
+ int portnum, u16 *pkey_table)
 {
+   if (!rdi->dparms.nports) {
+   rvt_pr_err(rdi, "Driver says it has no ports.\n");
+   return -EINVAL;
+   }
+
+   rdi->ports = kcalloc(rdi->dparms.nports,
+sizeof(struct rvt_ibport **),
+GFP_KERNEL);
+   if (!rdi->ports) {
+   rvt_pr_err(rdi, "Could not allocate port mem.\n");
+   return -ENOMEM;
+   }
+
rdi->ports[portnum] = port;
+   rdi->ports[portnum]->pkey_table = pkey_table;
+
+   return 0;
 }
-EXPORT_SYMBOL(rvt_attach_port);
+EXPORT_SYMBOL(rvt_init_port);
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index fd25d23..3a78f20 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -69,6 +69,8 @@
 #define RVT_FLAG_QP_INIT_DRIVER BIT(2)
 #define RVT_FLAG_CQ_INIT_DRIVER BIT(3)
 
+#define RVT_MAX_PKEY_VALUES 16
+
 struct rvt_ibport {
struct rvt_qp __rcu *qp[2];
struct ib_mad_agent *send_agent;/* agent for SMI (traps) */
@@ -125,6 +127,14 @@ struct rvt_ibport {
 
void *priv; /* driver private data */
 
+   /*
+* The pkey table is allocated and maintained by the driver. Drivers
+* need to have access to this before registering with rdmav. However
+* rdmavt will need access to it so drivers need to proviee this during
+* the attach port API call.
+*/
+   u16 *pkey_table;
+
/* TODO: Move sm_ah and smi_ah into here as well*/
 };
 
@@ -178,6 +188,7 @@ struct rvt_driver_params {
int qpn_res_start;
int qpn_res_end;
int nports;
+   int npkeys;
u8 qos_shift;
 };
 
@@ -238,8 +249,6 @@ struct rvt_dev_info {
struct rvt_mregion __rcu *dma_mr;
struct rvt_lkey_table lkey_table;
 
-   /* PKey Table goes here */
-
/* Driver specific helper functions */
struct rvt_driver_provided driver_f;
 
@@ -282,11 +291,32 @@ static inline struct rvt_srq *ibsrq_to_rvtsrq(struct 
ib_srq *ibsrq)
return container_of(ibsrq, struct rvt_srq, ibsrq);
 }
 
+static inline unsigned rvt_get_npkeys(struct rvt_dev_info *rdi)
+{
+   /*
+* All ports have same number of pkeys.
+*/
+   return rdi->dparms.npkeys;
+}
+
+/*
+ * Return the indexed PKEY from the port PKEY table.
+ */
+static inline u16 rvt_get_pkey(struct rvt_dev_info *rdi,

[PATCH v2 34/36] IB/rdmavt: Initialize and teardown of qpn table

[Dennis Dalessandro]

Add table init as well as teardown for handling qpn maps. Drivers can still
provide this functionality by setting the QP_INIT_DRIVER bit.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/qp.c |  197 +
 drivers/infiniband/sw/rdmavt/qp.h |2 
 drivers/infiniband/sw/rdmavt/vt.c |   35 ---
 include/rdma/rdma_vt.h|9 ++
 include/rdma/rdmavt_qp.h  |   33 ++
 5 files changed, 263 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/qp.c 
b/drivers/infiniband/sw/rdmavt/qp.c
index 23a5f68..17dd6ab 100644
--- a/drivers/infiniband/sw/rdmavt/qp.c
+++ b/drivers/infiniband/sw/rdmavt/qp.c
@@ -45,8 +45,205 @@
  *
  */
 
+#include 
+#include 
+#include "vt.h"
 #include "qp.h"
 
+static void get_map_page(struct rvt_qpn_table *qpt, struct rvt_qpn_map *map)
+{
+   unsigned long page = get_zeroed_page(GFP_KERNEL);
+
+   /*
+* Free the page if someone raced with us installing it.
+*/
+
+   spin_lock(>lock);
+   if (map->page)
+   free_page(page);
+   else
+   map->page = (void *)page;
+   spin_unlock(>lock);
+}
+
+/**
+ * init_qpn_table - initialize the QP number table for a device
+ * @qpt: the QPN table
+ */
+static int init_qpn_table(struct rvt_dev_info *rdi, struct rvt_qpn_table *qpt)
+{
+   u32 offset, i;
+   struct rvt_qpn_map *map;
+   int ret = 0;
+
+   if (!(rdi->dparms.qpn_res_end > rdi->dparms.qpn_res_start))
+   return -EINVAL;
+
+   spin_lock_init(>lock);
+
+   qpt->last = rdi->dparms.qpn_start;
+   qpt->incr = rdi->dparms.qpn_inc << rdi->dparms.qos_shift;
+
+   /*
+* Drivers may want some QPs beyond what we need for verbs let them use
+* our qpn table. No need for two. Lets go ahead and mark the bitmaps
+* for those. The reserved range must be *after* the range which verbs
+* will pick from.
+*/
+
+   /* Figure out number of bit maps needed before reserved range */
+   qpt->nmaps = rdi->dparms.qpn_res_start / RVT_BITS_PER_PAGE;
+
+   /* This should always be zero */
+   offset = rdi->dparms.qpn_res_start & RVT_BITS_PER_PAGE_MASK;
+
+   /* Starting with the first reserved bit map */
+   map = >map[qpt->nmaps];
+
+   rvt_pr_info(rdi, "Reserving QPNs from 0x%x to 0x%x for non-verbs use\n",
+   rdi->dparms.qpn_res_start, rdi->dparms.qpn_res_end);
+   for (i = rdi->dparms.qpn_res_start; i < rdi->dparms.qpn_res_end; i++) {
+   if (!map->page) {
+   get_map_page(qpt, map);
+   if (!map->page) {
+   ret = -ENOMEM;
+   break;
+   }
+   }
+   set_bit(offset, map->page);
+   offset++;
+   if (offset == RVT_BITS_PER_PAGE) {
+   /* next page */
+   qpt->nmaps++;
+   map++;
+   offset = 0;
+   }
+   }
+   return ret;
+}
+
+/**
+ * free_qpn_table - free the QP number table for a device
+ * @qpt: the QPN table
+ */
+static void free_qpn_table(struct rvt_qpn_table *qpt)
+{
+   int i;
+
+   for (i = 0; i < ARRAY_SIZE(qpt->map); i++)
+   free_page((unsigned long)qpt->map[i].page);
+}
+
+int rvt_driver_qp_init(struct rvt_dev_info *rdi)
+{
+   int i;
+   int ret = -ENOMEM;
+
+   if (rdi->flags & RVT_FLAG_QP_INIT_DRIVER) {
+   rvt_pr_info(rdi, "Driver is doing QP init.\n");
+   return 0;
+   }
+
+   if (!rdi->dparms.qp_table_size)
+   return -EINVAL;
+
+   /*
+* If driver is not doing any QP allocation then make sure it is
+* providing the necessary QP functions.
+*/
+   if (!rdi->driver_f.free_all_qps)
+   return -EINVAL;
+
+   /* allocate parent object */
+   rdi->qp_dev = kzalloc(sizeof(*rdi->qp_dev), GFP_KERNEL);
+   if (!rdi->qp_dev)
+   return -ENOMEM;
+
+   /* allocate hash table */
+   rdi->qp_dev->qp_table_size = rdi->dparms.qp_table_size;
+   rdi->qp_dev->qp_table_bits = ilog2(rdi->dparms.qp_table_size);
+   rdi->qp_dev->qp_table =
+   kmalloc(rdi->qp_dev->qp_table_size *
+   sizeof(*rdi->qp_dev->qp_table),
+   GFP_KERNEL);
+   if (!rdi->qp_dev->qp_table)
+   goto no_qp_table;
+
+   for (i = 0; i < rdi->qp_dev->qp_table_size; i++)
+   RCU_INIT_POINTER(rdi->qp_dev->qp_table[i], NULL);
+
+   spin_lock_init(>qp_dev->qpt_lock);
+
+   /* initialize qpn map */
+   if (init_qpn_table(rdi, >qp_dev->qpn_table))
+   goto fail_table;
+
+   return ret;
+

[PATCH v2 32/36] IB/rdmavt: Add driver notification for new AH

[Dennis Dalessandro]

Drivers may need to do some work once an address handle has been
created. Add a driver function for this purpose.

Reviewed-by: Ira Weiny 
Reviewed-by: Mike Marciniszyn 
Reviewed-by: Harish Chegondi 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/ah.c |7 +-
 include/rdma/rdma_vt.h|   41 +++--
 2 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/ah.c 
b/drivers/infiniband/sw/rdmavt/ah.c
index 2519db9..621afc3 100644
--- a/drivers/infiniband/sw/rdmavt/ah.c
+++ b/drivers/infiniband/sw/rdmavt/ah.c
@@ -83,8 +83,8 @@ int rvt_check_ah(struct ib_device *ibdev,
!(ah_attr->ah_flags & IB_AH_GRH))
return -EINVAL;
}
-   if (rdi->driver_f.check_ah(ibdev, ah_attr))
-   return -EINVAL;
+   if (rdi->driver_f.check_ah)
+   return rdi->driver_f.check_ah(ibdev, ah_attr);
return 0;
 }
 EXPORT_SYMBOL(rvt_check_ah);
@@ -123,6 +123,9 @@ struct ib_ah *rvt_create_ah(struct ib_pd *pd,
ah->attr = *ah_attr;
atomic_set(>refcount, 0);
 
+   if (dev->driver_f.notify_new_ah)
+   dev->driver_f.notify_new_ah(pd->device, ah_attr, ah);
+
return >ibah;
 }
 
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index a3d6a5b..ef66d2b 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -467,9 +467,21 @@ struct rvt_driver_params {
int nports;
 };
 
-/*
- * Functions that drivers are required to support
- */
+/* Protection domain */
+struct rvt_pd {
+   struct ib_pd ibpd;
+   int user;   /* non-zero if created from user space */
+};
+
+/* Address handle */
+struct rvt_ah {
+   struct ib_ah ibah;
+   struct ib_ah_attr attr;
+   atomic_t refcount;
+   u8 vl;
+   u8 log_pmtu;
+};
+
 struct rvt_dev_info;
 struct rvt_driver_provided {
/*
@@ -478,23 +490,20 @@ struct rvt_driver_provided {
 * instead drivers are responsible for setting the correct callback for
 * this.
 */
+
+   /* ---*/
+   /* Required functions */
+   /* ---*/
int (*port_callback)(struct ib_device *, u8, struct kobject *);
const char * (*get_card_name)(struct rvt_dev_info *rdi);
struct pci_dev * (*get_pci_dev)(struct rvt_dev_info *rdi);
-   int (*check_ah)(struct ib_device *, struct ib_ah_attr *);
-};
 
-/* Protection domain */
-struct rvt_pd {
-   struct ib_pd ibpd;
-   int user;   /* non-zero if created from user space */
-};
-
-/* Address handle */
-struct rvt_ah {
-   struct ib_ah ibah;
-   struct ib_ah_attr attr;
-   atomic_t refcount;
+   /**/
+   /* Optional functions */
+   /**/
+   int (*check_ah)(struct ib_device *, struct ib_ah_attr *);
+   void (*notify_new_ah)(struct ib_device *, struct ib_ah_attr *,
+ struct rvt_ah *);
 };
 
 struct rvt_dev_info {

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 30/36] IB/rdmavt: Move SRQ data structure into rdmavt

[Dennis Dalessandro]

Patch moves the srq data structure into rdmavt in preparation for
removal from qib and hfi1 which will follow in subsequent patches.

Reviewed-by: Ira Weiny 
Reviewed-by: Harish Chegondi 
Signed-off-by: Dennis Dalessandro 
---
 include/rdma/rdma_vt.h |   13 +
 1 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index 36cced6..fcf3ec0 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -349,6 +349,14 @@ struct rvt_qp {
cacheline_aligned_in_smp;
 };
 
+struct rvt_srq {
+   struct ib_srq ibsrq;
+   struct rvt_rq rq;
+   struct rvt_mmap_info *ip;
+   /* send signal when number of RWQEs < limit */
+   u32 limit;
+};
+
 /* End QP section */
 
 /*
@@ -485,6 +493,11 @@ static inline void rvt_get_mr(struct rvt_mregion *mr)
atomic_inc(>refcount);
 }
 
+static inline struct rvt_srq *ibsrq_to_rvtsrq(struct ib_srq *ibsrq)
+{
+   return container_of(ibsrq, struct rvt_srq, ibsrq);
+}
+
 int rvt_register_device(struct rvt_dev_info *rvd);
 void rvt_unregister_device(struct rvt_dev_info *rvd);
 int rvt_check_ah(struct ib_device *ibdev, struct ib_ah_attr *ah_attr);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 31/36] IB/rdmavt: Add an ibport data structure to rdmavt

[Dennis Dalessandro]

Converge the ibport data structures of qib and hfi1 into a common ib
port structure. Also provides a place to keep track of these ports
in case rdmavt needs it. Along with this goes an attach and detach
function for drivers to use to notify rdmavt of the ports.

Reviewed-by: Ira Weiny 
Reviewed-by: Harish Chegondi 
Signed-off-by: Dennis Dalessandro 
---
 drivers/infiniband/sw/rdmavt/vt.c |   24 +
 include/rdma/rdma_vt.h|   66 -
 2 files changed, 89 insertions(+), 1 deletions(-)

diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
b/drivers/infiniband/sw/rdmavt/vt.c
index 7dab0ca..44de280 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -300,6 +300,19 @@ int rvt_register_device(struct rvt_dev_info *rdi)
spin_lock_init(>n_pds_lock);
rdi->n_pds_allocated = 0;
 
+   if (rdi->dparms.nports) {
+   rdi->ports = kcalloc(rdi->dparms.nports,
+sizeof(struct rvt_ibport **),
+GFP_KERNEL);
+   if (!rdi->ports) {
+   rvt_pr_err(rdi, "Could not allocate port mem.\n");
+   ret = -ENOMEM;
+   goto bail_mr;
+   }
+   } else {
+   rvt_pr_warn(rdi, "Driver says it has no ports.\n");
+   }
+
/* We are now good to announce we exist */
ret =  ib_register_device(>ibdev, rdi->driver_f.port_callback);
if (ret) {
@@ -327,3 +340,14 @@ void rvt_unregister_device(struct rvt_dev_info *rdi)
rvt_mr_exit(rdi);
 }
 EXPORT_SYMBOL(rvt_unregister_device);
+
+/*
+ * Keep track of a list of ports. No need to have a detach port.
+ * They persist until the driver goes away.
+ */
+void rvt_attach_port(struct rvt_dev_info *rdi, struct rvt_ibport *port,
+int portnum)
+{
+   rdi->ports[portnum] = port;
+}
+EXPORT_SYMBOL(rvt_attach_port);
diff --git a/include/rdma/rdma_vt.h b/include/rdma/rdma_vt.h
index fcf3ec0..a3d6a5b 100644
--- a/include/rdma/rdma_vt.h
+++ b/include/rdma/rdma_vt.h
@@ -53,6 +53,8 @@
  * rdmavt layer.
  */
 
+#include 
+#include 
 #include "ib_verbs.h"
 
 #define RVT_MULTICAST_LID_BASE 0xC000
@@ -359,6 +361,65 @@ struct rvt_srq {
 
 /* End QP section */
 
+struct rvt_ibport {
+   struct rvt_qp __rcu *qp[2];
+   struct ib_mad_agent *send_agent;/* agent for SMI (traps) */
+   struct rb_root mcast_tree;
+   spinlock_t lock;/* protect changes in this struct */
+
+   /* non-zero when timer is set */
+   unsigned long mkey_lease_timeout;
+   unsigned long trap_timeout;
+   __be64 gid_prefix;  /* in network order */
+   __be64 mkey;
+   u64 tid;
+   u32 port_cap_flags;
+   u32 pma_sample_start;
+   u32 pma_sample_interval;
+   __be16 pma_counter_select[5];
+   u16 pma_tag;
+   u16 mkey_lease_period;
+   u16 sm_lid;
+   u8 sm_sl;
+   u8 mkeyprot;
+   u8 subnet_timeout;
+   u8 vl_high_limit;
+
+   /*
+* Driver is expected to keep these up to date. These
+* counters are informational only and not required to be
+* completely accurate.
+*/
+   u64 n_rc_resends;
+   u64 n_seq_naks;
+   u64 n_rdma_seq;
+   u64 n_rnr_naks;
+   u64 n_other_naks;
+   u64 n_loop_pkts;
+   u64 n_pkt_drops;
+   u64 n_vl15_dropped;
+   u64 n_rc_timeouts;
+   u64 n_dmawait;
+   u64 n_unaligned;
+   u64 n_rc_dupreq;
+   u64 n_rc_seqnak;
+   u16 pkey_violations;
+   u16 qkey_violations;
+   u16 mkey_violations;
+
+   /* Hot-path per CPU counters to avoid cacheline trading to update */
+   u64 z_rc_acks;
+   u64 z_rc_qacks;
+   u64 z_rc_delayed_comp;
+   u64 __percpu *rc_acks;
+   u64 __percpu *rc_qacks;
+   u64 __percpu *rc_delayed_comp;
+
+   void *priv; /* driver private data */
+
+   /* TODO: Move sm_ah and smi_ah into here as well*/
+};
+
 /*
  * Things that are driver specific, module parameters in hfi1 and qib
  */
@@ -403,6 +464,7 @@ struct rvt_driver_params {
 * For instance special module parameters. Goes here.
 */
unsigned int lkey_table_size;
+   int nports;
 };
 
 /*
@@ -465,6 +527,7 @@ struct rvt_dev_info {
spinlock_t n_ahs_lock; /* Protect ah allocated count */
 
int flags;
+   struct rvt_ibport **ports;
 };
 
 static inline struct rvt_pd *ibpd_to_rvtpd(struct ib_pd *ibpd)
@@ -501,9 +564,10 @@ static inline struct rvt_srq *ibsrq_to_rvtsrq(struct 
ib_srq *ibsrq)
 int rvt_register_device(struct rvt_dev_info *rvd);
 void rvt_unregister_device(struct rvt_dev_info *rvd);
 int rvt_check_ah(struct ib_device *ibdev, struct ib_ah_attr *ah_attr);
+void rvt_attach_port(struct rvt_dev_info *rdi, struct rvt_ibport *port,
+int 

[PATCH] IB/core: sysfs.c: Fix PerfMgt ClassPortInfo handling

[Hal Rosenstock]

Port number is not part of ClassPortInfo attribute but is
still needed as a parameter when invoking process_mad.

To properly handle this attribute, port_num is added as a
parameter to get_counter_table and get_perf_mad was changed
not to store port_num in the attribute itself when it's
querying the ClassPortInfo attribute.

This handles issue pointed out by Matan Barak 

Signed-off-by: Hal Rosenstock 
Acked-by: Matan Barak 
---
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 539040f..2daf832 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -438,7 +438,8 @@ static int get_perf_mad(struct ib_device *dev, int 
port_num, int attr,
in_mad->mad_hdr.method= IB_MGMT_METHOD_GET;
in_mad->mad_hdr.attr_id   = attr;
 
-   in_mad->data[41] = port_num;/* PortSelect field */
+   if (attr != IB_PMA_CLASS_PORT_INFO)
+   in_mad->data[41] = port_num;/* PortSelect field */
 
if ((dev->process_mad(dev, IB_MAD_IGNORE_MKEY,
 port_num, NULL, NULL,
@@ -714,11 +715,12 @@ err:
  * Figure out which counter table to use depending on
  * the device capabilities.
  */
-static struct attribute_group *get_counter_table(struct ib_device *dev)
+static struct attribute_group *get_counter_table(struct ib_device *dev,
+int port_num)
 {
struct ib_class_port_info cpi;
 
-   if (get_perf_mad(dev, 0, IB_PMA_CLASS_PORT_INFO,
+   if (get_perf_mad(dev, port_num, IB_PMA_CLASS_PORT_INFO,
, 40, sizeof(cpi)) >= 0) {
 
if (cpi.capability_mask && IB_PMA_CLASS_CAP_EXT_WIDTH)
@@ -776,7 +778,7 @@ static int add_port(struct ib_device *device, int port_num,
goto err_put;
}
 
-   p->pma_table = get_counter_table(device);
+   p->pma_table = get_counter_table(device, port_num);
ret = sysfs_create_group(>kobj, p->pma_table);
if (ret)
goto err_put_gid_attrs;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mad: Ensure fairness in ib_mad_completion_handler

[ira.weiny]

On Mon, Dec 28, 2015 at 06:51:30PM +0200, Eli Cohen wrote:
> On Thu, Dec 10, 2015 at 04:52:30PM -0500, ira.we...@intel.com wrote:
> > From: Dean Luick 
> > 
> >  
> > @@ -2555,6 +2567,7 @@ static void ib_mad_completion_handler(struct 
> > work_struct *work)
> >  {
> > struct ib_mad_port_private *port_priv;
> > struct ib_wc wc;
> > +   int count = 0;
> >  
> > port_priv = container_of(work, struct ib_mad_port_private, work);
> > ib_req_notify_cq(port_priv->cq, IB_CQ_NEXT_COMP);
> 
> I think you shoudld push the call to ib_req_notify_cq outside the
> while loop. You don't need to arm the CQ if you re-queued the work.
> Only when you have drained the CQ should you re-arm.

Will it hurt to rearm?  The way the code stands I think the worse that will
happen is an extra work item scheduled and an ib_poll_cq call.

I'm not quite sure what you mean about moving the ib_req_notify_cq outside of
the while loop.  It seems like to do what you say we would need another work
item which just does ib_poll_cq.  Is that what you meant?

Ira

> 
> > @@ -2574,6 +2587,11 @@ static void ib_mad_completion_handler(struct 
> > work_struct *work)
> > }
> > } else
> > mad_error_handler(port_priv, );
> > +
> > +   if (++count > MAD_COMPLETION_PROC_LIMIT) {
> > +   queue_work(port_priv->wq, _priv->work);
> > +   break;
> > +   }
> > }
> >  }
> >  
> > -- 
> > 1.8.2
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Generic InfiniBand transport done in software

[Dennis Dalessandro]

On Sun, Dec 27, 2015 at 07:54:46PM +0200, Moni Shoua wrote:


Point is others have looked at the code. No issues have been called out to
date as to why what is there won't work for everyone.


http://marc.info/?l=linux-rdma=144968107508268=2
Your answer to the send() issue is first an agreement with the comment
and later says that it can't be done because of how PIO and SDMA
(Intel specific implementation)
This is an example for a discussion that never ended with an agreement.


No. PIO and SDMA is driver specific and lives in the driver. Rdmavt has no 
concept of this. I'm agreeing that the send will be generic and have no hw 
specific stuff.



Yes it is specific to Intel *now*, that doesn't mean it should stay that
way. Rdmavt could, and in my opinion should, be extended to support
soft-roce. I don't think replicating the same thing is a great idea.


But you post *now* a so called generic driver so it must now fit any
possible driver (including Soft RoCE)


As I've stated a number of times across multiple threads: It must not do 
anything that would prevent another driver from using it.



As to the location, where do you think it should go. drivers/infiniband/sw
makes the most sense to me, but open to suggestions.

And for the question of why publish when it's not ready, the better question
is why not?  Is it not good to see the work in progress as it evolves so the
community can provide feedback?


What kind of a feedback you expect when I don't have an idea about
your plans for rdmavt
Interfaces, flows, data structures... all is missing from the
documentation to rdmavt.


I expect feedback based on the code submissions. More will be coming 
shortly. I have taken all the feedback from the first post and will be 
sending a v2 shortly.


-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 01/36] IB/rdmavt: Create module framework and handle driver registration

[Leon Romanovsky]

On Mon, Dec 28, 2015 at 12:59:45PM -0800, Dennis Dalessandro wrote:
> This patch introduces the basics for a new module called rdma_vt. This new
> driver is a software implementation of the InfiniBand verbs and aims to
> replace the multiple implementations that exist and duplicate each others'
> code.
> 
> While the call to actually register the device with the IB core happens in
> rdma_vt, most of the work is still done in the drivers themselves. This
> will be changing in a follow on patch this is just laying the groundwork
> for this infrastructure.
> 
> Reviewed-by: Ira Weiny 
> Reviewed-by: Mike Marciniszyn 
> Signed-off-by: Dennis Dalessandro 
> ---
> Changes since v1:
> remove rdmavt specific driver version
> move copyright above license text
> 
>  MAINTAINERS   |6 ++
>  drivers/infiniband/Kconfig|2 +
>  drivers/infiniband/Makefile   |1 
>  drivers/infiniband/sw/Makefile|1 
>  drivers/infiniband/sw/rdmavt/Kconfig  |6 ++
>  drivers/infiniband/sw/rdmavt/Makefile |   10 
>  drivers/infiniband/sw/rdmavt/vt.c |   83 
> +
>  drivers/infiniband/sw/rdmavt/vt.h |   53 +
>  include/rdma/rdma_vt.h|   70 
>  9 files changed, 232 insertions(+), 0 deletions(-)
>  create mode 100644 drivers/infiniband/sw/Makefile
>  create mode 100644 drivers/infiniband/sw/rdmavt/Kconfig
>  create mode 100644 drivers/infiniband/sw/rdmavt/Makefile
>  create mode 100644 drivers/infiniband/sw/rdmavt/vt.c
>  create mode 100644 drivers/infiniband/sw/rdmavt/vt.h
>  create mode 100644 include/rdma/rdma_vt.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 69c8a9c..37fc192 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -8880,6 +8880,12 @@ L: rds-de...@oss.oracle.com (moderated for 
> non-subscribers)
>  S:   Supported
>  F:   net/rds/
>  
> +RDMAVT - RDMA verbs software
> +M:   Dennis Dalessandro 
> +L:   linux-rdma@vger.kernel.org
> +S:   Supported
> +F:   drivers/infiniband/sw/rdmavt
> +
>  READ-COPY UPDATE (RCU)
>  M:   "Paul E. McKenney" 
>  M:   Josh Triplett 
> diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
> index aa26f3c..1b00bb8 100644
> --- a/drivers/infiniband/Kconfig
> +++ b/drivers/infiniband/Kconfig
> @@ -72,4 +72,6 @@ source "drivers/infiniband/ulp/srpt/Kconfig"
>  source "drivers/infiniband/ulp/iser/Kconfig"
>  source "drivers/infiniband/ulp/isert/Kconfig"
>  
> +source "drivers/infiniband/sw/rdmavt/Kconfig"
> +
>  endif # INFINIBAND
> diff --git a/drivers/infiniband/Makefile b/drivers/infiniband/Makefile
> index dc21836..fad0b44 100644
> --- a/drivers/infiniband/Makefile
> +++ b/drivers/infiniband/Makefile
> @@ -1,3 +1,4 @@
>  obj-$(CONFIG_INFINIBAND) += core/
>  obj-$(CONFIG_INFINIBAND) += hw/
>  obj-$(CONFIG_INFINIBAND) += ulp/
> +obj-$(CONFIG_INFINIBAND) += sw/
> diff --git a/drivers/infiniband/sw/Makefile b/drivers/infiniband/sw/Makefile
> new file mode 100644
> index 000..988b6a0
> --- /dev/null
> +++ b/drivers/infiniband/sw/Makefile
> @@ -0,0 +1 @@
> +obj-$(CONFIG_INFINIBAND_RDMAVT)  += rdmavt/
> diff --git a/drivers/infiniband/sw/rdmavt/Kconfig 
> b/drivers/infiniband/sw/rdmavt/Kconfig
> new file mode 100644
> index 000..11aa6a3
> --- /dev/null
> +++ b/drivers/infiniband/sw/rdmavt/Kconfig
> @@ -0,0 +1,6 @@
> +config INFINIBAND_RDMAVT
> + tristate "RDMA verbs transport library"
> + depends on 64BIT
> + default m
> + ---help---
> + This is a common software verbs provider for RDMA networks.
> diff --git a/drivers/infiniband/sw/rdmavt/Makefile 
> b/drivers/infiniband/sw/rdmavt/Makefile
> new file mode 100644
> index 000..98a664d
> --- /dev/null
> +++ b/drivers/infiniband/sw/rdmavt/Makefile
> @@ -0,0 +1,10 @@
> +#
> +# rdmavt driver
> +#
> +#
> +#
> +# Called from the kernel module build system.
> +#
> +obj-$(CONFIG_INFINIBAND_RDMAVT) += rdmavt.o
> +
> +rdmavt-y := vt.o
> diff --git a/drivers/infiniband/sw/rdmavt/vt.c 
> b/drivers/infiniband/sw/rdmavt/vt.c
> new file mode 100644
> index 000..aa325db
> --- /dev/null
> +++ b/drivers/infiniband/sw/rdmavt/vt.c
> @@ -0,0 +1,83 @@
> +/*
> + * Copyright(c) 2015 Intel Corporation.
> + *
> + * This file is provided under a dual BSD/GPLv2 license.  When using or
> + * redistributing this file, you may do so under either license.
> + *
> + * GPL LICENSE SUMMARY
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of version 2 of the GNU General Public License as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful, but
> + * WITHOUT ANY WARRANTY; without even the implied warranty of
> + * 

Re: [PATCH] IB/mad: Ensure fairness in ib_mad_completion_handler

[ira.weiny]

On Tue, Dec 29, 2015 at 01:25:33AM +0200, Eli Cohen wrote:
> On Mon, Dec 28, 2015 at 06:05:46PM -0500, ira.weiny wrote:
> > 
> > Will it hurt to rearm?  The way the code stands I think the worse that will
> > happen is an extra work item scheduled and an ib_poll_cq call.
> 
> If you re-arm unconditionally you call for extra interrupts which you
> can do without. When you break the loop of processing completions since
> you exhausted the quota, you queue the work so you can continue
> processing completions in the next time slot of the work task. After
> queueing the work you should call "return" instead of "break".
> If you processed all the completions before reaching
> MAD_COMPLETION_PROC_LIMIT you will exit the while loop and then
> re-arming can take place.

I'm still confused.  Here is the code with the patch applied:


/* 
 * IB MAD completion callback 
 */ 
static void ib_mad_completion_handler(struct work_struct *work) 
{ 
struct ib_mad_port_private *port_priv; 
struct ib_wc wc; 
int count = 0; 

port_priv = container_of(work, struct ib_mad_port_private, work);
ib_req_notify_cq(port_priv->cq, IB_CQ_NEXT_COMP);

while (ib_poll_cq(port_priv->cq, 1, ) == 1) {
if (wc.status == IB_WC_SUCCESS) {
switch (wc.opcode) {
case IB_WC_SEND:
ib_mad_send_done_handler(port_priv, );
break;
case IB_WC_RECV:
ib_mad_recv_done_handler(port_priv, );
break;
default:
BUG_ON(1);
break;
}
} else
mad_error_handler(port_priv, );

if (++count > MAD_COMPLETION_PROC_LIMIT) {
queue_work(port_priv->wq, _priv->work);
break;
}
}
}


How is "return" any different than "break"?  Calling return will still result
in a rearm on the next work task.

Perhaps this code is wrong in the first place?  Should it call ib_req_notify_cq
after the while loop?  This code has been this way forever...

1da177e4c3f41   (Linus Torvalds 2005-04-16 15:20:36 -0700   2568) 
ib_req_notify_cq(port_priv->cq, IB_CQ_NEXT_COMP);
1da177e4c3f41   (Linus Torvalds 2005-04-16 15:20:36 -0700   2569)
1da177e4c3f41   (Linus Torvalds 2005-04-16 15:20:36 -0700   2570)   while 
(ib_poll_cq(port_priv->cq, 1, ) == 1) {


Ira


> 
> > 
> > I'm not quite sure what you mean about moving the ib_req_notify_cq outside 
> > of
> > the while loop.  It seems like to do what you say we would need another work
> > item which just does ib_poll_cq.  Is that what you meant?
> > 
> > Ira
> > 
> > > 
> > > > @@ -2574,6 +2587,11 @@ static void ib_mad_completion_handler(struct 
> > > > work_struct *work)
> > > > }
> > > > } else
> > > > mad_error_handler(port_priv, );
> > > > +
> > > > +   if (++count > MAD_COMPLETION_PROC_LIMIT) {
> > > > +   queue_work(port_priv->wq, _priv->work);
> > > > +   break;
> > > > +   }
> > > > }
> > > >  }
> > > >  
> > > > -- 
> > > > 1.8.2
> > > > 
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majord...@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majord...@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/mad: Ensure fairness in ib_mad_completion_handler

[Eli Cohen]

On Mon, Dec 28, 2015 at 06:05:46PM -0500, ira.weiny wrote:
> 
> Will it hurt to rearm?  The way the code stands I think the worse that will
> happen is an extra work item scheduled and an ib_poll_cq call.

If you re-arm unconditionally you call for extra interrupts which you
can do without. When you break the loop of processing completions since
you exhausted the quota, you queue the work so you can continue
processing completions in the next time slot of the work task. After
queueing the work you should call "return" instead of "break".
If you processed all the completions before reaching
MAD_COMPLETION_PROC_LIMIT you will exit the while loop and then
re-arming can take place.

> 
> I'm not quite sure what you mean about moving the ib_req_notify_cq outside of
> the while loop.  It seems like to do what you say we would need another work
> item which just does ib_poll_cq.  Is that what you meant?
> 
> Ira
> 
> > 
> > > @@ -2574,6 +2587,11 @@ static void ib_mad_completion_handler(struct 
> > > work_struct *work)
> > >   }
> > >   } else
> > >   mad_error_handler(port_priv, );
> > > +
> > > + if (++count > MAD_COMPLETION_PROC_LIMIT) {
> > > + queue_work(port_priv->wq, _priv->work);
> > > + break;
> > > + }
> > >   }
> > >  }
> > >  
> > > -- 
> > > 1.8.2
> > > 
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majord...@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] IB/core: sysfs.c: Fix PerfMgt ClassPortInfo handling

[ira.weiny]

On Mon, Dec 28, 2015 at 04:53:18PM -0500, Hal Rosenstock wrote:
> 
> Port number is not part of ClassPortInfo attribute but is
> still needed as a parameter when invoking process_mad.
> 
> To properly handle this attribute, port_num is added as a
> parameter to get_counter_table and get_perf_mad was changed
> not to store port_num in the attribute itself when it's
> querying the ClassPortInfo attribute.
> 
> This handles issue pointed out by Matan Barak 
> 
> Signed-off-by: Hal Rosenstock 

Reviewed-by: Ira Weiny 

> Acked-by: Matan Barak 
> ---
> diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
> index 539040f..2daf832 100644
> --- a/drivers/infiniband/core/sysfs.c
> +++ b/drivers/infiniband/core/sysfs.c
> @@ -438,7 +438,8 @@ static int get_perf_mad(struct ib_device *dev, int 
> port_num, int attr,
>   in_mad->mad_hdr.method= IB_MGMT_METHOD_GET;
>   in_mad->mad_hdr.attr_id   = attr;
>  
> - in_mad->data[41] = port_num;/* PortSelect field */
> + if (attr != IB_PMA_CLASS_PORT_INFO)
> + in_mad->data[41] = port_num;/* PortSelect field */
>  
>   if ((dev->process_mad(dev, IB_MAD_IGNORE_MKEY,
>port_num, NULL, NULL,
> @@ -714,11 +715,12 @@ err:
>   * Figure out which counter table to use depending on
>   * the device capabilities.
>   */
> -static struct attribute_group *get_counter_table(struct ib_device *dev)
> +static struct attribute_group *get_counter_table(struct ib_device *dev,
> +  int port_num)
>  {
>   struct ib_class_port_info cpi;
>  
> - if (get_perf_mad(dev, 0, IB_PMA_CLASS_PORT_INFO,
> + if (get_perf_mad(dev, port_num, IB_PMA_CLASS_PORT_INFO,
>   , 40, sizeof(cpi)) >= 0) {
>  
>   if (cpi.capability_mask && IB_PMA_CLASS_CAP_EXT_WIDTH)
> @@ -776,7 +778,7 @@ static int add_port(struct ib_device *device, int 
> port_num,
>   goto err_put;
>   }
>  
> - p->pma_table = get_counter_table(device);
> + p->pma_table = get_counter_table(device, port_num);
>   ret = sysfs_create_group(>kobj, p->pma_table);
>   if (ret)
>   goto err_put_gid_attrs;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html