Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
On Mon, Oct 24, 2011 at 2:12 PM, David Miller da...@davemloft.net wrote: 1. We would like to recommend that all the patches get included in Roland's infiniband tree since it has build dependencies. This is fine with me. It just means that Roland's tree has to have net-next included in it already, because otherwise the paths to the cxgb4 driver under drivers/net won't be right. OK, let's do that... - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
On 21-10-2011 02:27, David Miller wrote: From: Steve Wise sw...@opengridcomputing.com Date: Thu, 20 Oct 2011 12:28:07 -0500 On 10/20/2011 12:17 PM, Roland Dreier wrote: I believe 5 and 7 have build dependencies. Right, missed that one too. But it seems 4,6,8,9,10 are independent of the rest of the series? ie I can trivially apply them and then worry about working out the drivers/net / drivers/infiniband interdependency a bit later? Some of these might be dependent on prior patches the series. But if they aren't, yes, you could do that. So, how do you guys want to do this? If you give me a list of which patches I should put into net-next and leave the rest to the infiniband tree, that'd work fine for me as long as net-next is left in a working state independent of the infiniband tree. Hi Dave Miller/Roland, With respect to above dependencies we did some experiments and found following things 1. We can apply three cxgb4 patches, 01 02 and 03, on net-next tree successfully and build it. 2. Out of 7 RDMA/cxgb4 patches only 04, 08 and 10 can be applied trivially and driver can be built successfully. If we try to apply remaining patches, 05 06 07 and 09, either they will fail to apply or give build failure. Moreover patches 05, 06, 07 and 09 can be applied on top of 04, 08 and 10 cleanly. Based on above results we would like to propose following two things. 1. We would like to recommend that all the patches get included in Roland's infiniband tree since it has build dependencies. 2. Alternatively, - Patches 01, 02 and 03 can be included in net-next tree. - Patches 04, 08 and 10 can be included in Roland's infiniband tree at present. - Patches 05, 06, 07 and 09 have to wait till the net-next hits the 3.2-rc1. Please let us know if you have any other suggestions. Thanks, Vipul -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
From: Vipul Pandya vi...@chelsio.com Date: Mon, 24 Oct 2011 20:46:31 +0530 1. We would like to recommend that all the patches get included in Roland's infiniband tree since it has build dependencies. This is fine with me. It just means that Roland's tree has to have net-next included in it already, because otherwise the paths to the cxgb4 driver under drivers/net won't be right. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
On 10/19/2011 05:12 PM, Roland Dreier wrote: This looks like the only drivers/infiniband patch that depends on the drivers/net changes? - R. I believe 5 and 7 have build dependencies. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
I believe 5 and 7 have build dependencies. Right, missed that one too. But it seems 4,6,8,9,10 are independent of the rest of the series? ie I can trivially apply them and then worry about working out the drivers/net / drivers/infiniband interdependency a bit later? - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
On 10/20/2011 12:17 PM, Roland Dreier wrote: I believe 5 and 7 have build dependencies. Right, missed that one too. But it seems 4,6,8,9,10 are independent of the rest of the series? ie I can trivially apply them and then worry about working out the drivers/net / drivers/infiniband interdependency a bit later? Some of these might be dependent on prior patches the series. But if they aren't, yes, you could do that. Stevo -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
From: Steve Wise sw...@opengridcomputing.com Date: Thu, 20 Oct 2011 12:28:07 -0500 On 10/20/2011 12:17 PM, Roland Dreier wrote: I believe 5 and 7 have build dependencies. Right, missed that one too. But it seems 4,6,8,9,10 are independent of the rest of the series? ie I can trivially apply them and then worry about working out the drivers/net / drivers/infiniband interdependency a bit later? Some of these might be dependent on prior patches the series. But if they aren't, yes, you could do that. So, how do you guys want to do this? If you give me a list of which patches I should put into net-next and leave the rest to the infiniband tree, that'd work fine for me as long as net-next is left in a working state independent of the infiniband tree. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
- add module option db_fc_threshold which is the count of active QPs
that trigger automatic db flow control mode.
- automatically transition to/from flow control mode when the active qp
count crosses db_fc_theshold.
- add more db debugfs stats
- on DB DROP event from the LLD, recover all the iwarp queues.
Signed-off-by: Vipul Pandya vi...@chelsio.com
Signed-off-by: Steve Wise sw...@opengridcomputing.com
---
drivers/infiniband/hw/cxgb4/device.c | 176 ++-
drivers/infiniband/hw/cxgb4/iw_cxgb4.h | 24 -
drivers/infiniband/hw/cxgb4/qp.c | 47 -
drivers/infiniband/hw/cxgb4/t4.h | 24 +
4 files changed, 259 insertions(+), 12 deletions(-)
diff --git a/drivers/infiniband/hw/cxgb4/device.c
b/drivers/infiniband/hw/cxgb4/device.c
index 9062ed9..bdb398f 100644
--- a/drivers/infiniband/hw/cxgb4/device.c
+++ b/drivers/infiniband/hw/cxgb4/device.c
@@ -246,6 +246,8 @@ static const struct file_operations stag_debugfs_fops = {
.llseek = default_llseek,
};
+static char *db_state_str[] = {NORMAL, FLOW_CONTROL, RECOVERY};
+
static int stats_show(struct seq_file *seq, void *v)
{
struct c4iw_dev *dev = seq-private;
@@ -272,6 +274,9 @@ static int stats_show(struct seq_file *seq, void *v)
seq_printf(seq, DB FULL: %10llu\n, dev-rdev.stats.db_full);
seq_printf(seq, DB EMPTY: %10llu\n, dev-rdev.stats.db_empty);
seq_printf(seq, DB DROP: %10llu\n, dev-rdev.stats.db_drop);
+ seq_printf(seq, DB State: %s Transitions %llu\n,
+ db_state_str[dev-db_state],
+ dev-rdev.stats.db_state_transitions);
return 0;
}
@@ -295,6 +300,7 @@ static ssize_t stats_clear(struct file *file, const char
__user *buf,
dev-rdev.stats.db_full = 0;
dev-rdev.stats.db_empty = 0;
dev-rdev.stats.db_drop = 0;
+ dev-rdev.stats.db_state_transitions = 0;
mutex_unlock(dev-rdev.stats.lock);
return count;
}
@@ -677,8 +683,11 @@ static int disable_qp_db(int id, void *p, void *data)
static void stop_queues(struct uld_ctx *ctx)
{
spin_lock_irq(ctx-dev-lock);
- ctx-dev-db_state = FLOW_CONTROL;
- idr_for_each(ctx-dev-qpidr, disable_qp_db, NULL);
+ if (ctx-dev-db_state == NORMAL) {
+ ctx-dev-rdev.stats.db_state_transitions++;
+ ctx-dev-db_state = FLOW_CONTROL;
+ idr_for_each(ctx-dev-qpidr, disable_qp_db, NULL);
+ }
spin_unlock_irq(ctx-dev-lock);
}
@@ -693,9 +702,165 @@ static int enable_qp_db(int id, void *p, void *data)
static void resume_queues(struct uld_ctx *ctx)
{
spin_lock_irq(ctx-dev-lock);
- ctx-dev-db_state = NORMAL;
- idr_for_each(ctx-dev-qpidr, enable_qp_db, NULL);
+ if (ctx-dev-qpcnt = db_fc_threshold
+ ctx-dev-db_state == FLOW_CONTROL) {
+ ctx-dev-db_state = NORMAL;
+ ctx-dev-rdev.stats.db_state_transitions++;
+ idr_for_each(ctx-dev-qpidr, enable_qp_db, NULL);
+ }
+ spin_unlock_irq(ctx-dev-lock);
+}
+
+struct qp_list {
+ unsigned idx;
+ struct c4iw_qp **qps;
+};
+
+static int add_and_ref_qp(int id, void *p, void *data)
+{
+ struct qp_list *qp_listp = data;
+ struct c4iw_qp *qp = p;
+
+ c4iw_qp_add_ref(qp-ibqp);
+ qp_listp-qps[qp_listp-idx++] = qp;
+ return 0;
+}
+
+static int count_qps(int id, void *p, void *data)
+{
+ unsigned *countp = data;
+ (*countp)++;
+ return 0;
+}
+
+static void deref_qps(struct qp_list qp_list)
+{
+ int idx;
+
+ for (idx = 0; idx qp_list.idx; idx++)
+ c4iw_qp_rem_ref(qp_list.qps[idx]-ibqp);
+}
+
+static void recover_lost_dbs(struct uld_ctx *ctx, struct qp_list *qp_list)
+{
+ int idx;
+ int ret;
+
+ for (idx = 0; idx qp_list-idx; idx++) {
+ struct c4iw_qp *qp = qp_list-qps[idx];
+
+ ret = cxgb4_sync_txq_pidx(qp-rhp-rdev.lldi.ports[0],
+ qp-wq.sq.qid,
+ t4_sq_host_wq_pidx(qp-wq),
+ t4_sq_wq_size(qp-wq));
+ if (ret) {
+ printk(KERN_ERR MOD %s: Fatal error -
+ DB overflow recovery failed -
+ error syncing SQ qid %u\n,
+ pci_name(ctx-lldi.pdev), qp-wq.sq.qid);
+ return;
+ }
+
+ ret = cxgb4_sync_txq_pidx(qp-rhp-rdev.lldi.ports[0],
+ qp-wq.rq.qid,
+ t4_rq_host_wq_pidx(qp-wq),
+ t4_rq_wq_size(qp-wq));
+
+ if (ret) {
+ printk(KERN_ERR MOD %s: Fatal error -
+ DB overflow recovery failed -
+ error
Re: [PATCH 07/10] RDMA/cxgb4: DB Drop Recovery for RDMA and LLD queues.
This looks like the only drivers/infiniband patch that depends on the drivers/net changes? - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
