> Should I expect to be able to use the kernel directories in branches/iwarp > directly with linux-2.6.17.1? It looks like your branch may be out > of date with respect to trunk for a few files. I used it anyway and > it does seem to build and run.
I haven't tried branches/iwarp with a 2.6.17 kernel. It works fine with 2.6.16 though, and I expect it to work fine in 2.6.17. The branch is a snapshot of the main trunk and we only update it occasionally. > > In the userspace source, amso_create_qp limits max_send_sge and > max_recv_sge to 4. Is this really the hardware limit? It seems > quite low. > Yep. That's a HW limit. > Should I expect the examples in > branches/iwarp/src/userspace/libibverbs/examples to work? I was > hoping to use rc_pingpong.c as a way to understand what was going > wrong with my code, but it exits when it finds that its local lid is > zero (line 578). Those examples only work for IB transports. The examples in librdma/examples will run over iwarp because they utilize the RDMA CMA. > > One spot in my code where I'm trying to understand why libamso > errors is this transition to RTS (not using rdmacm, just bringing > it up by hand): > > /* transition qp to ready-to-send */ > mask = > IBV_QP_STATE > | IBV_QP_SQ_PSN > | IBV_QP_MAX_QP_RD_ATOMIC > | IBV_QP_TIMEOUT > | IBV_QP_RETRY_CNT > | IBV_QP_RNR_RETRY; > memset(&attr, 0, sizeof(attr)); > attr.qp_state = IBV_QPS_RTS; > attr.sq_psn = 0; > attr.max_rd_atomic = 1; > attr.timeout = 26; /* 4.096us * 2^26 = 5 min */ > attr.retry_cnt = 20; > attr.rnr_retry = 20; > ret = ibv_modify_qp(qp, &attr, mask); > if (ret) > error_xerrno(ret, "%s: ibv_modify_qp RTR -> RTS", __func__); > > The return value is 11, EAGAIN. > > With C2_DEBUG on, the kernel says to the console: > > c2: c2_qp_modify:145 qp=ffff81003e71f180, IB_QPS_RTR --> IB_QPS_RTS > c2: c2_qp_modify: c2_errno=-11 > c2: c2_qp_modify:243 qp=ffff81003e71f180, cur_state=IB_QPS_RTR > > I'm guessing one of those values must be off, but can't see where > anything is enforced in the lib or kernel driver. Some of these > fields don't make sense for non-IB fabrics. Just using a mask of > IBV_QP_STATE caused the same return value. Can you see the problem > right off? (This code does work fine on mthca.) > You need to use librdmacm to setup iwarp connections. That's the only way it will work for the amso device. See librdma/examples. I also posted a patch to perftest/rdma_lat.c and rdma_bw.c that added a -c option to utilize the RDMA CMA. The patch didn't get pulled in, however... Steve. _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
