2010/6/21 Dotan Barak <[email protected]>: > On 20/06/2010 07:51, Ding Dinghua wrote: >> >> hello, >> >> 2010/6/19 Dotan Barak<[email protected]>: >> >>> >>> >>>> >>>> I call rdma_create_id to create an ib id, then do resolve remote addr, >>>> resolve route work, then >>>> setup qp and call rdma_connect to setup connection, before ack or >>>> error replies, the thread will >>>> wait on a wait queue. The listening ib id of remote node will catch >>>> the connect request, >>>> setup qp, allocate and map pages to construct the RDMA-WRITE space, >>>> and call rdma_accept to reply >>>> the request. >>>> >>>> Some other information which may be useful: >>>> 1.All the "RETRY EXCEEDED" problems happened when there were two >>>> connections which use RDMA-WRITE to transfer things. >>>> And the latter connection had a high possibility to get into this >>>> problem. >>>> 2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE >>>> space is 256MB each(that is, for two connections, consumes 512MB mem), >>>> when the RDMA-WRITE space is 64MB, this problem never happened in our >>>> test. Remote node's total memory is 2GB. >>>> >>>> Thanks a lot. >>>> >>>> >>> >>> Some more questions: >>> * Is the WR that "produces" the RETRY EXCEEDED is the first one/last >>> one/in >>> the middle? >>> >> >> it's the first one >> >> >>> >>> * Which values are you using in the QP context for retry exceeded counter >>> + >>> retry timeout? >>> * Did you try to increase those values? >>> >> >> I haven't set these values(actually I don't know where to set these >> values), i just set max_send_wr and max_send_sge >> fields of struct ib_qp_cap when creating qp. >> >> > > Can you perform query QP after establishing a connection between the QPs and > check those values? >
All the QPs (local and remote, 2 connections) : 3 qp_state, 19 retry_cnt, 7 timeout, that's, all QPs' qp_state is IB_QPS_RTS(Should remote QP's state be this, or IB_QPS_RTR? But QP's state of first connection is the same and it can work...). >>> * How many more QPs do you have between those nodes and which operations >>> do >>> they use >>> (only RDMA-WRITEs?) >>> >>> >> >> 4096 QPs for each connection, only do RDMA-WRITES. >> > > So, you send in parallel total of 4K (QPs) * 64M (Bytes) = 256 GB > (am i missing something, or this is the amount of data that will be sent > between two nodes?) > if RDMA-WRITE space is 64M(Bytes), this means upper level applications send at most 64M(Bytes) to remote node one time. These QPs may send to different piece of the 64M space in parallel. > Dotan > -- Ding Dinghua -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
