I call rdma_create_id to create an ib id, then do resolve remote addr,
resolve route work, then
setup qp and call rdma_connect to setup connection, before ack or
error replies, the thread will
wait on a wait queue. The listening ib id of remote node will catch
the connect request,
setup qp, allocate and map pages to construct the RDMA-WRITE space,
and call rdma_accept to reply
the request.
Some other information which may be useful:
1.All the "RETRY EXCEEDED" problems happened when there were two
connections which use RDMA-WRITE to transfer things.
And the latter connection had a high possibility to get into this problem.
2. All the "RETRY EXCEEDED" problems happened when the RMDA-WRITE
space is 256MB each(that is, for two connections, consumes 512MB mem),
when the RDMA-WRITE space is 64MB, this problem never happened in our
test. Remote node's total memory is 2GB.
Thanks a lot.
Some more questions:
* Is the WR that "produces" the RETRY EXCEEDED is the first one/last
one/in the middle?
* Which values are you using in the QP context for retry exceeded
counter + retry timeout?
* Did you try to increase those values?
* How many more QPs do you have between those nodes and which operations
do they use
(only RDMA-WRITEs?)
Thanks
Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
