Hi,
I'm looking into the use of Infiniband send/recv Buffers (ibv_reg_mr) in
the C++ RDMA connector code (M4 RC3). As far as I can tell, the buffer
size is based on the ConnectionSettings.maxFrameSize specified by the
client via the following object chain:
ConnectionSettings.maxFrameSize
|
RdmaConnector
|
ConnectionParams.maxRecvBufferSize
|
AsynchIO.bufferSize
|
createBuffer(bufferSize)
I've noticed that default maxFrameSize is 65535 and clients can set it
LOWER without any problem. I tried setting it to 200,000 just to see what
would happen. It appears the code iteratively tries to reduce large
maxFrameSizes to something below 65535:
$ grep createBuffer broker_trace.txt // My own tracing
rdma_wrap.h: QueuePair: createBuffer(3392)
rdma_wrap.h: QueuePair: createBuffer(3392)
.
.
$ grep "createBuffer(3392)" broker_trace.txt | wc -l
128 // 64 each for send & recv
Here is the calculation that appears to happen:
3392 = 200,000 - (65536 * 3)
This appears to be over compensation for too large of maxFrameSize. Could
someone point me to the file in the code base where this adjustment is
happening, I'd like to experiment a bit.
Thanks,
Gregory Marsh
