Zane, Great work.
I have run through the patches, and they do work with a few changes, although there is a serious problem with the way ibv_reg_mr is handled. ibv_reg_mr is a super expensive operation. As an example, here is iba using your changes: [root@mrg-03 test]# ./cpgbench 151413 messages received 1000 bytes per write 10.000 Seconds runtime 15140.575 TP/s 15.141 MB/s. 81012 messages received 2000 bytes per write 10.000 Seconds runtime 8101.139 TP/s 16.202 MB/s. 54438 messages received 3000 bytes per write 10.000 Seconds runtime 5443.771 TP/s 16.331 MB/s. 41695 messages received 4000 bytes per write 10.000 Seconds runtime 4169.472 TP/s 16.678 MB/s. 35227 messages received 5000 bytes per write 10.001 Seconds runtime 3522.369 TP/s 17.612 MB/s. 29290 messages received 6000 bytes per write 10.000 Seconds runtime 2928.972 TP/s 17.574 MB/s. 25038 messages received 7000 bytes per write 10.001 Seconds runtime 2503.661 TP/s 17.526 MB/s. 22071 messages received 8000 bytes per write 10.001 Seconds runtime 2206.945 TP/s 17.656 MB/s. and the unpatched is: [root@mrg-03 test]# ./cpgbench 519008 messages received 1000 bytes per write 10.000 Seconds runtime 51900.39 5 TP/s 51.900 MB/s. 491084 messages received 2000 bytes per write 10.000 Seconds runtime 49108.26 7 TP/s 98.217 MB/s. 473848 messages received 3000 bytes per write 10.000 Seconds runtime 47384.59 6 TP/s 142.154 MB/s. 461259 messages received 4000 bytes per write 10.000 Seconds runtime 46125.81 2 TP/s 184.503 MB/s. 425705 messages received 5000 bytes per write 10.000 Seconds runtime 42570.34 7 TP/s 212.852 MB/s. 405176 messages received 6000 bytes per write 10.000 Seconds runtime 40517.50 7 TP/s 243.105 MB/s. 354461 messages received 7000 bytes per write 10.000 Seconds runtime 35446.00 4 TP/s 248.122 MB/s. 312289 messages received 8000 bytes per write 10.000 Seconds runtime 31228.66 3 TP/s 249.829 MB/s. 277114 messages received 9000 bytes per write 10.000 Seconds runtime 27711.02 6 TP/s 249.399 MB/s. 254190 messages received 10000 bytes per write 10.000 Seconds runtime 25418.93 9 TP/s 254.189 MB/s. I am not sure if I mentioned this previously, but it is mandatory to preallocate the buffers including their memory registrations. This is why there are separate send and receive lists in the original code. The first time I wrote iba I had a similar implementation as you have done (re-register and free the memory regions each time) and performance was about on par with your implementation. I'll follow up with some specific bugs in the patches in following emails. On 04/16/2011 02:10 PM, Zane Bitter wrote: > The following patches put in place the infrastructure to receive a packet > directly into the buffer that will be enqueued for later transmission. This > is implemented for the UDP and Inifiniband drivers. > > The Inifiniband implementation is completely untested due to lack of > hardware, so if someone with access to the appropriate hardware could give > it a spin and send me the inevitable stack traces, that would be much > appreciated ;). For everyone else: here be dragons. > > The implementation for the UDPU driver should be not unidentical to the UDP > one, so I would prefer to get comments on this before copying the changes > over. The ultimate solution is to split the auth stuff out into a separate > file (I notice there is a TODO entry for this already). > > cheers, > Zane. > > --- > > Zane Bitter (4): > Add a new totemsrp_buffer_retain() function > Avoid malloc()s by keeping a list of free packet buffers > Avoid a copy on packet reception in the UDP driver > Avoid copying buffers where possible in the Infiniband driver > > > exec/Makefile.am | 2 > exec/totembuf.c | 212 +++++++++++++++++++++++++++++ > exec/totembuf.h | 46 ++++++ > exec/totemiba.c | 389 > ++++++++++++++++++++++++++++++------------------------ > exec/totemiba.h | 2 > exec/totemnet.c | 13 ++ > exec/totemnet.h | 2 > exec/totemrrp.c | 7 + > exec/totemrrp.h | 4 + > exec/totemsrp.c | 11 +- > exec/totemudp.c | 155 ++++++++++++---------- > exec/totemudp.h | 2 > exec/totemudpu.c | 9 + > exec/totemudpu.h | 2 > 14 files changed, 608 insertions(+), 248 deletions(-) > create mode 100644 exec/totembuf.c > create mode 100644 exec/totembuf.h > > _______________________________________________ > Openais mailing list > [email protected] > https://lists.linux-foundation.org/mailman/listinfo/openais _______________________________________________ Openais mailing list [email protected] https://lists.linux-foundation.org/mailman/listinfo/openais
