At 03:36 AM 3/11/2008, yangdong wrote: > >Hello! I implemented a message passing subsystem -- AMP for our cluster >file system--DCFS3, >I want AMP to run over Infiniband, I designed it in kernel(verbs). >when client writing file, my protocol is implemented by RDMA WRITE, and >reading file by RDMA >READ, but perf of AMP is less than my expectation. Single thread (1t) >read is 215MB/s, 4t is 340MB/s. >And I see that the read file perf of NFS/RDMA is close to 600MB/s (1t).
NFS/RDMA is not single threaded. The client has several ways to keep multiple I/Os active, among them use of a kernel rpc state machine, enlisting application threads performing synch i/o, use of kernel helpers (rpciod's), the filesystem buffer cache, and the RPC slot table. Also, its protocol supports credits and RDMA scatter/gather, and is tuned to optimize server transfers. BTW, NFS/RDMA achieves much more than 600MB/s. On DDR IB, for example, Sandia's testing achieved over double that. Just some ideas, since you give so little information about your implementation it's hard to conclude anything. Tom. >I made use of FMR and All physical reg ..and scatter/gather, but why the >perf of AMP is lower? > > > >-- >Dong Yang <[EMAIL PROTECTED]> >Institute of Computing Technology, Chinese Academy of Sciences >Address: National Research Center for Intelligent Computing Systems >(NCIC), P.O. Box 2704, Beijing 100080, P.R. China >Phone: +86-10-62601005 > >_______________________________________________ >general mailing list >[email protected] >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
