On Fri, 2008-02-29 at 09:29 +0100, Sebastian Schmitzdorff wrote: > hi pawel, > > I was wondering if you have achieved better nfs rdma benchmark results > by now?
Pawel: What is your network hardware setup? Thanks, Tom > > regards > Sebastian > > Pawel Dziekonski schrieb: > > hi, > > > > the saga continues. ;) > > > > very basic benchmarks and surprising (at least for me) results - it > > look's like reading is much slower than writing and NFS/RDMA is twice > > slower in reading than classic NFS. :o > > > > results below - comments appreciated! > > regards, Pawel > > > > > > both nfs server and client have 8-cores, 16 GB RAM, Mellanox DDR HCAs > > (MT25204) connected port-port (no switch). > > > > local_hdd - 2 sata2 disks in soft-raid0, > > nfs_ipoeth - classic nfs over ethernet, > > nfs_ipoib - classic nfs over IPoIB, > > nfs_rdma - NFS/RDMA. > > > > simple write of 36GB file with dd (both machines have 16GB RAM): > > /usr/bin/time -p dd if=/dev/zero of=/mnt/qqq bs=1M count=36000 > > > > local_hdd sys 54.52 user 0.04 real 254.59 > > > > nfs_ipoib sys 36.35 user 0.00 real 266.63 > > nfs_rdma sys 39.03 user 0.02 real 323.77 > > nfs_ipoeth sys 34.21 user 0.01 real 375.24 > > > > remount /mnt to clear cache and read a file from nfs share and > > write it to /dev/: > > /usr/bin/time -p dd if=/mnt/qqq of=/scratch/qqq bs=1M > > > > nfs_ipoib sys 59.04 user 0.02 real 571.57 > > nfs_ipoeth sys 58.92 user 0.02 real 606.61 > > nfs_rdma sys 62.57 user 0.03 real 1296.36 > > > > > > > > results from bonnie++: > > > > Version 1.03c ------Sequential Write ------ --Sequential Read -- > > --Random- > > -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > > --Seeks-- > > Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP > > /sec %CP > > local_hdd 35G:128k 93353 12 58329 6 143293 7 > > 243.6 1 > > local_hdd 35G:256k 92283 11 58189 6 144202 8 > > 172.2 2 > > local_hdd 35G:512k 93879 12 57715 6 144167 8 > > 128.2 4 > > local_hdd 35G:1024k 93075 12 58637 6 144172 8 > > 95.3 7 > > nfs_ipoeth 35G:128k 91325 7 31848 4 64299 4 > > 170.2 1 > > nfs_ipoeth 35G:256k 90668 7 32036 5 64542 4 > > 163.2 2 > > nfs_ipoeth 35G:512k 93348 7 31757 5 64454 4 > > 85.7 3 > > nfs_ipoet 35G:1024k 91283 7 31869 5 64241 5 > > 51.7 4 > > nfs_ipoib 35G:128k 91733 7 36641 5 65839 4 > > 178.4 2 > > nfs_ipoib 35G:256k 92453 7 36567 6 66682 4 > > 166.9 3 > > nfs_ipoib 35G:512k 91157 7 37660 6 66318 4 > > 86.8 3 > > nfs_ipoib 35G:1024k 92111 7 35786 6 66277 5 > > 53.3 4 > > nfs_rdma 35G:128k 91152 8 29942 5 32147 2 > > 187.0 1 > > nfs_rdma 35G:256k 89772 7 30560 5 34587 2 > > 158.4 3 > > nfs_rdma 35G:512k 91290 7 29698 5 34277 2 > > 60.9 2 > > nfs_rdma 35G:1024k 91336 8 29052 5 31742 2 > > 41.5 3 > > ------Sequential Create------ --------Random > > Create-------- > > -Create-- --Read--- -Delete-- -Create-- --Read--- > > -Delete-- > > files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec > > %CP > > local_hdd 16 10587 36 +++++ +++ 8674 29 10727 35 +++++ +++ 7015 > > 28 > > local_hdd 16 11372 41 +++++ +++ 8490 29 11192 43 +++++ +++ 6881 > > 27 > > local_hdd 16 10789 35 +++++ +++ 8520 29 11468 46 +++++ +++ 6651 > > 24 > > local_hdd 16 10841 40 +++++ +++ 8443 28 11162 41 +++++ +++ 6441 > > 22 > > nfs_ipoeth 16 3753 7 13390 12 3795 7 3773 8 22181 16 3635 > > 7 > > nfs_ipoeth 16 3762 8 12358 7 3713 8 3753 7 20448 13 3632 > > 6 > > nfs_ipoeth 16 3834 7 12697 6 3729 8 3725 9 22807 11 3673 > > 7 > > nfs_ipoeth 16 3729 8 14260 10 3774 7 3744 7 25285 14 3688 > > 7 > > nfs_ipoib 16 6803 17 +++++ +++ 6843 15 6820 14 +++++ +++ 5834 > > 11 > > nfs_ipoib 16 6587 16 +++++ +++ 4959 9 6832 14 +++++ +++ 5608 > > 12 > > nfs_ipoib 16 6820 18 +++++ +++ 6636 15 6479 15 +++++ +++ 5679 > > 13 > > nfs_ipoib 16 6475 14 +++++ +++ 6435 14 5543 11 +++++ +++ 5431 > > 11 > > nfs_rdma 16 7014 15 +++++ +++ 6714 10 7001 14 +++++ +++ 5683 > > 8 > > nfs_rdma 16 7038 13 +++++ +++ 6713 12 6956 11 +++++ +++ 5488 > > 8 > > nfs_rdma 16 7058 12 +++++ +++ 6797 11 6989 14 +++++ +++ 5761 > > 9 > > nfs_rdma 16 7201 13 +++++ +++ 6821 12 7072 15 +++++ +++ 5609 > > 9 > > > > > > > > _______________________________________________ general mailing list general@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general