Hi Rob. But I already tested with "--without-bmi-tcp" and the flag gave about 10% improving performance.
I found my problem. I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz. Maybe I think the PIC-X performance was my bottle neck. Thank you for your comments. Regards, Hyeyoung Cho -----Original Message----- From: Rob Ross [mailto:[EMAIL PROTECTED] Sent: Thursday, October 04, 2007 12:14 AM To: hyeyoung cho Cc: [email protected] Subject: Re: [Pvfs2-users] pvfs performance over IB you should disable TCP. -- rob hyeyoung cho wrote: > Hello. > > > > I am working on the performance test of PVFS2 over IB(IBGD-1.8.2). > > But the performance was terrible. (70~80MB using pvfs2-cp) > > So just wonder anybody has observed the same behavior earlier? > > And anybody can help me with some insight here? > > > > When I was IB performance using perf_main, which is a benchmark tool in > IBGD, > > The IB performance was about 400MB. > > > > my settings > > - AMD Opteron Processor 240 - dual CPU > > - kernel 2.6.9-55.0.2.ELsmp > > - running pvfs2-2.6.3 > > - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB) > > > > I compiled pvfs2 with following flags: > > ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd > > --with-ib-includes=/usr/local/ibgd/driver/infinihost/include > > --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64 > > --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64 > > > > > > configuration > > 1 server, 1 client > > > > - Client mount command: > > mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2 > > > > - server pvfs2-fs.conf: > > <Aliases> > > Alias c0-10-ib ib://c0-10-ib:3335 > > </Aliases> > > > > - pvfs2-server.conf-c0-10-ib: > > StorageSpace /state/pvfs-part/pvfs2-stoarge-space > > HostID "ib://c0-10-ib:3335" > > LogFile /tmp/pvfs2-server-ib1server.log > > > > Test pvfs2-cp > > [EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5 > > Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY > /mnt/pvfs2/aa > > Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY > /mnt/pvfs2/bb > > Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds > > > > Test IB performance(usingperf_main) > > perf_main --send -trc -mbw -s10240 -n1000 > > BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000] > > > > Regards, > > Thank you in advance for valuable comments. > > Hyeyoung cho > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
