Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you were able to get 400MB/s with the test utilities and not on the fs. You should have a definite and measurable performance increase moving to a pci-e platform of course.
Could you let us know if this turns out to be the final solution? ~Kyle On 10/3/07, hyeyoung cho <[EMAIL PROTECTED]> wrote: > > > Hi Rob. > But I already tested with "--without-bmi-tcp" and the flag gave about 10% > improving performance. > > I found my problem. > I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz. > Maybe I think the PIC-X performance was my bottle neck. > > Thank you for your comments. > > Regards, > Hyeyoung Cho > > -----Original Message----- > From: Rob Ross [mailto:[EMAIL PROTECTED] > Sent: Thursday, October 04, 2007 12:14 AM > To: hyeyoung cho > Cc: [email protected] > Subject: Re: [Pvfs2-users] pvfs performance over IB > > you should disable TCP. -- rob > > hyeyoung cho wrote: > > Hello. > > > > > > > > I am working on the performance test of PVFS2 over IB(IBGD-1.8.2). > > > > But the performance was terrible. (70~80MB using pvfs2-cp) > > > > So just wonder anybody has observed the same behavior earlier? > > > > And anybody can help me with some insight here? > > > > > > > > When I was IB performance using perf_main, which is a benchmark tool in > > IBGD, > > > > The IB performance was about 400MB. > > > > > > > > my settings > > > > - AMD Opteron Processor 240 - dual CPU > > > > - kernel 2.6.9-55.0.2.ELsmp > > > > - running pvfs2-2.6.3 > > > > - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB) > > > > > > > > I compiled pvfs2 with following flags: > > > > ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd > > > > --with-ib-includes=/usr/local/ibgd/driver/infinihost/include > > > > --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64 > > > > --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64 > > > > > > > > > > > > configuration > > > > 1 server, 1 client > > > > > > > > - Client mount command: > > > > mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2 > > > > > > > > - server pvfs2-fs.conf: > > > > <Aliases> > > > > Alias c0-10-ib ib://c0-10-ib:3335 > > > > </Aliases> > > > > > > > > - pvfs2-server.conf-c0-10-ib: > > > > StorageSpace /state/pvfs-part/pvfs2-stoarge-space > > > > HostID "ib://c0-10-ib:3335" > > > > LogFile /tmp/pvfs2-server-ib1server.log > > > > > > > > Test pvfs2-cp > > > > [EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY > /mnt/pvfs2/5 > > > > Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds > > > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY > > /mnt/pvfs2/aa > > > > Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds > > > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY > > /mnt/pvfs2/bb > > > > Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds > > > > > > > > Test IB performance(usingperf_main) > > > > perf_main --send -trc -mbw -s10240 -n1000 > > > > BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000] > > > > > > > > Regards, > > > > Thank you in advance for valuable comments. > > > > Hyeyoung cho > > > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Pvfs2-users mailing list > > [email protected] > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > -- Kyle Schochenmaier
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
