Hi Kyle.
I tested PVFS performance on IB with moving to a PCI-E(x8) platform. I got about 850-950MB network performance. (using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED - I tested the PVFS performance on both IBGD and OFDE) But the performance was also terrible. (50~60MB using pvfs2-cp with changing buffer size) Maybe I got other problems. But now I have no clue why that would happen. Anybody can give me any clues and help me? My test platform Hardware - Intel XeonRProcessor 2.80GHz - dual CPU - Memory 1GB - PCI Express (X8)( using Riser) - SCSI Controller : 2 Channel LSI Logic U 320 Software - OS : 2.6.9-5.EL-smp-i686 - Pvfs2 2.6.1 - IBGD-1.8.2 or OFED-1.0 ( I tested on both ) _____ From: Kyle Schochenmaier [mailto:[EMAIL PROTECTED] Sent: Thursday, October 04, 2007 11:03 AM To: hyeyoung cho Cc: Rob Ross; [email protected] Subject: Re: [Pvfs2-users] pvfs performance over IB Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you were able to get 400MB/s with the test utilities and not on the fs. You should have a definite and measurable performance increase moving to a pci-e platform of course. Could you let us know if this turns out to be the final solution? ~Kyle On 10/3/07, hyeyoung cho < <mailto:[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote: Hi Rob. But I already tested with "--without-bmi-tcp" and the flag gave about 10% improving performance. I found my problem. I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz. Maybe I think the PIC-X performance was my bottle neck. Thank you for your comments. Regards, Hyeyoung Cho -----Original Message----- From: Rob Ross [mailto:[EMAIL PROTECTED] Sent: Thursday, October 04, 2007 12:14 AM To: hyeyoung cho Cc: [email protected] Subject: Re: [Pvfs2-users] pvfs performance over IB you should disable TCP. -- rob hyeyoung cho wrote: > Hello. > > > > I am working on the performance test of PVFS2 over IB(IBGD-1.8.2). > > But the performance was terrible. (70~80MB using pvfs2-cp) > > So just wonder anybody has observed the same behavior earlier? > > And anybody can help me with some insight here? > > > > When I was IB performance using perf_main, which is a benchmark tool in > IBGD, > > The IB performance was about 400MB. > > > > my settings > > - AMD Opteron Processor 240 - dual CPU > > - kernel 2.6.9-55.0.2.ELsmp > > - running pvfs2-2.6.3 > > - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB) > > > > I compiled pvfs2 with following flags: > > ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd > > --with-ib-includes=/usr/local/ibgd/driver/infinihost/include > > --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64 > > --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64 > > > > > > configuration > > 1 server, 1 client > > > > - Client mount command: > > mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2 > > > > - server pvfs2-fs.conf: > > <Aliases> > > Alias c0-10-ib ib://c0-10-ib:3335 > > </Aliases> > > > > - pvfs2-server.conf-c0-10-ib: > > StorageSpace /state/pvfs-part/pvfs2-stoarge-space > > HostID "ib://c0-10-ib:3335" > > LogFile /tmp/pvfs2-server-ib1server.log > > > > Test pvfs2-cp > > [EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5 > > Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY > /mnt/pvfs2/aa > > Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds > > [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY > /mnt/pvfs2/bb > > Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds > > > > Test IB performance(usingperf_main) > > perf_main --send -trc -mbw -s10240 -n1000 > > BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000] > > > > Regards, > > Thank you in advance for valuable comments. > > Hyeyoung cho > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users -- Kyle Schochenmaier
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
