Hi, Kyle and Rob. Thank you for your reply. I used 2.6.3 at the last test.
When I tested using iozone, the results was read 145MB and write 100MB(MAX). I will send to you test results file. Kyle. There are no logs. After the server is staring, There are no logs. [D 10/10 12:41] PVFS2 Server version 2.6.3 starting. Thanks, Hyeyoung Cho -----Original Message----- From: Rob Ross [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 10, 2007 1:13 PM To: hyeyoung cho Cc: 'Kyle Schochenmaier'; [email protected] Subject: Re: [Pvfs2-users] pvfs performance over IB Hi, Have you run iozone or something similar to test the performance of the file system on which you're storing your PVFS server's data? You list pvfs version 2.6.1 below, but you listed 2.6.3 before. I assume you're using 2.6.3? Thanks, Rob hyeyoung cho wrote: > > > Hi Kyle. > > > > I tested PVFS performance on IB with moving to a PCI-E(x8) platform. > > I got about 850-950MB network performance. > > (using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED - > > I tested the PVFS performance on both IBGD and OFDE) > > > > But the performance was also terrible. (50~60MB using pvfs2-cp with > changing buffer size) > > > > Maybe I got other problems. > > But now I have no clue why that would happen. > > Anybody can give me any clues and help me? > > > > > > My test platform > > Hardware > > - Intel XeonRProcessor 2.80GHz - dual CPU > > - Memory 1GB > > - PCI Express (X8)( using Riser) > > - SCSI Controller : 2 Channel LSI Logic U 320 > > > > Software > > - OS : 2.6.9-5.EL-smp-i686 > > - Pvfs2 2.6.1 > > - IBGD-1.8.2 or OFED-1.0 ( I tested on both ) > > > > > > > > ------------------------------------------------------------------------ > > *From:* Kyle Schochenmaier [mailto:[EMAIL PROTECTED] > *Sent:* Thursday, October 04, 2007 11:03 AM > *To:* hyeyoung cho > *Cc:* Rob Ross; [email protected] > *Subject:* Re: [Pvfs2-users] pvfs performance over IB > > > > Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you > were able to get 400MB/s with the test utilities and not on the fs. > You should have a definite and measurable performance increase moving to > a pci-e platform of course. > > Could you let us know if this turns out to be the final solution? > > ~Kyle > > On 10/3/07, *hyeyoung cho* < [EMAIL PROTECTED] > <mailto:[EMAIL PROTECTED]>> wrote: > > > Hi Rob. > But I already tested with "--without-bmi-tcp" and the flag gave about 10% > improving performance. > > I found my problem. > I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz. > Maybe I think the PIC-X performance was my bottle neck. > > Thank you for your comments. > > Regards, > Hyeyoung Cho > > -----Original Message----- > From: Rob Ross [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>] > Sent: Thursday, October 04, 2007 12:14 AM > To: hyeyoung cho > Cc: [email protected] > <mailto:[email protected]> > Subject: Re: [Pvfs2-users] pvfs performance over IB > > you should disable TCP. -- rob > > hyeyoung cho wrote: >> Hello. >> >> >> >> I am working on the performance test of PVFS2 over IB(IBGD-1.8.2). >> >> But the performance was terrible. (70~80MB using pvfs2-cp) >> >> So just wonder anybody has observed the same behavior earlier? >> >> And anybody can help me with some insight here? >> >> >> >> When I was IB performance using perf_main, which is a benchmark tool in >> IBGD, >> >> The IB performance was about 400MB. >> >> >> >> my settings >> >> - AMD Opteron Processor 240 - dual CPU >> >> - kernel 2.6.9-55.0.2.ELsmp >> >> - running pvfs2-2.6.3 >> >> - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB) >> >> >> >> I compiled pvfs2 with following flags: >> >> ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd >> >> --with-ib-includes=/usr/local/ibgd/driver/infinihost/include >> >> --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64 >> >> --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64 >> >> >> >> >> >> configuration >> >> 1 server, 1 client >> >> >> >> - Client mount command: >> >> mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2 >> >> >> >> - server pvfs2-fs.conf: >> >> <Aliases> >> >> Alias c0-10-ib ib://c0-10-ib:3335 >> >> </Aliases> >> >> >> >> - pvfs2-server.conf-c0-10-ib: >> >> StorageSpace /state/pvfs-part/pvfs2-stoarge-space >> >> HostID "ib://c0-10-ib:3335" >> >> LogFile /tmp/pvfs2-server-ib1server.log >> >> >> >> Test pvfs2-cp >> >> [EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5 >> >> Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds >> >> [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY >> /mnt/pvfs2/aa >> >> Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds >> >> [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY >> /mnt/pvfs2/bb >> >> Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds >> >> >> >> Test IB performance(usingperf_main) >> >> perf_main --send -trc -mbw -s10240 -n1000 >> >> BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000] >> >> >> >> Regards, >> >> Thank you in advance for valuable comments. >> >> Hyeyoung cho >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Pvfs2-users mailing list >> [email protected] > <mailto:[email protected]> >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > <mailto:[email protected]> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > -- > Kyle Schochenmaier > _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
