Hi,
Have you run iozone or something similar to test the performance of the
file system on which you're storing your PVFS server's data?
You list pvfs version 2.6.1 below, but you listed 2.6.3 before. I assume
you're using 2.6.3?
Thanks,
Rob
hyeyoung cho wrote:
Hi Kyle.
I tested PVFS performance on IB with moving to a PCI-E(x8) platform.
I got about 850-950MB network performance.
(using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED -
I tested the PVFS performance on both IBGD and OFDE)
But the performance was also terrible. (50~60MB using pvfs2-cp with
changing buffer size)
Maybe I got other problems.
But now I have no clue why that would happen.
Anybody can give me any clues and help me?
My test platform
Hardware
- Intel Xeon®Processor 2.80GHz - dual CPU
- Memory 1GB
- PCI Express (X8)( using Riser)
- SCSI Controller : 2 Channel LSI Logic U 320
Software
- OS : 2.6.9-5.EL-smp-i686
- Pvfs2 2.6.1
- IBGD-1.8.2 or OFED-1.0 ( I tested on both )
------------------------------------------------------------------------
*From:* Kyle Schochenmaier [mailto:[EMAIL PROTECTED]
*Sent:* Thursday, October 04, 2007 11:03 AM
*To:* hyeyoung cho
*Cc:* Rob Ross; [email protected]
*Subject:* Re: [Pvfs2-users] pvfs performance over IB
Yes, PCI-X should be a huge bottleneck here, but doesnt explain how you
were able to get 400MB/s with the test utilities and not on the fs.
You should have a definite and measurable performance increase moving to
a pci-e platform of course.
Could you let us know if this turns out to be the final solution?
~Kyle
On 10/3/07, *hyeyoung cho* < [EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Hi Rob.
But I already tested with "--without-bmi-tcp" and the flag gave about 10%
improving performance.
I found my problem.
I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.
Maybe I think the PIC-X performance was my bottle neck.
Thank you for your comments.
Regards,
Hyeyoung Cho
-----Original Message-----
From: Rob Ross [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>]
Sent: Thursday, October 04, 2007 12:14 AM
To: hyeyoung cho
Cc: [email protected]
<mailto:[email protected]>
Subject: Re: [Pvfs2-users] pvfs performance over IB
you should disable TCP. -- rob
hyeyoung cho wrote:
Hello.
I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).
But the performance was terrible. (70~80MB using pvfs2-cp)
So just wonder anybody has observed the same behavior earlier?
And anybody can help me with some insight here?
When I was IB performance using perf_main, which is a benchmark tool in
IBGD,
The IB performance was about 400MB.
my settings
- AMD Opteron Processor 240 - dual CPU
- kernel 2.6.9-55.0.2.ELsmp
- running pvfs2-2.6.3
- IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)
I compiled pvfs2 with following flags:
./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd
--with-ib-includes=/usr/local/ibgd/driver/infinihost/include
--with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64
--with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64
configuration
1 server, 1 client
- Client mount command:
mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2
- server pvfs2-fs.conf:
<Aliases>
Alias c0-10-ib ib://c0-10-ib:3335
</Aliases>
- pvfs2-server.conf-c0-10-ib:
StorageSpace /state/pvfs-part/pvfs2-stoarge-space
HostID "ib://c0-10-ib:3335"
LogFile /tmp/pvfs2-server-ib1server.log
Test pvfs2-cp
[EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5
Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds
[EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY
/mnt/pvfs2/aa
Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds
[EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY
/mnt/pvfs2/bb
Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds
Test IB performance(usingperf_main)
perf_main --send -trc -mbw -s10240 -n1000
BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]
Regards,
Thank you in advance for valuable comments.
Hyeyoung cho
------------------------------------------------------------------------
_______________________________________________
Pvfs2-users mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
Pvfs2-users mailing list
[email protected]
<mailto:[email protected]>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
--
Kyle Schochenmaier
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
