Re: [Pvfs2-users] pvfs performance over IB

Rob Ross Wed, 10 Oct 2007 05:29:52 -0700

Hi,

Ok, so given those results, it's pretty easy to understand how pvfs2-cpobtained the performance that it did.


Thanks,

Rob

hyeyoung cho wrote:

Hi,
Kyle and Rob.

Thank you for your reply.
I used 2.6.3 at the last test.

When I tested using iozone, the results was read 145MB and write 100MB(MAX).

I will send to you test results file.

Kyle.
There are no logs.
After the server is staring, There are no logs.

[D 10/10 12:41] PVFS2 Server version 2.6.3 starting.

Thanks,
Hyeyoung Cho

-----Original Message-----

From: Rob Ross [mailto:[EMAIL PROTECTED]Sent: Wednesday, October 10, 2007 1:13 PM

To: hyeyoung cho
Cc: 'Kyle Schochenmaier'; [email protected]
Subject: Re: [Pvfs2-users] pvfs performance over IB

Hi,

Have you run iozone or something similar to test the performance of thefile system on which you're storing your PVFS server's data?

You list pvfs version 2.6.1 below, but you listed 2.6.3 before. I assumeyou're using 2.6.3?


Thanks,

Rob

hyeyoung cho wrote:

Hi Kyle.

I tested PVFS performance on IB with moving to a PCI-E(x8) platform.

I got about 850-950MB network performance.

(using perf_main in IBGD or using ib_write_bw, ib_send_bw on OFED -

I tested the PVFS performance on both IBGD and OFDE)

But the performance was also terrible. (50~60MB using pvfs2-cp withchanging buffer size)

Maybe I got other problems.

But now I have no clue why that would happen.

Anybody can give me any clues and help me?

My test platform

Hardware

-         Intel XeonRProcessor 2.80GHz - dual CPU

-         Memory 1GB

-         PCI Express (X8)( using Riser)

-         SCSI Controller : 2 Channel LSI Logic U 320



Software

-         OS : 2.6.9-5.EL-smp-i686

-         Pvfs2 2.6.1

-         IBGD-1.8.2 or OFED-1.0 ( I tested on both )

------------------------------------------------------------------------

*From:* Kyle Schochenmaier [mailto:[EMAIL PROTECTED]
*Sent:* Thursday, October 04, 2007 11:03 AM
*To:* hyeyoung cho
*Cc:* Rob Ross; [email protected]
*Subject:* Re: [Pvfs2-users] pvfs performance over IB

Yes, PCI-X should be a huge bottleneck here, but doesnt explain how youwere able to get 400MB/s with the test utilities and not on the fs.You should have a definite and measurable performance increase moving toa pci-e platform of course.


Could you let us know if this turns out to be the final solution?

~Kyle

On 10/3/07, *hyeyoung cho* < [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:



Hi Rob.
But I already tested with "--without-bmi-tcp" and the flag gave about 10%
improving performance.

I found my problem.
I doubt PCI-X performance. My testbed is old system using PCI-X 100MHz.
Maybe I think the PIC-X performance was my bottle neck.

Thank you for your comments.

Regards,
Hyeyoung Cho

-----Original Message-----
From: Rob Ross [mailto:[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>]
Sent: Thursday, October 04, 2007 12:14 AM
To: hyeyoung cho

Cc: [email protected]<mailto:[email protected]>

Subject: Re: [Pvfs2-users] pvfs performance over IB

you should disable TCP. -- rob

hyeyoung cho wrote:

 Hello.



 I am working on the performance test of PVFS2 over IB(IBGD-1.8.2).

 But the performance was terrible.  (70~80MB using pvfs2-cp)

 So just wonder anybody has observed the same behavior earlier?

 And anybody can help me with some insight here?



 When I was IB performance using perf_main, which is a benchmark tool in
 IBGD,

 The IB performance was about 400MB.



 my settings

 - AMD Opteron Processor 240 - dual CPU

 - kernel 2.6.9-55.0.2.ELsmp

 - running pvfs2-2.6.3

 - IBGD-1.8.2 (Topspin HCA - 2pt, 10GB, PCI-X, 128MB)



 I compiled pvfs2 with following flags:

 ./configure --with-kernel=/usr/src/linux-2.6 --with-ib=/usr/local/ibgd

 --with-ib-includes=/usr/local/ibgd/driver/infinihost/include

 --with-ib-libs=/usr/local/ibgd/driver/infinihost/lib64

 --with-gm=/opt/gm --with-gm-libs=/opt/gm/lib64





 configuration

 1 server, 1 client



 - Client mount command:

  mount -t pvfs2 ib://c0-10-ib:3335/pvfs2-fs /mnt/pvfs2



 - server pvfs2-fs.conf:

 <Aliases>

        Alias c0-10-ib ib://c0-10-ib:3335

 </Aliases>



 - pvfs2-server.conf-c0-10-ib:

 StorageSpace /state/pvfs-part/pvfs2-stoarge-space

 HostID "ib://c0-10-ib:3335"

 LogFile /tmp/pvfs2-server-ib1server.log



 Test pvfs2-cp

  [EMAIL PROTECTED] ~]# pvfs2-cp -t /state/partition1/testHY /mnt/pvfs2/5

  Wrote 429588480 bytes in 4.854962 seconds. 84.385318 MB/seconds

  [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 8388608 /state/partition1/testHY
 /mnt/pvfs2/aa

  Wrote 429588480 bytes in 5.449869 seconds. 75.173826 MB/seconds

 [EMAIL PROTECTED] ~]# pvfs2-cp -t -b 4194304 /state/partition1/testHY
 /mnt/pvfs2/bb

 Wrote 429588480 bytes in 5.600240 seconds. 73.155347 MB/seconds



 Test IB performance(usingperf_main)

  perf_main --send -trc -mbw -s10240 -n1000

 BW: 391.8 MBytes/sec [size: 10240 bytes, iter: 1000, total 10240000]



 Regards,

 Thank you in advance for valuable comments.

 Hyeyoung cho




 ------------------------------------------------------------------------

 _______________________________________________
 Pvfs2-users mailing list

[email protected]

<mailto:[email protected]>

 http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

_______________________________________________
Pvfs2-users mailing list

[email protected]<mailto:[email protected]>

http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users




--
Kyle Schochenmaier

_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

Re: [Pvfs2-users] pvfs performance over IB

Reply via email to