RE: [Pvfs2-users] PVFS2 over Infiniband

조혜영 Sun, 10 Sep 2006 23:40:05 -0700

Thanks for your replay.

I measured performance using bonnie and pvfs2-cp.

When I run bonnie at mount directory by pvfs2(/mnt/pvfs2), the result was

3~8MB/seconds(write/block). When pvfs-cp(from local to /mnt/pvfs2) was run, the result was about 17~20MB/seconds (write).

What is the problem? I can not find the reason...

This is my configuration information.

I used OFDE-1.0 and Linux 2.6.9-5.ELsmp.

I used 4 servers(intel Xeon 2.8GHz) and Flextronics 24 4X port switch and Topspin HCA Card(PCI-X).

My configuration is like this.

============================================================================

[EMAIL PROTECTED] pvfs2]# configure --with-kernel=/usr/src/kernels/2.6.9-5.EL-smp-

i686 --with-openib=/usr/local/ofed --with-openib- includes=/usr/local/ofed/include --with-openib-libs=/usr/local/ofed/lib

**********************************************************************

Welcome to the PVFS2 Configuration Generator:

This interactive script will generate configuration files suitable for use with a new PVFS2 file system. Please see the PVFS2 quickstart guide for details.

**********************************************************************

You must first select the network protocol that your file system will use.

The only currently supported options are "tcp", "gm", and "ib".

(For multi-homed configurations, use e.g. "ib,tcp".)

* Enter protocol type [Default is tcp]: ib

Choose a TCP/IP port for the servers to listen on for IB communications.

Note that this

script assumes that all servers will use the same port number.

* Enter port number [Default is 3335]:

Choose a directory for each server to store data in.

* Enter directory name: [Default is /pvfs2-storage-space]: /home2/pvfs2- storage-space

Choose a file for each server to write log messages to.

* Enter log file location [Default is /tmp/pvfs2-server.log]: /tmp/pvfs2- server.log

Next you must list the hostnames of the machines that will act as I/O servers. Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}".

* Enter hostnames [Default is localhost]: 11.0.0.1, 11.0.0.2, 11.0.0.3,

11.0.0.4

Now list the hostnames of the machines that will act as Metadata servers. This list may or may not overlap with the I/O server list.

* Enter hostnames [Default is localhost]: 11.0.0.4

Configured a total of 4 servers:

4 of them are I/O servers.

1 of them are Metadata servers.

* Would you like to verify server list (y/n) [Default is n]? y

****** I/O servers:

****** Metadata servers:

11.0.0.4

* Does this look ok (y/n) [Default is y]? y

Writing fs config file... Done.

Writing 4 server config file(s)... Done.

11.0.0.3

11.0.0.2

11.0.0.1

11.0.0.4

=================================================================

[EMAIL PROTECTED] ~]# cat /etc/pvfs2-fs.conf

UnexpectedRequests 50

LogFile /tmp/pvfs2-server.log

EventLogging none

LogStamp datetime

BMIModules bmi_ib

FlowModules flowproto_multiqueue

PerfUpdateInterval 1000

ServerJobBMITimeoutSecs 30

ServerJobFlowTimeoutSecs 30

ClientJobBMITimeoutSecs 300

ClientJobFlowTimeoutSecs 300

ClientRetryLimit 5

ClientRetryDelayMilliSecs 2000

</Defaults>

Alias 11.0.0.1 ib://11.0.0.1:3335

Alias 11.0.0.2 ib://11.0.0.2:3335

Alias 11.0.0.3 ib://11.0.0.3:3335

Alias 11.0.0.4 ib://11.0.0.4:3335 </Aliases>

Name pvfs2-fs

ID 1069359617

RootHandle 1048576

Range 11.0.0.1 4-1073741826

</MetaHandleRanges>

Range 11.0.0.1 1073741827-2147483649

Range 11.0.0.2 2147483650-3221225472

Range 11.0.0.3 3221225473-4294967295

Range 11.0.0.4 4294967296-5368709118

</DataHandleRanges>

TroveSyncMeta yes

TroveSyncData no

</StorageHints>

</Filesystem>

================================================================= [EMAIL PROTECTED] ~]# cat /etc/pvfs2-server.conf-11.0.0.1 StorageSpace /pvfs2-storage-space HostID "ib://11.0.0.1:3335"

LogFile /tmp/pvfs2-server.log

[EMAIL PROTECTED] ~]# cat /etc/pvfs2-server.conf-11.0.0.2 StorageSpace /pvfs2-storage-space HostID "ib://11.0.0.2:3335"

LogFile /tmp/pvfs2-server.log

[EMAIL PROTECTED] ~]# cat /etc/pvfs2-server.conf-11.0.0.3 StorageSpace /pvfs2-storage-space HostID "ib://11.0.0.3:3335"

LogFile /tmp/pvfs2-server.log

[EMAIL PROTECTED] ~]# cat /etc/pvfs2-server.conf-11.0.0.4 StorageSpace /pvfs2-storage-space HostID "ib://11.0.0.4:3335"

LogFile /tmp/pvfs2-server.log

=================================================================

[EMAIL PROTECTED] pvfs2-ib-config]# cat pvfs2tab

ib://11.0.0.4:3335/pvfs2-fs /mnt/pvfs2 pvfs2 default,noauto 0 0 =================================================================

[EMAIL PROTECTED] linux]# cat mount_pvfs2.sh

#!/bin/bash

if [ "`echo $PATH | grep -c /usr/src/modtools/sbin`" = "0" ]; then

echo "Loading up bashrc to get latest mod-tools"

. ~/.bashrc

if [ "`lsmod | grep -c pvfs2`" = "0" ]; then

if test "x`uname -r | cut -b 1-3`" = "x2.4"; then

echo "Loading the 2.4.x pvfs2 module"

insmod ../../../../src/kernel/linux-2.4/pvfs2.o $1

else

echo "Loading the 2.6.x pvfs2 module"

insmod ../../../../src/kernel/linux-2.6/pvfs2.ko $1

CUR_DEV="pvfs2-flow"

for f in `cat /proc/devices | grep pvfs2 | awk '{print $1}'`; do

if [ ! -c /dev/$CUR_DEV ]; then

echo "Creating device file /dev/$CUR_DEV with major number $f"

mknod /dev/$CUR_DEV c $f 0 -m 666

CUR_DEV="pvfs2-req"

done

if [ ! -d /mnt/pvfs2 ]; then

echo "Creating /mnt/pvfs2"

mkdir /mnt/pvfs2

echo "Starting pvfs2-client"

./pvfs2-client -p ./pvfs2-client-core

if [ "`mount | grep -c pvfs2`" = "0" ]; then

if test "x`uname -r | cut -b 1-3`" = "x2.4"; then

echo "Mounting 2.4.x pvfs2 on /mnt/pvfs2"

mount -t pvfs2 pvfs2 /mnt/pvfs2 -o ib://11.0.0.4:3335/pvfs2-fs

else

echo "Mounting 2.6.x pvfs2 on /mnt/pvfs2"

mount -t pvfs2 ib://11.0.0.4:3335/pvfs2-fs /mnt/pvfs2

else

echo "Uh...I think pvfs2 is already mounted."

============================================================================

[EMAIL PROTECTED] bonnie]# mount | grep pvfs

ib://11.0.0.4:3335/pvfs2-fs on /mnt/pvfs2 type pvfs2 (rw)

============================================================================

This is the performance result.

============================================================================

[EMAIL PROTECTED] etc]# pvfs2-cp -t /root/Down/IBGD-1.8.2.tgz /mnt/pvfs2/test-a

Wrote 57914150 bytes in 2.447529 seconds. 22.566122 MB/seconds

============================================================================

[EMAIL PROTECTED] bonnie]# pwd

/mnt/pvfs2/bonnie

[EMAIL PROTECTED] bonnie]# Bonnie

File './Bonnie.14537', size: 104857600

Writing with putc()...done

Rewriting...done

Writing intelligently...done

Reading with getc()...done

Reading intelligently...done

Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...

-------Sequential Output-------- ---Sequential Input-- --Random--

-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---

Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU

100 20574 86.1 6615 1.2 1759 0.6 8396 37.4 3240 1.4 265.7 1.7

============================================================================

What is the problem?...

Regards,

Hyeyoung cho

_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users

RE: [Pvfs2-users] PVFS2 over Infiniband

Reply via email to