Thanks for the replies so far.
Here's answers to all the questions so far:
I do have dell openmanage installed, but no obvious things to tweak show up.
cpu frequency scaling is not enabled
I have tested underlying disk performance, but I don't think I've run
your test. There are some heavy pvfs2-io functions taking place right
now (for the next many hours, I suspect...someone trying to
recursively delete a directory with hundred-thousand+ files). Still,
I'm seeing less than 1% utilization.
I tested using dd as follows:
dd if=/dev/zero of=/mnt/pvfs2/testfile bs=1024k count=2048
Local on the pvfs server (direct to disk, not going through
pvfs2-server), I got 262MB/s. Going through pvfs2, I got 76.1MB/s
My pvfs2-fs.conf:
<Defaults>
UnexpectedRequests 50
EventLogging none
LogStamp datetime
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
StorageSpace /mnt/pvfs2
LogFile /var/log/pvfs2-server.log
</Defaults>
<Aliases>
Alias pvfs2-io-0-0 tcp://pvfs2-io-0-0:3334
Alias pvfs2-io-0-1 tcp://pvfs2-io-0-1:3334
Alias pvfs2-io-0-2 tcp://pvfs2-io-0-2:3334
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 62659950
RootHandle 1048576
<MetaHandleRanges>
Range pvfs2-io-0-0 4-715827885
Range pvfs2-io-0-1 715827886-1431655767
Range pvfs2-io-0-2 1431655768-2147483649
</MetaHandleRanges>
<DataHandleRanges>
Range pvfs2-io-0-0 2147483650-2863311531
Range pvfs2-io-0-1 2863311532-3579139413
Range pvfs2-io-0-2 3579139414-4294967295
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
</StorageHints>
</Filesystem>
--------------------------------------
Also, I forgot to mention, I'm running my physical disk through lvm; I
have 2 hardware raid6 array's lvm'ed together to /mnt/pvfs2. All 3
servers are running identical configuration; each has 15 750GB disks
in raid6 and 10 2TB disks in seperate raid6; each pvfs2 io server has
18TB of disk space online.
--Jim
On Fri, Apr 29, 2011 at 12:34 PM, Kevin Harms <[email protected]> wrote:
> Jim,
>
> sounds like you have 2 main issues, metadata performance and data
> performance? When you compare performance to "local disk". Are you comparing
> performance of your disk/raid using like 'dd' to the performance you get
> against sending data over the network to your pvfs2-server to disk? You might
> want to step back and look at each component. Run a single server and client
> on your storage machine and see how well the pvfs2-server utilizes the
> storage. Then perhaps evaluate network performance between two clients and
> then compare that to using a single pvfs2-server with a single remote client
> and see what things look like.
>
> i think metadata is trickier. I'm not sure a serial chmod of lots of files
> will ever be fast. Can you use more than one host to do it?
>
> You might also want to provide your .conf file so people can see what tuning
> parameters you are using.
>
> kevin
>
> On Apr 29, 2011, at 11:41 AM, Jim Kusznir wrote:
>
>> Hi all:
>>
>> I believe I'm getting very poor performance from my pvfs2 system, and
>> would like to start trying to identify why.
>>
>> System details:
>> 3 pvfs2-io servers, all dedicated to pvfs2:
>> Dell Poweredge 1950 with
>> dual-socket E5310 (quad core 1.6Ghz)
>> 4GB RAM
>> dual onboard gig-E (using balance-alb bonding)
>> PERC 6/e raid for storage
>> PERC 5/i raid1 for metadata/OS storage
>> all servers are metadata servers as well as data servers
>> running pvfs2.8.2 on the servers
>>
>> My cluster has 24 nodes and 1 dedicated compute node, all single-gig
>> connected.
>>
>> Performance to the pvfs2 is extremely slow. Every time I check the
>> load on the pvfs2 nodes, its never greater than 50% and usually well
>> under that. Last time I had to chown a whole bunch of files (a user
>> had 30 directories with over 10k files each), it took nearly 2 full
>> days. I/O from the head node appears to be less than half (possibly
>> less than 1/4) of the rate to local disk. Calculated data rates are
>> well under 1GB. And previous tests showed that once I got I/O from
>> more than 3 or 4 nodes simultaneously, the performance plateaued.
>>
>> I'm not even sure how to proceed with the troubleshooting. The only
>> obvious question that came to mind is whether the bonding-alb is
>> helping or hurting things.
>>
>> --Jim
>> _______________________________________________
>> Pvfs2-users mailing list
>> [email protected]
>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
>
>
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
