Rob/Murali/Dean,
The read throughput seems to be variable
considering the iozone record size.
The experimental setup was same as described before. The variation of
record size and throughput for a sample ..
---------------------------------------------------------------------------------------------
record_size throughput comments
---------------------------------------------------------------------------------------------
( Steady increment of
throughput till 128k)
128k - 65 MB/s ( Steady decrement of throughput
till 1M)
1M - 6 MB/s ( Steady increment till - )
4 M - 16 MB/s
A graph plot would look wavy.
Iozone is not our final application :) We just
wanted to observe the performance space and also make sure installation
was done right. So your inputs were definitely helpful.
Thanks,
Praveen
Murali Vilayannur wrote:
Thanks Dean!
I will take a look at the VFS code paths/performance analysis.
Murali
On Wed, 30 Nov 2005, Dean Hildebrand wrote:
I've seen this through VFS. Here are some graphs (ignore the pNFS lines)
with 4MB block size and the standard read/write interfaces.
Overview:
http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/
Single file:
http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/experiments-5_13095_image001.gif
Separate files:
http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/experiments-5_15651_image001.gif
As the # of data servers increases, aggregate read performance increases
but individual client read performance suffers a little.
Dean
On Wed, 30 Nov 2005, Murali Vilayannur wrote:
Hi Praveen,
Sorry for the late response. Unfortunately, we have not seen this kind of
behavior on many of our setups thru the MPI-I/O/ADIO-PVFS2 interfaces.
It is conceivable that performance thru the VFS do not scale with
number of servers.
Could you tell us how the file is being read (i.e using what interfaces,
what block sizes etc)?
There still aren't that many tweakable settings just yet, but we are
working on some I/O performance tuning and we will hopefully have a better
handle on what settings affect performance.
Thanks for the reports,
Murali
On Mon, 21 Nov 2005, Praveen KJ wrote:
Hi,
We have a pvfs2 client reading a 5 GB file sequentially from
multiple pvfs2 data servers.
There is just one metadata server. All the nodes are connected via
GigE network.
The problem we are seeing is that the single-client read throughput
decreases as the number of servers increase.
With 24 servers we see around 5 MB/s read compared to 50 MB/s read
with just 2 servers.
The setup between each experiment is exactly same, except the
number of servers.
We have tried a couple of pvfs2 versions including the latest 1.3.1.
Is there any setting we have to tweak ? We have taken only pvfs2
defaults till now.
Some of the relevant options are pasted below.
Thanks,
Praveen
------------------------------------------
pvfs2-fs.conf ( except hostnames )
------------------------------------------
<Defaults>
UnexpectedRequests 50
LogFile /tmp/pvfs2-server.log
EventLogging none
LogStamp usec
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
</Defaults>
<Filesystem>
Name pvfs2-fs
ID 1175641402
RootHandle 1048576
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
AttrCacheKeywords datafile_handles,metafile_dist
AttrCacheKeywords dir_ent, symlink_target
AttrCacheSize 4093
AttrCacheMaxNumElems 32768
</StorageHints>
</Filesystem>
_______________________________________________
PVFS2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
PVFS2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
_______________________________________________
PVFS2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
