I've seen this through VFS. Here are some graphs (ignore the pNFS lines) with 4MB block size and the standard read/write interfaces. Overview: http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/
Single file: http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/experiments-5_13095_image001.gif Separate files: http://www.citi.umich.edu/projects/asci/pnfs/darkstar-exp/experiments-5_15651_image001.gif As the # of data servers increases, aggregate read performance increases but individual client read performance suffers a little. Dean On Wed, 30 Nov 2005, Murali Vilayannur wrote: > Hi Praveen, > Sorry for the late response. Unfortunately, we have not seen this kind of > behavior on many of our setups thru the MPI-I/O/ADIO-PVFS2 interfaces. > It is conceivable that performance thru the VFS do not scale with > number of servers. > > Could you tell us how the file is being read (i.e using what interfaces, > what block sizes etc)? > There still aren't that many tweakable settings just yet, but we are > working on some I/O performance tuning and we will hopefully have a better > handle on what settings affect performance. > Thanks for the reports, > Murali > > On Mon, 21 Nov 2005, Praveen KJ wrote: > > > Hi, > > We have a pvfs2 client reading a 5 GB file sequentially from > > multiple pvfs2 data servers. > > There is just one metadata server. All the nodes are connected via > > GigE network. > > > > The problem we are seeing is that the single-client read throughput > > decreases as the number of servers increase. > > With 24 servers we see around 5 MB/s read compared to 50 MB/s read > > with just 2 servers. > > The setup between each experiment is exactly same, except the > > number of servers. > > > > We have tried a couple of pvfs2 versions including the latest 1.3.1. > > Is there any setting we have to tweak ? We have taken only pvfs2 > > defaults till now. > > Some of the relevant options are pasted below. > > > > Thanks, > > Praveen > > > > ------------------------------------------ > > pvfs2-fs.conf ( except hostnames ) > > ------------------------------------------ > > <Defaults> > > UnexpectedRequests 50 > > LogFile /tmp/pvfs2-server.log > > EventLogging none > > LogStamp usec > > BMIModules bmi_tcp > > FlowModules flowproto_multiqueue > > PerfUpdateInterval 1000 > > ServerJobBMITimeoutSecs 30 > > ServerJobFlowTimeoutSecs 30 > > ClientJobBMITimeoutSecs 300 > > ClientJobFlowTimeoutSecs 300 > > ClientRetryLimit 5 > > ClientRetryDelayMilliSecs 2000 > > </Defaults> > > > > <Filesystem> > > Name pvfs2-fs > > ID 1175641402 > > RootHandle 1048576 > > <StorageHints> > > TroveSyncMeta yes > > TroveSyncData no > > AttrCacheKeywords datafile_handles,metafile_dist > > AttrCacheKeywords dir_ent, symlink_target > > AttrCacheSize 4093 > > AttrCacheMaxNumElems 32768 > > </StorageHints> > > </Filesystem> > > _______________________________________________ > > PVFS2-users mailing list > > [email protected] > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > _______________________________________________ > PVFS2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > _______________________________________________ PVFS2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
