There are some tuning params that you can look into here, by default there is a round robin loading on the servers and that is done in chunks of FlowBufferSize (iirc?), you can set this in your config file but by default the size is quite small (64k) and I've pushed it up over 1-2MB and seen drastic improvements in bandwidth for larger requests; but if you're doing tons of small requests this obviously wont help.
Can you attach your config file so we can see how things are setup? Kyle Schochenmaier On Mon, Dec 16, 2013 at 11:57 AM, Moye,Roger V <[email protected]>wrote: > > > Over the past weekend one of my users reported that his compute jobs > running on a server with local disks usually takes about 5 hours. However, > running the same jobs on our small Linux cluster using a PVFS filesystem > exceeded 24 hours. > > > > Here is the environment we are using: > > 1. RHEL 6.4 on PVFS servers and clients. > > 2. Computations are performed on any of 16 Linux clients, all > running RHEL 6.4. > > 3. We are running Orangefs-2.8.7. > > 4. We have 4 PVFS servers, each with an XFS filesystem on a ~35TB > RAID 6. Total PVFS filesystem is 146TB. > > 5. All components are connected via a 10GigE network. > > > > I started looking for the source of the problem. For the user(s) showing > this poor performance, I found that pvfs-client is using about 65% of the > CPU while the compute jobs themselves are using only 4% each. Thus the > compute nodes are very lightly loaded and the compute jobs are hardly doing > anything. The pvfs2-server process on each PVFS server is using about > 140% CPU. No time is being spent in the wait state (so I assume the speed > of the disks are not an issue). While the system was exhibiting poor > performance I tried to read/write some 10GB files myself and found the > performance to be normal for this system (around 450MB/s). I used ‘iperf’ > to measure the network bandwidth between the affected nodes and the PVFS > serves and found it normal at 9.38Gb/s. The directories that the users are > reading/writing only have a few files in each. > > > > Iostat shows that the disk system is being constantly read by something as > shown by ‘iostat –d 2’ on the PVFS servers: > > Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > > sda 0.00 0.00 0.00 0 0 > > sdb 19.00 4864.00 0.00 9728 0 > > dm-0 0.00 0.00 0.00 0 0 > > dm-1 0.00 0.00 0.00 0 0 > > > > iostat has looked like over the last 48 hours (since Saturday). > > > > I can not find any documentation on how to get stats directly from pvfs2 > so I tried this command: > > pvfs2-statfs –m /pvfs2-mnt > > > > I received these results: > > I/O server statistics: > > --------------------------------------- > > > > server: tcp://dqspvfs01:3334 > > RAM bytes total : 33619419136 > > RAM bytes free : 284790784 > > uptime (seconds) : 14499577 > > load averages : 0 0 0 > > handles available: 2305843009213589192 > > handles total : 2305843009213693950 > > bytes available : 31456490479616 > > bytes total : 40000112558080 > > mode: serving both metadata and I/O data > > > > server: tcp://dqspvfs02:3334 > > RAM bytes total : 33619419136 > > RAM bytes free : 217452544 > > uptime (seconds) : 14499840 > > load averages : 0 0 0 > > handles available: 2305843009213589104 > > handles total : 2305843009213693950 > > bytes available : 31456971476992 > > bytes total : 40000112558080 > > mode: serving both metadata and I/O data > > > > server: tcp://dqspvfs03:3334 > > RAM bytes total : 33619419136 > > RAM bytes free : 428965888 > > uptime (seconds) : 5437269 > > load averages : 320 192 0 > > handles available: 2305843009213588929 > > handles total : 2305843009213693950 > > bytes available : 31439132123136 > > bytes total : 40000112558080 > > mode: serving both metadata and I/O data > > > > server: tcp://dqspvfs04:3334 > > RAM bytes total : 33619419136 > > RAM bytes free : 223281152 > > uptime (seconds) : 10089825 > > load averages : 1664 3072 0 > > handles available: 2305843009213588989 > > handles total : 2305843009213693950 > > bytes available : 31452933193728 > > bytes total : 40000112558080 > > mode: serving both metadata and I/O data > > > > Notice that the ‘load averages’ are 0 for servers #1 and #2 but not #3 and > #4. Earlier this morning only #4 showed a non-zero load average. The > other three were 0. What does this number mean? > > > > My two theories about the source of the problem are: > > 1. Someone is doing ‘a lot’ of tiny reads. > > 2. Or, based on the load averages the PVFS filesystem is somehow > not balanced. All of the load is on a single server. > > > > How can I prove either of these? Or what other types of diagnostics can I > do? > > > > Thank you! > > -Roger > > > > ----------------------------------------------- > > Roger V. Moye > > Systems Analyst III > > XSEDE Campus Champion > > University of Texas - MD Anderson Cancer Center > > Division of Quantitative Sciences > > Pickens Academic Tower - FCT4.6109 > > Houston, Texas > > (713) 792-2134 > > ----------------------------------------------------------- > > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
