The dd block size determines how much data is given to PVFS2 in any one write request. Thus, if the write request is given 2MB of data, that data is divided up and sent to the 20 I/O servers all at the same time (see note below). If the write request is given only 64K of data, then a request is sent to the one server where the next 64k is to be written. So, throughput for larger requests generally perform better than for small requests, depending on your network delay, how busy your servers are, and the number of I/O servers in your filesystem. There is also some overhead associated with moving the data from user space to kernel space; so, you incur more OS overhead using 64k blocks than you would with 2MB blocks.
For example, if you use the linux command "cp" and compare its performance with "pvfs2-cp" to copy a large amount of data from a unix filesystem into a PVFS filesystem, you will immediately notice that pvfs2-cp is faster than cp. pvfs2-cp performs better than cp because it uses a default buffer size of 10MB, while cp uses the stripe size, in your case 64k. So, it will take cp longer to transfer the same amount of data than it will with pvfs2-cp. NOTE: By default, PVFS uses 8, 256k buffers to transfer data to a server. Once the connection is made, PVFS transmits data to the server using these 256k-sized buffers as fast as it can. You can think of the 8 buffers as the PVFS window size (if you are familiar with TCP terminology). With 20 I/O servers, you have 20 of these windows pushing out data over the network just as fast as possible. Hope this helps! Becky On Sun, Oct 9, 2011 at 5:34 AM, belcampo <[email protected]> wrote: > On 10/06/2011 10:36 PM, Kshitij Mehta wrote: > >> Hello, >> I have a pvfs2 file system configured over 20 IO servers with a default >> stripe size of 64Kbytes. >> I am running a simple test program where I write a matrix to file. >> >> This is what I see: >> If the 1GByte matrix is written in block sizes of 2Mbytes, the performance >> is much better than writing the matrix in blocks of 64Kbytes. I am not sure >> I understand why. Since the stripe size is 64KB, every block of 2MB >> eventually gets broken into 64KB blocks which are written to the IO servers, >> so the performance should nearly be equal. I would understand why writing >> block size < stripe_size should perform badly, but when the block size >> exceeds the stripe size, I expect the performance to peak out. >> >> Can someone explain what happens here? Your help is appreciated. >> > I can't explain only confirm. I also did some test with following results. > > with pvfs2-cp > 18.18 > over pvfs2fuse > dd blocksize MB/s > 4k 4.4 > 8k 6.3 > 16k 7.3 > 32k 8.8 > 64k 9.9 > 128k 18.7 > 256k 18.7 > 512k 18.8 > 1024k 18.8 > 2048k 18.8 > > over pvfs2fuse > cp 8.2 > rsync 14.8 > > over nfs > cp 10.6 > rsync 11.0 > > Further earlier was/is mentioned that ongoing effort is put in optimizing > pvfs2/orangefs with small file-sizes. So AFAIK it is by design, but not > knowing the reasoning behind it. > > >> Best, >> Kshitij Mehta >> >> ______________________________**_________________ >> Pvfs2-users mailing list >> Pvfs2-users@beowulf-**underground.org<[email protected]> >> http://www.beowulf-**underground.org/mailman/**listinfo/pvfs2-users<http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users> >> > > ______________________________**_________________ > Pvfs2-users mailing list > Pvfs2-users@beowulf-**underground.org<[email protected]> > http://www.beowulf-**underground.org/mailman/**listinfo/pvfs2-users<http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users> > -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
