Hi Kshitij, A couple other questions and things to look at:
1) what interface are you using with IOR, MPIIO or POSIX? 2) what protocol are you using, (tcp, ib) and what is the link speed? 3) is the PVFS2 file system you're comparing to ext4 just the single host or is it both hosts attached to SSD 4) With 32MB transfer size (from IOR, right?) does that match the stripe size you're using in the PVFS2 file system? 5) are you using directio or alt-aio? Beyond that, if you could watch top for something CPU bound or swapping during testing that may show what's going on. Also, if you could watch iostat to see what's happening with the disks while running the test on PVFS2.. Michael On Wed, Dec 14, 2011 at 2:43 AM, Kshitij Mehta <[email protected]> wrote: > I am using a transfer size of 32 MB, which should have shown much better > performance (My apologies for not mentioning this before). The total file > size being written is 8GB. > > - Kshitij > > On Dec 14, 2011, at 1:34 AM, Kyle Schochenmaier <[email protected]> > wrote: > > Hi Kshitij - > > This is the expected behaviour, PVFS2 is not highly optimized for small > writes/reads, which is what IOR is typically performing. So you will > always see degraded performances here compared to the underlying > filesystem's base performance. > > There are ways to tune to help optimize for this type of access. > > If you set your IOR block accesses to something larger such as 64K instead > of the default (4K?) I think you would see performances which are closer. > > This used to be pretty well documented in the FAQ documents for PVFS, i'm > not sure where the links are now.. > > Cheers, > Kyle Schochenmaier > > > On Wed, Dec 14, 2011 at 1:09 AM, Kshitij Mehta <[email protected]> wrote: > >> Well , heres why I wanted to trace in the first place. >> >> I have a test configuration where we have configured PVFS2 over an SSD >> storage. There are two I/O servers that talk to the SSD storage through >> Infiniband (There are 2 IB channels going into the SSD, and each storage >> server can 'see' one half of the SSD). >> >> Now I used the IOR benchmark to test the write bandwidth. I first spawn a >> process on the I/O server such that it writes data to the underlying ext4 >> file system on the SSD instead of PVFS2. I see a bandwidth of ~350 MB/s. >> Now I spawn a process on the same I/O server and write data to the PVFS2 >> file system configured over the SSD, and I see a write bandwidth of ~180 >> MB/s. >> >> This seems to represent some kind of overhead with PVFS2, but seems too >> large. Has anybody else seen similar results? Is the overhead of pvfs2 >> documented? >> >> Do let me know if something is not clear or if you have additional >> questions about the above setup. >> >> Here are some other details: >> I/O servers: dual core with 2G main memory each. >> PVFS 2.8.2 >> >> Thanks, >> Kshitij >> >> -----Original Message----- >> From: Julian Kunkel [mailto:[email protected]] >> Sent: Tuesday, December 13, 2011 3:10 AM >> To: Kshitij Mehta >> Cc: [email protected] >> Subject: Re: [Pvfs2-users] Tracing pvfs2 internals >> >> Dear Kshitij, >> we have a version of OrangeFS which is instrumented with HDTrace, there >> you can record detailed information about activity of statemachines and I/O. >> For a description see the thesis: >> >> http://wr.informatik.uni-hamburg.de/_media/research:theses:Tien%20Duc%20Tien_Tracing%20Internal%20Behavior%20in%20PVFS.pdf >> >> The code is available in our redmine (here is a link to the wiki): >> http://redmine.wr.informatik.uni-hamburg.de/projects/piosimhd/wiki >> >> I consider the tracing implemented in PVFS as rather robust, since it is >> our second implementation with PVFS_hints. >> However, you might encounter some issues with the build system. >> If you want to try it and you need help, just ask. >> >> Regards, >> Julian Kunkel >> >> >> >> 2011/12/13 Kshitij Mehta <[email protected]>: >> > Hello, >> > >> > Is there a way I can trace/measure the internal behavior of pvfs2? >> > Suppose I have a simple I/O code that writes to pvfs2, I would like to >> > find out how much time exactly do various internal operations of Pvfs2 >> > take (metadata lookup, creating iovecs, etc.), before data is finally >> pushed to disk. >> > >> > >> > >> > Is there a configure option (what does `enabletracing` do in the >> > config >> > file) ? Or is there any other way to determine this ? >> > >> > >> > >> > Thanks, >> > Kshitij >> > >> > >> > >> > >> > _______________________________________________ >> > Pvfs2-users mailing list >> > [email protected] >> > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> > >> >> >> _______________________________________________ >> Pvfs2-users mailing list >> [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
