Hi Kshitij - What kind of performance do you get with pvfs2-cp ? If you set the block size for pvfs2-cp of some large file (1GB+) from /tmp/ on your client to the pvfs2-fs to 1MB+ do you get decent performance ? -- we should be testing the performance of in-memory pvfs2 at this point..
Kyle Schochenmaier On Wed, Dec 14, 2011 at 1:09 PM, Kshitij Mehta <[email protected]> wrote: > 1) what interface are you using with IOR, MPIIO or POSIX?**** > > ** ** > > MPIIO**** > > ** ** > > 2) what protocol are you using, (tcp, ib) and what is the link speed? **** > > ** ** > > IB SDR , with a theoretical of 1 GB/s**** > > ** ** > > 3) is the PVFS2 file system you're comparing to ext4 just the single host > or is it both hosts attached to SSD**** > > ** ** > > Both hosts.**** > > ** ** > > 4) With 32MB transfer size (from IOR, right?) does that match the stripe > size you're using in the PVFS2 file system?**** > > ** ** > > Yes, we ran the test from IOR. The stripe size on PVFS2 was > set to 1 MB. I am seeing similar results when using varying transfer sizes > from 1MB through 1GB, doubling the transfer size in every run.**** > > ** ** > > 5) are you using directio or alt-aio?**** > > ** ** > > Alt-aio **** > > ** ** > > ** ** > > Thanks, **** > > Kshitij**** > > ** ** > > *From:* Michael Moore [mailto:[email protected]] > *Sent:* Wednesday, December 14, 2011 5:21 AM > *To:* Kshitij Mehta > *Cc:* Kyle Schochenmaier; [email protected] > > *Subject:* Re: [Pvfs2-users] Tracing pvfs2 internals**** > > ** ** > > Hi Kshitij,**** > > ** ** > > A couple other questions and things to look at:**** > > ** ** > > 1) what interface are you using with IOR, MPIIO or POSIX?**** > > 2) what protocol are you using, (tcp, ib) and what is the link speed? **** > > 3) is the PVFS2 file system you're comparing to ext4 just the single host > or is it both hosts attached to SSD > 4) With 32MB transfer size (from IOR, right?) does that match the stripe > size you're using in the PVFS2 file system?**** > > 5) are you using directio or alt-aio?**** > > ** ** > > Beyond that, if you could watch top for something CPU bound or swapping > during testing that may show what's going on. Also, if you could watch > iostat to see what's happening with the disks while running the test on > PVFS2..**** > > ** ** > > Michael**** > > ** ** > > On Wed, Dec 14, 2011 at 2:43 AM, Kshitij Mehta <[email protected]> wrote:** > ** > > I am using a transfer size of 32 MB, which should have shown much better > performance (My apologies for not mentioning this before). The total file > size being written is 8GB. > > - Kshitij **** > > > On Dec 14, 2011, at 1:34 AM, Kyle Schochenmaier <[email protected]> > wrote:**** > > Hi Kshitij -**** > > ** ** > > This is the expected behaviour, PVFS2 is not highly optimized for small > writes/reads, which is what IOR is typically performing. So you will > always see degraded performances here compared to the underlying > filesystem's base performance.**** > > ** ** > > There are ways to tune to help optimize for this type of access.**** > > ** ** > > If you set your IOR block accesses to something larger such as 64K instead > of the default (4K?) I think you would see performances which are closer.* > *** > > ** ** > > This used to be pretty well documented in the FAQ documents for PVFS, i'm > not sure where the links are now..**** > > ** ** > > Cheers,**** > > Kyle Schochenmaier > > **** > > On Wed, Dec 14, 2011 at 1:09 AM, Kshitij Mehta <[email protected]> wrote:** > ** > > Well , heres why I wanted to trace in the first place. > > I have a test configuration where we have configured PVFS2 over an SSD > storage. There are two I/O servers that talk to the SSD storage through > Infiniband (There are 2 IB channels going into the SSD, and each storage > server can 'see' one half of the SSD). > > Now I used the IOR benchmark to test the write bandwidth. I first spawn a > process on the I/O server such that it writes data to the underlying ext4 > file system on the SSD instead of PVFS2. I see a bandwidth of ~350 MB/s. > Now I spawn a process on the same I/O server and write data to the PVFS2 > file system configured over the SSD, and I see a write bandwidth of ~180 > MB/s. > > This seems to represent some kind of overhead with PVFS2, but seems too > large. Has anybody else seen similar results? Is the overhead of pvfs2 > documented? > > Do let me know if something is not clear or if you have additional > questions about the above setup. > > Here are some other details: > I/O servers: dual core with 2G main memory each. > PVFS 2.8.2 > > Thanks, > Kshitij**** > > > -----Original Message----- > From: Julian Kunkel [mailto:[email protected]] > Sent: Tuesday, December 13, 2011 3:10 AM > To: Kshitij Mehta > Cc: [email protected] > Subject: Re: [Pvfs2-users] Tracing pvfs2 internals > > Dear Kshitij, > we have a version of OrangeFS which is instrumented with HDTrace, there > you can record detailed information about activity of statemachines and I/O. > For a description see the thesis: > > http://wr.informatik.uni-hamburg.de/_media/research:theses:Tien%20Duc%20Tien_Tracing%20Internal%20Behavior%20in%20PVFS.pdf > > The code is available in our redmine (here is a link to the wiki): > http://redmine.wr.informatik.uni-hamburg.de/projects/piosimhd/wiki > > I consider the tracing implemented in PVFS as rather robust, since it is > our second implementation with PVFS_hints. > However, you might encounter some issues with the build system. > If you want to try it and you need help, just ask. > > Regards, > Julian Kunkel > > > > 2011/12/13 Kshitij Mehta <[email protected]>: > > Hello, > > > > Is there a way I can trace/measure the internal behavior of pvfs2? > > Suppose I have a simple I/O code that writes to pvfs2, I would like to > > find out how much time exactly do various internal operations of Pvfs2 > > take (metadata lookup, creating iovecs, etc.), before data is finally > pushed to disk. > > > > > > > > Is there a configure option (what does `enabletracing` do in the > > config > > file) ? Or is there any other way to determine this ? > > > > > > > > Thanks, > > Kshitij > > > > > > > > > > _______________________________________________ > > Pvfs2-users mailing list > > [email protected] > > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users**** > > ** ** > > > _______________________________________________ > Pvfs2-users mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users**** > > ** ** >
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
