Hi Jeremy, > > I write a single 64 byte file (with one I/O request), iozone tells me > > something like '295548', which means ~295 MB/s. Dividing the file size > > by the bandwidth, I get the time that was needed to create the file and > > write the 64 bytes (with a single request). In this case, the time is > > about 0,2 micro seconds which is way below the RTT. > > That seems oddly fast for such a small file over any latency. Since you > shouldn't even be able to lock the file in that time.
OK, thanks. So I have to see at least a latency of one RTT. I think I need to dig through the data again. I might have made a mistake in one of the formulas ... > > That mean for a Lustre file system, if I create a file and write 64 > > bytes, the client sends two(?) RPCs to the server and does not wait for > > the completion. Is this correct? But it will wait for the completion of > > both RPCs when the file ist closed? > You can see what Lustre is doing if the client isn't doing any other > activity and you enable some tracing. "sysctl -w lnet.debug='+rpctrace > vfstrace'" should allow you to see the VFS ops ll_file_open, > ll_file_writev/ll_file_aio_write, ll_file_release, along with any RPCs > generated by them. You should see an RPC for the file open which will > be a 101 opcode for requesting the lock and you should see a reply AFAIK > before the client actually attempts to write any data. So that should > bring your time upto at least 4 ms for 1 RTT. The initial write should > request a lock from the first stripe followed by a OST write RPC (opcode > 4) followed by a file close (opcode 35). I ran a test over 4 ms > latency so you can see what I'm referring to. I thought that there was > a patch in Lustre a few months back that forced a flush before a file > close, but this is from a 1.8.5 client so I'm guessing that isn't how it > works because between when I closed the file and the end I had to "sync" > for the OST write to show up. OK, understood. > > The numbers look different when I disable the client side cache by > > setting max_dirty_mb to 0. > > Without any grant I think all RPCs have to be synchronous so you'll see > a huge performance hit over latency. These numbers look different :) I'am still trying to make sense of a couple of measurements and to put some useful data into some charts for the LUG this year. Michael -- Michael Kluge, M.Sc. Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) D-01062 Dresden Germany Contact: Willersbau, Room A 208 Phone: (+49) 351 463-34217 Fax: (+49) 351 463-37773 e-mail: [email protected] WWW: http://www.tu-dresden.de/zih
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
