On Nov 20, 2006 23:47 -0400, Peter Bojanic wrote: > It's true. I did raise users' expectations for improved metadata > performance for patchless client. There will probably be a number of > users who are interested in plans about this.
> On 2006-11-20, at 11:33 , Bernadat, Philippe wrote: > >At last week's CFS BOF you said that even without the kernel patches > >you may be able to get as good or even better performance than before. > > > >I remember you telling me that I could get more info from Andreas or > >someone else. So can we know more ? The patchless client came about because many customers do not want to have support issues with a patched client. Also, these patches are in a tricky part of the kernel and are a source of bugs. So removing them is beneficial for both CFS and customers. In theory (and practise in some cases) this hurts metadata performance, and that needs to be addressed. The patchless client still utilizes the intent open operation (which was accepted into the upstream kernels in a "not useful for anything but open" form). Lustre uses normal VFS operations for everything else (meaning that some operations might take multiple RPCs where they would previously take only a single one). The truth is that in many real-world use cases (such as GNU rm, ls as opposed to focussed benchmarks) the Lustre "raw" operations weren't of much benefit because the user-space code is doing an explicit "stat" and "access" call on the target file anyways, destroying any advantage that the raw operations provide. More significantly, in detailed RPC investigation for these use cases it is seen that the Lustre network protocol can be optimized to either reduce the RPC count, or overlap serial operations to reduce total time. In the "rm -r" case (where it is doing stat+access calls on the target anyways), we have observed that the MDS unlink and OST destroy operations are always immediately cancelling the locks the client was just granted for stat. This suggests that if we bundle the lock cancellation with the original unlink/destroy RPC request that we save a full round-trip RPC latency for each during the unlink syscall. This is expected to improve the single-file "rm -r" performance by 25% (remove 2 of 8 RPCs) over the existing code, regardless of whether the client is patched or not. Similar cases exist for rename, open(O_WRONLY), etc where high-level analysis of the RPCs being done show opportunity to improve the wire protocol to remove costly RPCs without need to modify the kernel. There are other similar (though longer-term) optimizations to the protocol such as size-on-MDS, client side stat-ahead for "ls -l" (readdir+stat) type operations, etc that can also be implemented. In the "rm -r" case above, stat-ahead would hide the latency of the MDS and OST stat/getattr operations, and size-on-MDS would avoid the OST getattr RPCs for closed files (major win all around, though very complex to get right). Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
