Comparing apples and oranges.
Lustre is great filesystem but has no native fault tolerance. If you want POSIX filesystem with high performance than Lustre does it. However, if you want to access data in a heterogeneous environment and not POSIX complaint then hdfs is the tool. I've read an earlier thread from you, before you choose a filesystem some things to consider: Cost: Any exoctic software hardware needed? (Lustre and hdfs can run very well on commodity hardware) Transparency: Any application change needed? Lustre wins in this! With hdfs you would have to convert or make changes in the way you access the data Scalability: Both scale well. Implementation cost: The cost of implementing a solution and maintaining it. HDFS wins. It will run on any server which will run java. No kernel modules, no kernel configuration, etc...it just works out of the box On Thu, Jan 27, 2011 at 4:44 PM, Nathan Rutman <nrut...@gmail.com> wrote: > In case others are interested, I ran a comparison of TestDFSIO on HDFS vs > Lustre. > This is on an 8-node Infiniband-connected cluster. For the Lustre test, we > replaced the HTTP transfer during the shuffle phase with a simple hardlink > to the data (since all data is always visible on all nodes with Lustre). > > > Max Map Thread = 80; Max Reduce Thread = 1; File Size = 512MB; Scheduler = > JobQueue; Buffer Size = Default; Number of Nodes = 8; Drive Speed = 80MB/s > > > The conclusion is that Lustre TestDFSIO performance is significantly better > than HDFS when using a fast network (as it theoretically should be). On a > slower network (e.g. 1gigE), I would not expect Lustre to show much > advantage over HDFS. > > -- --- Get your facts first, then you can distort them as you please.--
