On Thu, Feb 18, 2010 at 4:29 PM, Darren Govoni <[email protected]> wrote: > Hi, > I'm developing a python DFS/DHT and Software RAID file system that > resembles Hadoop (among others). > I wanted to convey the traits of my filesystem and see how it compares > to HDFS but my aim is to develop different capabilities, not the exact > same. Basically, what my DFS can do now is: > > - zero-conf distributed file system. No node manager or database. > - fully de-centralized, distributed. Peer like. No single point of > failure. > - Can stripe and stagger blocks across servers with any level of > RAID/redundancy. > - Can retrieve files by key from any node in the mesh. > - Reconstructs ordered blocks from the mesh on-the-fly. > - Flat keyspace > - Can reconstruct the keyspace on-the-fly, from any node. There is no > database. > - Entirely stateless. Requires no database or persistent information. > - Fault-tolerant streaming. > - Auto-matic performance load balancing > - Auto-matic diskspace load balancing > - Pluggable blocking classes. When adding files to the DFS, they can be > chopped into blocks using pluggable classes with different rules. > - Small and fast footprint (less than 20k of Python code) > - HTTP/web friendly. Any URL client can use it. > > There's more, but I wanted to craft my requirements to provide new or > different capabilities from HDFS. > So is this identical to HDFS? Does it sound useful? Thanks for any > thoughts. > > Darren >
Your description sounds much like http://www.danga.com/mogilefs/ Many of your points like: > - Can stripe and stagger blocks across servers with any level of > RAID/redundancy. Seem like carryover ideas from typical large/multi-disk storage systems. That is different then hadoop in many ways. HDFS was created to be a file system for batch systems. Supporting striping or RAID levels has not been the target of hadoop. If you want to compare what you are looking to do with what hadoop does http://hadoop.apache.org/common/docs/current/hdfs_design.html should help you.
