Steve, >Summary: I'm not sure that HDFS is the right FS in this world, as it >contains a lot of assumptions about system stability and HDD persistence >that aren't valid any more. With the ability to plug in new placers you >could do tricks like ensure 1 replica lives in a persistent blockstore >(and rely on it always being there), and add other replicas in transient >storage if the data is about to be needed in jobs.
Can you please shed more light on the statement "... as it contains a lot of assumptions about system stability and HDD persistence that aren't valid any more..." ? I know that you were doing some analysis of disk failure modes sometime ago. Is this the result of that research ? I am very interested. Thanks, - milind --- Milind Bhandarkar Greenplum Labs, EMC (Disclaimer: Opinions expressed in this email are those of the author, and do not necessarily represent the views of any organization, past or present, the author might be affiliated with.)
