A Model for Data Durability
---------------------------
Key: HDFS-2535
URL: https://issues.apache.org/jira/browse/HDFS-2535
Project: Hadoop HDFS
Issue Type: Task
Components: documentation
Reporter: Robert Chansler
Assignee: Robert Chansler
Attachments: LosingBlocks.xlsx
This is a statistical model considering data durability in HDFS in the presence
of Data Node failures. The attached spreadsheet considers the probability of
losing a block with three replicas in the case of uncorrelated failures of DNs.
Also included is a section that looks at the consequences of simultaneous
failures.
The model parameters reflect experience at Yahoo with a large cluster. But it
is easy to change the parameters in the spreadsheet. Number of replicas is not
a easily adjusted parameter. And while published reports (the Google papers)
suggest that node failures are not really uncorrelated, this does give some
practical insight into HDFS durability.
I and others have quoted from this work in the past. I thought it good to make
the details conveniently available.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
