[
https://issues.apache.org/jira/browse/HDFS-3566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sumadhur Reddy Bolli updated HDFS-3566:
---------------------------------------
Target Version/s: 1-win (was: 1.1.0)
> Custom Replication Policy for Azure
> -----------------------------------
>
> Key: HDFS-3566
> URL: https://issues.apache.org/jira/browse/HDFS-3566
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: Sumadhur Reddy Bolli
>
> Azure has logical concepts like fault and upgrade domains. Each fault domain
> spans multiple upgrade domains and each upgrade domain spans multiple fault
> domains. Machines are spread typically evenly across both fault and upgrade
> domains. Fault domain failures are typically catastrophic/unplanned failures
> and data loss possibility is high. An upgrade domain can be taken down by
> azure for maintenance periodically. Each time an upgrade domain is taken down
> a small percentage of machines in the upgrade domain(typically 1-2%) are
> replaced due to disk failures, thus losing data. Assuming the default
> replication factor 3, any 3 data nodes going down at the same time would mean
> potential data loss. So, it is important to have a policy that spreads
> replicas across both fault and upgrade domains to ensure practically no data
> loss. The problem here is two dimensional and the default policy in hadoop is
> one-dimensional. This policy would spread the datanodes across atleast 2
> fault domains and three upgrade domains to prevent data loss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
