[
https://issues.apache.org/jira/browse/HDFS-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038091#comment-14038091
]
Chris Nauroth commented on HDFS-5442:
-------------------------------------
Thanks for the response, [~vinayrpet]. I read your comments and the revised
design document, but I still have a few questions.
bq. But in case of async mode, operations will not fail, but there will be a
resync mechanism.
bq. Since as mentioned above, there will not be chance of missing the edits,
Operator need not discard complete instance of any DC.
I didn't understand this response, because I didn't find anything in the design
about a "resync mechanism" or something that can reconcile the two divergent
namespaces, and of course this is quite a hard problem in general. This leads
me to believe that after a split-brain accident (ouch!), the only possible
recovery is to pick the namespace that is "better" according to the operator's
judgment, discard the other one, bootstrap a new mirror, and wait for a full
resync from the primary. Is that right, or am I missing something?
Just to reiterate, I think claiming that we can switch to the mirror "in a
matter of minutes" is over-stating the capabilities. Even for a quick admin
that has addressed split-brain concerns and has the failover steps fully
scripted, there is still the block reporting and safe mode time to consider.
I'm sure some admins will want to fail back over to their "usual primary" even
after activating the mirror. To do this, I imagine the admin will need to put
the new primary into safe mode to block incoming writes, and then make sure the
original primary DC gets caught up with the new primary on any asynchronous
metadata or block writes. Will there be a tool to help admins check this?
Without consideration of software upgrades, it looks like deployments that use
this will face a tough choice between sticking to an old version vs.
bootstrapping a whole new mirror cluster from scratch and doing a possibly
costly full re-replication.
> Zero loss HDFS data replication for multiple datacenters
> --------------------------------------------------------
>
> Key: HDFS-5442
> URL: https://issues.apache.org/jira/browse/HDFS-5442
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Avik Dey
> Assignee: Dian Fu
> Attachments: Disaster Recovery Solution for Hadoop.pdf, Disaster
> Recovery Solution for Hadoop.pdf, Disaster Recovery Solution for Hadoop.pdf
>
>
> Hadoop is architected to operate efficiently at scale for normal hardware
> failures within a datacenter. Hadoop is not designed today to handle
> datacenter failures. Although HDFS is not designed for nor deployed in
> configurations spanning multiple datacenters, replicating data from one
> location to another is common practice for disaster recovery and global
> service availability. There are current solutions available for batch
> replication using data copy/export tools. However, while providing some
> backup capability for HDFS data, they do not provide the capability to
> recover all your HDFS data from a datacenter failure and be up and running
> again with a fully operational Hadoop cluster in another datacenter in a
> matter of minutes. For disaster recovery from a datacenter failure, we should
> provide a fully distributed, zero data loss, low latency, high throughput and
> secure HDFS data replication solution for multiple datacenter setup.
> Design and code for Phase-1 to follow soon.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
