[
https://issues.apache.org/jira/browse/HDFS-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HDFS-6440:
------------------------------
Attachment: hdfs-6440-cdh-4.5-full.patch
Attaching patch for CDH 4.5.0, since this is what we run on at Salesforce. I'll
update to the proper open source branches once I've got some consensus that
this is the 'right' way to go about doing these changes.
For what its worth, all the unit tests have passed (at one point.. they are a
bit flaky :)) and we've been doing some m/r based load tests with a chaos
monkey(1) and have been successful (2).
As mentioned in the issue description, there is a majority of the complexity in
the checkpointing. For this, I went with a 'first writer wins' approach. From
the standpoint of the standby node, if you're checkpoint isn't accepted (the
other NN got one there first) then you back-off for 2x the usual wait time
before trying to send it again. I had to add another response code to the
GetImageServlet to support the 'someone else won' logic - its not the cleanest
solution as other HTTP response codes fit better, but they are already being
used to indicate other failure cases.
Other notable changes:
- EditLogTailer checks all NN when rolling logs
- BootstapSTanby uses all namenodes when attempting bootstrap
- update block token creation to segment integer space by NN id
- updating NN dir creation to include ns index (3)
- updated a lot of the tests to support testing across all the NNs, including
HAStressTestHarness, and a circular linked list writing test
- moved to using a multi-map of NNs in MiniDFSCluster as they are no longer
limited to two NNs.
(1) each mapper writes a linked list of files, then ensures it can read it back
(2) required a bit of tuning to ride over reconnections once we started killing
NNs more than every 60 seconds
(3) Not sure the best way to update the tests for this. Right now made some
changes to TestDFSUpgradeFromImage, but that might need a little rework.
> Support more than 2 NameNodes
> -----------------------------
>
> Key: HDFS-6440
> URL: https://issues.apache.org/jira/browse/HDFS-6440
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: auto-failover, ha, namenode
> Reporter: Jesse Yates
> Attachments: hdfs-6440-cdh-4.5-full.patch
>
>
> Most of the work is already done to support more than 2 NameNodes (one
> active, one standby). This would be the last bit to support running multiple
> _standby_ NameNodes; one of the standbys should be available for fail-over.
> Mostly, this is a matter of updating how we parse configurations, some
> complexity around managing the checkpointing, and updating a whole lot of
> tests.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
