Ashutosh Gupta created HDFS-16792:
-------------------------------------
Summary: Add -newEditsOnly option to name node
initializeSharedEdits command to format unformatted journal nodes for master
replacement
Key: HDFS-16792
URL: https://issues.apache.org/jira/browse/HDFS-16792
Project: Hadoop HDFS
Issue Type: New Feature
Components: journal-node
Affects Versions: 3.3.4, 3.3.3
Reporter: Ashutosh Gupta
Assignee: Ashutosh Gupta
Add -newEditsOnly option to name node initializeSharedEdits command to format
unformatted journal nodes for master replacement.
The current hadoop has limitations on formatting journal nodes. First, it
requires all journal nodes to be present when formatting name node. After that,
new journal nodes could not be formatted unless we reformat all of them and
destroy all HDFS edit logs.
$ hdfs namenode -initializeSharedEdits -newEditsOnly
There are two cases here:
1) the replaced master instance has name node running on it, then the above
command will only be run once on the existing name node.
2) the replaced master instance does not have a name node running on it, then
the above command will be run twice. But the above command is idempotent as
long as the same command will not be issued at the same time, which could be
guaranteed by instance controller when reconfiguring master instances. The
reason is that all hadoop name node commands such as format, bootstrap, and
initializeSharedEdits are not designed to be thread safe and they rely on an
external mechanism, which is a manual work for hadoop admin for on-premise
clusters, to make sure that they are not executed concurrently. We still keep
the same behavior of the initializeSharedEdits command to minimize the
code/logic changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
