[hadoop] 35/50: HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
This is an automated email from the ASF dual-hosted git repository. cliang pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit b8df864ab0e562fa0f3e498877afb65e8423 Author: Erik Krogen AuthorDate: Tue Dec 11 12:22:12 2018 -0800 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. --- .../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 + .../src/site/markdown/ObserverNameNode.md | 173 + 2 files changed, 175 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md index 1fe445b..9c72528 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md @@ -500,6 +500,7 @@ Usage: hdfs haadmin -transitionToActive [--forceactive] hdfs haadmin -transitionToStandby +hdfs haadmin -transitionToObserver hdfs haadmin -failover [--forcefence] [--forceactive] hdfs haadmin -getServiceState hdfs haadmin -getAllServiceState @@ -515,6 +516,7 @@ Usage: | `-getAllServiceState` | returns the state of all the NameNodes | | | `-transitionToActive` | transition the state of the given NameNode to Active (Warning: No fencing is done) | | `-transitionToStandby` | transition the state of the given NameNode to Standby (Warning: No fencing is done) | +| `-transitionToObserver` | transition the state of the given NameNode to Observer (Warning: No fencing is done) | | `-help` [cmd] | Displays help for the given command or all commands if none is specified. | See [HDFS HA with NFS](./HDFSHighAvailabilityWithNFS.html#Administrative_commands) or [HDFS HA with QJM](./HDFSHighAvailabilityWithQJM.html#Administrative_commands) for more information on this command. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md new file mode 100644 index 000..2548315 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md @@ -0,0 +1,173 @@ + + +Consistent Reads from HDFS Observer NameNode += + + + +Purpose + + +This guide provides an overview of the HDFS Observer NameNode feature +and how to configure/install it in a typical HA-enabled cluster. For a +detailed technical design overview, please check the doc attached to +HDFS-12943. + +Background +--- + +In a HA-enabled HDFS cluster (for more information, check +[HDFSHighAvailabilityWithQJM](./HDFSHighAvailabilityWithQJM.md)), there +is a single Active NameNode and one or more Standby NameNode(s). The +Active NameNode is responsible for serving all client requests, while +Standby NameNode just keep the up-to-date information regarding the +namespace, by tailing edit logs from JournalNodes, as well as block +location information, by receiving block reports from all the DataNodes. +One drawback of this architecture is that the Active NameNode could be a +single bottle-neck and be overloaded with client requests, especially in +a busy cluster. + +The Consistent Reads from HDFS Observer NameNode feature addresses the +above by introducing a new type of NameNode called **Observer +NameNode**. Similar to Standby NameNode, Observer NameNode keeps itself +up-to-date regarding the namespace and block location information. +In addition, it also has the ability to serve consistent reads, like +Active NameNode. Since read requests are the majority in a typical +environment, this can help to load balancing the NameNode traffic and +improve overall throughput. + +Architecture +-- + +In the new architecture, a HA cluster could consists of namenodes in 3 +different states: active, standby and observer. State transition can +happen between active and standby, standby and observer, but not +directly between active and observer. + +To ensure read-after-write consistency within a single client, a state +ID, which is implemented using transaction ID within NameNode, is +introduced in RPC headers. When a client performs write through Active +NameNode, it updates its state ID using the latest transaction ID from +the NameNode. When performing a subsequent read, the client passes this +state ID to Observe NameNode, which will then check against its own +transaction ID, and will ensure its own transaction ID has caught up +with the request's state ID, before serving the read request. + +Edit log tailing is critical for Observer NameNode as it directly affects +the latency between when a transaction is applied in Active NameNode and +when it is applied in the Observer NameNode. A new edit log tailing +mechanism, named "Edit Tailing Fast-Path", is introduced to +significantly reduce this
[hadoop] 35/50: HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
This is an automated email from the ASF dual-hosted git repository. cliang pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/hadoop.git commit f1c5b247b9a218066216828056637739b4b7351c Author: Erik Krogen AuthorDate: Tue Dec 11 12:22:12 2018 -0800 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. --- .../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 + .../src/site/markdown/ObserverNameNode.md | 173 + 2 files changed, 175 insertions(+) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md index e4f36a2..05372a1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md @@ -508,6 +508,7 @@ Usage: hdfs haadmin -transitionToActive [--forceactive] hdfs haadmin -transitionToStandby +hdfs haadmin -transitionToObserver hdfs haadmin -failover [--forcefence] [--forceactive] hdfs haadmin -getServiceState hdfs haadmin -getAllServiceState @@ -523,6 +524,7 @@ Usage: | `-getAllServiceState` | returns the state of all the NameNodes | | | `-transitionToActive` | transition the state of the given NameNode to Active (Warning: No fencing is done) | | `-transitionToStandby` | transition the state of the given NameNode to Standby (Warning: No fencing is done) | +| `-transitionToObserver` | transition the state of the given NameNode to Observer (Warning: No fencing is done) | | `-help` [cmd] | Displays help for the given command or all commands if none is specified. | See [HDFS HA with NFS](./HDFSHighAvailabilityWithNFS.html#Administrative_commands) or [HDFS HA with QJM](./HDFSHighAvailabilityWithQJM.html#Administrative_commands) for more information on this command. diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md new file mode 100644 index 000..2548315 --- /dev/null +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md @@ -0,0 +1,173 @@ + + +Consistent Reads from HDFS Observer NameNode += + + + +Purpose + + +This guide provides an overview of the HDFS Observer NameNode feature +and how to configure/install it in a typical HA-enabled cluster. For a +detailed technical design overview, please check the doc attached to +HDFS-12943. + +Background +--- + +In a HA-enabled HDFS cluster (for more information, check +[HDFSHighAvailabilityWithQJM](./HDFSHighAvailabilityWithQJM.md)), there +is a single Active NameNode and one or more Standby NameNode(s). The +Active NameNode is responsible for serving all client requests, while +Standby NameNode just keep the up-to-date information regarding the +namespace, by tailing edit logs from JournalNodes, as well as block +location information, by receiving block reports from all the DataNodes. +One drawback of this architecture is that the Active NameNode could be a +single bottle-neck and be overloaded with client requests, especially in +a busy cluster. + +The Consistent Reads from HDFS Observer NameNode feature addresses the +above by introducing a new type of NameNode called **Observer +NameNode**. Similar to Standby NameNode, Observer NameNode keeps itself +up-to-date regarding the namespace and block location information. +In addition, it also has the ability to serve consistent reads, like +Active NameNode. Since read requests are the majority in a typical +environment, this can help to load balancing the NameNode traffic and +improve overall throughput. + +Architecture +-- + +In the new architecture, a HA cluster could consists of namenodes in 3 +different states: active, standby and observer. State transition can +happen between active and standby, standby and observer, but not +directly between active and observer. + +To ensure read-after-write consistency within a single client, a state +ID, which is implemented using transaction ID within NameNode, is +introduced in RPC headers. When a client performs write through Active +NameNode, it updates its state ID using the latest transaction ID from +the NameNode. When performing a subsequent read, the client passes this +state ID to Observe NameNode, which will then check against its own +transaction ID, and will ensure its own transaction ID has caught up +with the request's state ID, before serving the read request. + +Edit log tailing is critical for Observer NameNode as it directly affects +the latency between when a transaction is applied in Active NameNode and +when it is applied in the Observer NameNode. A new edit log tailing +mechanism, named "Edit Tailing Fast-Path", is introduced to +significantly reduce this
