This is an automated email from the ASF dual-hosted git repository.
cliang pushed a change to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 860606f YARN-9581. Add support for get multiple RM webapp URLs.
Contributed by Prabhu Joseph
new 3932ac4 HDFS-12975. [SBN read] Changes to the NameNode to support
reads from standby. Contributed by Chao Sun.
new 1eeca2d HDFS-12977. [SBN read] Add stateId to RPC headers.
Contributed by Plamen Jeliazkov.
new c0ca2bb HDFS-13331. [SBN read] Add lastSeenStateId to
RpcRequestHeader. Contributed by Plamen Jeliazkov.
new 49d7bb6 HDFS-13286. [SBN read] Add haadmin commands to transition
between standby and observer. Contributed by Chao Sun.
new dbf777e HDFS-13578. [SBN read] Add ReadOnly annotation to methods in
ClientProtocol. Contributed by Chao Sun.
new a3521c5 HDFS-13399. [SBN read] Make Client field AlignmentContext
non-static. Contributed by Plamen Jeliazkov.
new 174c41c HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance
JournalNode with an in-memory cache of recent edit transactions. Contributed by
Erik Krogen.
new b13110a HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add
ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
new ac3eeee HDFS-13609. [SBN read] Edit Tail Fast Path Part 3:
NameNode-side changes to support tailing edits via RPC. Contributed by Erik
Krogen.
new 1f5852f HDFS-13706. [SBN read] Rename client context to
ClientGSIContext. Contributed by Konstantin Shvachko.
new a97f00e HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider.
Contributed by Chao Sun.
new 7e0a71d HDFS-13665. [SBN read] Move RPC response serialization into
Server.doResponse(). Contributed by Plamen Jeliazkov.
new ea40285 HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup.
Integration test, documentation, remove unnecessary dummy sync, minors fixups.
Contributed by Erik Krogen.
new e3f3fb1 HDFS-13688. [SBN read] Introduce msync API call. Contributed
by Chen Liang.
new 6fe755f HDFS-13789. Reduce logging frequency of
QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
new e2337bf HDFS-13767. Add msync server implementation. Contributed by
Chen Liang.
new 0c9d846 HDFS-13851. Remove AlignmentContext from
AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 0b56a53 HDFS-13782. ObserverReadProxyProvider should work with
IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
new afeaead HDFS-13779. [SBN read] Implement proper failover and observer
failure handling logic for for ObserverReadProxyProvider. Contributed by Erik
Krogen.
new 4c4ca86 HDFS-13880. Add mechanism to allow certain RPC calls to
bypass sync. Contributed by Chen Liang.
new 0de06c0 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should
use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider.
Contributed by Konstantin Shvachko and Plamen Jeliazkov.
new 556b7a8 HDFS-13749. [SBN read] Use getServiceStatus to discover
observer namenodes. Contributed by Chao Sun.
new a856eea HDFS-13898. [SBN read] Throw retriable exception for
getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
new 3998bab HDFS-13791. Limit logging frequency of edit tail related
statements. Contributed by Erik Krogen.
new c2bfaf4 HDFS-13961. [SBN read] TestObserverNode refactoring.
Contributed by Konstantin Shvachko.
new dea7e1a HDFS-13523. Support observer nodes in MiniDFSCluster.
Contributed by Konstantin Shvachko.
new 8eef380 HDFS-13925. Unit Test for transitioning between different
states. Contributed by Sherwood Zheng.
new cc70e81 HDFS-13924. [SBN read] Handle BlockMissingException when
reading from observer. Contributed by Chao Sun.
new fc54850 HDFS-14016. [SBN read] ObserverReadProxyProvider should
enable observer read by default. Contributed by Chen Liang.
new 3d3686a HDFS-14035. NN status discovery does not leverage delegation
token. Contributed by Chen Liang.
new 03a2c60 HDFS-14017. [SBN read]
ObserverReadProxyProviderWithIPFailover should work with HA configuration.
Contributed by Chen Liang.
new 45d6834 HDFS-14067. [SBN read] Allow manual failover between standby
and observer. Contributed by Chao Sun.
new f0b7350 HDFS-14094. [SBN read] Fix the order of logging arguments in
ObserverReadProxyProvider. Contributed by Ayush Saxena.
new f7f7cac HDFS-14120. [SBN read] ORFPP should also clone DT for the
virtual IP. Contributed by Chen Liang.
new f1c5b24 HDFS-14131. [SBN read] Create user guide for Consistent Reads
from Observer feature. Contributed by Chao Sun.
new 589d2c5 HDFS-14142. Move ipfailover config key out of
HdfsClientConfigKeys. Contributed by Chen Liang.
new c1c061d HDFS-13873. [SBN read] ObserverNode should reject read
requests when it is too far behind. Contributed by Konstantin Shvachko.
new 7908318 HDFS-14138. [SBN read] Description errors in the comparison
logic of transaction ID. Contributed by xiangheng.
new e917ac2 HDFS-14146. [SBN read] Handle exceptions from and prevent
handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
new 25913f1 HDFS-14116. [SBN read] Fix class cast error in
NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
new eb7d7fa HDFS-14149. [SBN read] Fix annotations on new
interfaces/classes for SBN reads. Contributed by Chao Sun.
new 2be93bd HDFS-14160. [SBN read] ObserverReadInvocationHandler should
implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
new 720eed9 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in
user guide. Contributed by Chao Sun.
new c936562 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN
reads. Contributed by Konstantin V Shvachko.
new 73d56b9 HDFS-14250. [SBN read]. msync should always direct to active
NameNode to get latest stateID. Contributed by Chao Sun.
new 98434be HDFS-14279. [SBN read] Fix race condition in
ObserverReadProxyProvider. Contributed by Erik Krogen.
new e1e4056 HDFS-14272. [SBN read] Make ObserverReadProxyProvider
initialize its state ID against the active NN on startup. Contributed by Erik
Krogen.
new 3c6b74a HDFS-14211. [SBN Read]. Add a configurable flag to enable
always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
new e53cee0 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to
gracefully handle StandbyException when fetching HAServiceState. Contributed by
Erik Krogen.
new 66920e6 HDFS-14537. Journaled Edits Cache is not cleared when
formatting the JN. Contributed by Ranith Sardar.
The 50 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/ha/FailoverController.java | 2 +-
.../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++
.../org/apache/hadoop/ha/HAServiceProtocol.java | 18 +
.../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 +
.../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 +
.../HAServiceProtocolClientSideTranslatorPB.java | 16 +
.../HAServiceProtocolServerSideTranslatorPB.java | 20 +
.../hadoop/io/retry/RetryInvocationHandler.java | 5 +
.../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++
.../main/java/org/apache/hadoop/ipc/Client.java | 40 +-
.../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +-
.../hadoop/ipc/ObserverRetryOnActiveException.java | 37 ++
.../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +-
.../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++-
.../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 137 +++++-
.../org/apache/hadoop/ipc/WritableRpcEngine.java | 47 ++-
.../org/apache/hadoop/log/LogThrottlingHelper.java | 34 ++
.../java/org/apache/hadoop/util/ProtoUtil.java | 13 +
.../src/main/proto/HAServiceProtocol.proto | 20 +
.../hadoop-common/src/main/proto/RpcHeader.proto | 2 +
.../hadoop-common/src/site/markdown/Metrics.md | 5 +
.../java/org/apache/hadoop/ha/DummyHAService.java | 18 +-
.../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 +
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +-
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 ++++
.../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++
.../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +-
.../hadoop/hdfs/protocol/ClientProtocol.java | 67 +++
.../ClientNamenodeProtocolTranslatorPB.java | 39 ++
.../ha/AbstractNNFailoverProxyProvider.java | 19 +-
.../server/namenode/ha/ClientHAProxyFactory.java | 12 +
.../namenode/ha/IPFailoverProxyProvider.java | 2 +-
.../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++
.../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++
.../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++
.../src/main/proto/ClientNamenodeProtocol.proto | 18 +
.../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 +++++
.../main/native/libhdfspp/lib/proto/CMakeLists.txt | 1 +
.../resolver/FederationNamenodeServiceState.java | 3 +
.../federation/router/RouterClientProtocol.java | 13 +
.../server/federation/router/RouterRpcServer.java | 12 +
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 +
...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 +
.../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 +
.../hdfs/qjournal/client/AsyncLoggerSet.java | 14 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 137 +++++-
.../hdfs/qjournal/client/QuorumOutputStream.java | 13 +-
.../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +-
.../QJournalProtocolServerSideTranslatorPB.java | 14 +
.../protocolPB/QJournalProtocolTranslatorPB.java | 20 +
.../hadoop/hdfs/qjournal/server/Journal.java | 75 +++-
.../hdfs/qjournal/server/JournalMetrics.java | 20 +-
.../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 +
.../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++
.../hdfs/server/common/HdfsServerConstants.java | 4 +-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../server/namenode/EditLogFileInputStream.java | 44 ++
.../hdfs/server/namenode/FSEditLogLoader.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSImage.java | 19 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 31 +-
.../hdfs/server/namenode/GlobalStateIdContext.java | 158 +++++++
.../hadoop/hdfs/server/namenode/NameNode.java | 41 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 23 +-
.../namenode/RedundantEditLogInputStream.java | 15 +-
.../hdfs/server/namenode/ha/EditLogTailer.java | 8 +-
.../hdfs/server/namenode/ha/StandbyState.java | 19 +-
.../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 +
.../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 +
.../src/main/proto/QJournalProtocol.proto | 18 +
.../src/main/resources/hdfs-default.xml | 16 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../site/markdown/HDFSHighAvailabilityWithQJM.md | 35 +-
.../src/site/markdown/ObserverNameNode.md | 231 ++++++++++
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +-
.../hdfs/TestStateAlignmentContextWithHA.java | 347 +++++++++++++++
.../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +-
.../qjournal/client/TestQuorumJournalManager.java | 134 ++++++
.../client/TestQuorumJournalManagerUnit.java | 103 ++++-
.../hdfs/qjournal/server/JournalTestUtil.java | 48 +++
.../hadoop/hdfs/qjournal/server/TestJournal.java | 47 +++
.../qjournal/server/TestJournaledEditsCache.java | 257 +++++++++++
.../server/namenode/NNThroughputBenchmark.java | 6 +-
.../hdfs/server/namenode/NameNodeAdapter.java | 13 +
.../namenode/TestEditLogFileInputStream.java | 18 +
.../hdfs/server/namenode/TestFSEditLogLoader.java | 48 +++
.../hadoop/hdfs/server/namenode/TestFsck.java | 2 +-
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 144 ++++++-
.../namenode/ha/TestConsistentReadsObserver.java | 385 +++++++++++++++++
.../namenode/ha/TestDelegationTokensWithHA.java | 46 ++
.../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++
.../hdfs/server/namenode/ha/TestObserverNode.java | 364 ++++++++++++++++
.../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++
.../namenode/ha/TestStandbyInProgressTail.java | 164 +++++--
.../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +-
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++
.../yarn/server/resourcemanager/AdminService.java | 7 +
100 files changed, 5928 insertions(+), 163 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
