This is an automated email from the ASF dual-hosted git repository.
cliang pushed a change to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 773591e YARN-9626. UI2 - Fair scheduler queue apps page issues.
Contributed by Zoltan Siegl
new 4cf6390 HDFS-12975. [SBN read] Changes to the NameNode to support
reads from standby. Contributed by Chao Sun.
new 21a886a HDFS-12977. [SBN read] Add stateId to RPC headers.
Contributed by Plamen Jeliazkov.
new b999e96 HDFS-13331. [SBN read] Add lastSeenStateId to
RpcRequestHeader. Contributed by Plamen Jeliazkov.
new 07a6510 HDFS-13286. [SBN read] Add haadmin commands to transition
between standby and observer. Contributed by Chao Sun.
new 18e545f HDFS-13578. [SBN read] Add ReadOnly annotation to methods in
ClientProtocol. Contributed by Chao Sun.
new 7d0c3ba HDFS-13399. [SBN read] Make Client field AlignmentContext
non-static. Contributed by Plamen Jeliazkov.
new c40ddf9 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance
JournalNode with an in-memory cache of recent edit transactions. Contributed by
Erik Krogen.
new abb8614 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add
ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
new 8e55170 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3:
NameNode-side changes to support tailing edits via RPC. Contributed by Erik
Krogen.
new 405f6b6 HDFS-13706. [SBN read] Rename client context to
ClientGSIContext. Contributed by Konstantin Shvachko.
new 9840dae HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider.
Contributed by Chao Sun.
new cfb4399 HDFS-13665. [SBN read] Move RPC response serialization into
Server.doResponse(). Contributed by Plamen Jeliazkov.
new ac2e793 HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup.
Integration test, documentation, remove unnecessary dummy sync, minors fixups.
Contributed by Erik Krogen.
new 849a6c6 HDFS-13688. [SBN read] Introduce msync API call. Contributed
by Chen Liang.
new d96ddb3 HDFS-13789. Reduce logging frequency of
QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
new ee9977c HDFS-13767. Add msync server implementation. Contributed by
Chen Liang.
new 1ad1c80 HDFS-13851. Remove AlignmentContext from
AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 1be8e18 HDFS-13782. ObserverReadProxyProvider should work with
IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 504cb70 HDFS-13779. [SBN read] Implement proper failover and observer
failure handling logic for for ObserverReadProxyProvider. Contributed by Erik
Krogen.
new 8ec17f6 HDFS-13880. Add mechanism to allow certain RPC calls to
bypass sync. Contributed by Chen Liang.
new e639297 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should
use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider.
Contributed by Konstantin Shvachko and Plamen Jeliazkov.
new 0755862 HDFS-13749. [SBN read] Use getServiceStatus to discover
observer namenodes. Contributed by Chao Sun.
new d4adf92 HDFS-13898. [SBN read] Throw retriable exception for
getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
new 91803c8 HDFS-13791. Limit logging frequency of edit tail related
statements. Contributed by Erik Krogen.
new a054c65 HDFS-13961. [SBN read] TestObserverNode refactoring.
Contributed by Konstantin Shvachko.
new e860882 HDFS-13523. Support observer nodes in MiniDFSCluster.
Contributed by Konstantin Shvachko.
new 66db260 HDFS-13925. Unit Test for transitioning between different
states. Contributed by Sherwood Zheng.
new 8100dcb HDFS-13924. [SBN read] Handle BlockMissingException when
reading from observer. Contributed by Chao Sun.
new e0436b3 HDFS-14016. [SBN read] ObserverReadProxyProvider should
enable observer read by default. Contributed by Chen Liang.
new 53a88bf HDFS-14035. NN status discovery does not leverage delegation
token. Contributed by Chen Liang.
new f5c923b HDFS-14017. [SBN read]
ObserverReadProxyProviderWithIPFailover should work with HA configuration.
Contributed by Chen Liang.
new c4d944a HDFS-14067. [SBN read] Allow manual failover between standby
and observer. Contributed by Chao Sun.
new 6cb1b5a HDFS-14094. [SBN read] Fix the order of logging arguments in
ObserverReadProxyProvider. Contributed by Ayush Saxena.
new 6e54d97 HDFS-14120. [SBN read] ORFPP should also clone DT for the
virtual IP. Contributed by Chen Liang.
new 265bd68 HDFS-14131. [SBN read] Create user guide for Consistent Reads
from Observer feature. Contributed by Chao Sun.
new 8e77349 HDFS-14142. Move ipfailover config key out of
HdfsClientConfigKeys. Contributed by Chen Liang.
new 67c4768 HDFS-13873. [SBN read] ObserverNode should reject read
requests when it is too far behind. Contributed by Konstantin Shvachko.
new c23755a HDFS-14138. [SBN read] Description errors in the comparison
logic of transaction ID. Contributed by xiangheng.
new 6c1bf61 HDFS-14146. [SBN read] Handle exceptions from and prevent
handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
new 7a650b0 HDFS-14116. [SBN read] Fix class cast error in
NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
new 8f36bdc HDFS-14149. [SBN read] Fix annotations on new
interfaces/classes for SBN reads. Contributed by Chao Sun.
new b8f9739 HDFS-14160. [SBN read] ObserverReadInvocationHandler should
implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
new b028d62 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in
user guide. Contributed by Chao Sun.
new 321e26c HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN
reads. Contributed by Konstantin V Shvachko.
new ee0416b HDFS-14250. [SBN read]. msync should always direct to active
NameNode to get latest stateID. Contributed by Chao Sun.
new 98684fb HDFS-14279. [SBN read] Fix race condition in
ObserverReadProxyProvider. Contributed by Erik Krogen.
new 35200b3 HDFS-14272. [SBN read] Make ObserverReadProxyProvider
initialize its state ID against the active NN on startup. Contributed by Erik
Krogen.
new 769b5a8 HDFS-14211. [SBN Read]. Add a configurable flag to enable
always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
new cc22373 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to
gracefully handle StandbyException when fetching HAServiceState. Contributed by
Erik Krogen.
new 3ba7826 HDFS-14537. Journaled Edits Cache is not cleared when
formatting the JN. Contributed by Ranith Sardar.
new 2a1451a HDFS-13791. Amend fix LOG string format.
The 51 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/ha/FailoverController.java | 2 +-
.../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++
.../org/apache/hadoop/ha/HAServiceProtocol.java | 18 +
.../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 +
.../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 +
.../HAServiceProtocolClientSideTranslatorPB.java | 16 +
.../HAServiceProtocolServerSideTranslatorPB.java | 20 +
.../hadoop/io/retry/RetryInvocationHandler.java | 5 +
.../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++
.../main/java/org/apache/hadoop/ipc/Client.java | 40 +-
.../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +-
.../hadoop/ipc/ObserverRetryOnActiveException.java | 37 ++
.../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +-
.../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++-
.../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 147 ++++++-
.../org/apache/hadoop/ipc/WritableRpcEngine.java | 47 ++-
.../org/apache/hadoop/log/LogThrottlingHelper.java | 34 ++
.../java/org/apache/hadoop/util/ProtoUtil.java | 13 +
.../src/main/proto/HAServiceProtocol.proto | 20 +
.../hadoop-common/src/main/proto/RpcHeader.proto | 2 +
.../hadoop-common/src/site/markdown/Metrics.md | 5 +
.../java/org/apache/hadoop/ha/DummyHAService.java | 18 +-
.../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 +
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +-
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 ++++
.../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++
.../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +-
.../hadoop/hdfs/protocol/ClientProtocol.java | 67 +++
.../ClientNamenodeProtocolTranslatorPB.java | 37 ++
.../ha/AbstractNNFailoverProxyProvider.java | 19 +-
.../server/namenode/ha/ClientHAProxyFactory.java | 12 +
.../namenode/ha/IPFailoverProxyProvider.java | 2 +-
.../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++
.../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++
.../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++
.../src/main/proto/ClientNamenodeProtocol.proto | 18 +
.../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 +++++
.../resolver/FederationNamenodeServiceState.java | 3 +
.../federation/router/RouterClientProtocol.java | 13 +
.../server/federation/router/RouterRpcServer.java | 12 +
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 +
...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 +
.../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 +
.../hdfs/qjournal/client/AsyncLoggerSet.java | 14 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 137 +++++-
.../hdfs/qjournal/client/QuorumOutputStream.java | 13 +-
.../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +-
.../QJournalProtocolServerSideTranslatorPB.java | 14 +
.../protocolPB/QJournalProtocolTranslatorPB.java | 20 +
.../hadoop/hdfs/qjournal/server/Journal.java | 75 +++-
.../hdfs/qjournal/server/JournalMetrics.java | 20 +-
.../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 +
.../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++
.../hdfs/server/common/HdfsServerConstants.java | 4 +-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../server/namenode/EditLogFileInputStream.java | 44 ++
.../hdfs/server/namenode/FSEditLogLoader.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSImage.java | 19 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 31 +-
.../hdfs/server/namenode/GlobalStateIdContext.java | 158 +++++++
.../hadoop/hdfs/server/namenode/NameNode.java | 41 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 23 +-
.../namenode/RedundantEditLogInputStream.java | 15 +-
.../hdfs/server/namenode/ha/EditLogTailer.java | 8 +-
.../hdfs/server/namenode/ha/StandbyState.java | 19 +-
.../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 +
.../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 +
.../src/main/proto/QJournalProtocol.proto | 18 +
.../src/main/resources/hdfs-default.xml | 16 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../site/markdown/HDFSHighAvailabilityWithQJM.md | 42 +-
.../src/site/markdown/ObserverNameNode.md | 231 ++++++++++
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +-
.../hdfs/TestStateAlignmentContextWithHA.java | 347 +++++++++++++++
.../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +-
.../qjournal/client/TestQuorumJournalManager.java | 134 ++++++
.../client/TestQuorumJournalManagerUnit.java | 103 ++++-
.../hdfs/qjournal/server/JournalTestUtil.java | 48 +++
.../hadoop/hdfs/qjournal/server/TestJournal.java | 46 ++
.../qjournal/server/TestJournaledEditsCache.java | 257 +++++++++++
.../server/namenode/NNThroughputBenchmark.java | 6 +-
.../hdfs/server/namenode/NameNodeAdapter.java | 13 +
.../namenode/TestEditLogFileInputStream.java | 18 +
.../hdfs/server/namenode/TestFSEditLogLoader.java | 48 +++
.../hadoop/hdfs/server/namenode/TestFsck.java | 2 +-
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 143 ++++++-
.../namenode/ha/TestConsistentReadsObserver.java | 385 +++++++++++++++++
.../namenode/ha/TestDelegationTokensWithHA.java | 46 ++
.../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++
.../hdfs/server/namenode/ha/TestObserverNode.java | 364 ++++++++++++++++
.../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++
.../namenode/ha/TestStandbyInProgressTail.java | 164 +++++--
.../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +-
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++
.../yarn/server/resourcemanager/AdminService.java | 7 +
99 files changed, 5940 insertions(+), 163 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
