This is an automated email from the ASF dual-hosted git repository.
cliang pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 6aa76ea YARN-9668. UGI conf doesn't read user overridden
configurations on RM and NM startup. (Contributed by Jonanthan Hung)
new 44d8b43 HDFS-12975. [SBN read] Changes to the NameNode to support
reads from standby. Contributed by Chao Sun.
new ccad9ce HDFS-12977. [SBN read] Add stateId to RPC headers.
Contributed by Plamen Jeliazkov.
new c2585f7 HDFS-13331. [SBN read] Add lastSeenStateId to
RpcRequestHeader. Contributed by Plamen Jeliazkov.
new 849e1f0 HDFS-13286. [SBN read] Add haadmin commands to transition
between standby and observer. Contributed by Chao Sun.
new d7e0011 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in
ClientProtocol. Contributed by Chao Sun.
new d502e48 HDFS-13399. [SBN read] Make Client field AlignmentContext
non-static. Contributed by Plamen Jeliazkov.
new eacf432 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance
JournalNode with an in-memory cache of recent edit transactions. Contributed by
Erik Krogen.
new db9cec6 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add
ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
new f847983 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3:
NameNode-side changes to support tailing edits via RPC. Contributed by Erik
Krogen.
new c02f4cc HDFS-13706. [SBN read] Rename client context to
ClientGSIContext. Contributed by Konstantin Shvachko.
new 13e8692 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider.
Contributed by Chao Sun.
new ac0cdb0 HDFS-13665. [SBN read] Move RPC response serialization into
Server.doResponse(). Contributed by Plamen Jeliazkov.
new 9976b7b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup.
Integration test, documentation, remove unnecessary dummy sync, minors fixups.
Contributed by Erik Krogen.
new 0f811b0 HDFS-13688. [SBN read] Introduce msync API call. Contributed
by Chen Liang.
new b41997c HDFS-13789. Reduce logging frequency of
QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
new 60f0e4df HDFS-13767. Add msync server implementation. Contributed by
Chen Liang.
new ad80383 HDFS-13851. Remove AlignmentContext from
AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
new be8beda HDFS-13782. ObserverReadProxyProvider should work with
IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
new 34b05a2 HDFS-13779. [SBN read] Implement proper failover and observer
failure handling logic for for ObserverReadProxyProvider. Contributed by Erik
Krogen.
new 613c9e4 HDFS-13880. Add mechanism to allow certain RPC calls to
bypass sync. Contributed by Chen Liang.
new ea2d862 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should
use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider.
Contributed by Konstantin Shvachko and Plamen Jeliazkov.
new 3273736 HDFS-13749. [SBN read] Use getServiceStatus to discover
observer namenodes. Contributed by Chao Sun.
new 894f095 HDFS-13898. [SBN read] Throw retriable exception for
getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
new 56af83c HDFS-13791. Limit logging frequency of edit tail related
statements. Contributed by Erik Krogen.
new 28b11b0 HDFS-13961. [SBN read] TestObserverNode refactoring.
Contributed by Konstantin Shvachko.
new 7b425c8 HDFS-13523. Support observer nodes in MiniDFSCluster.
Contributed by Konstantin Shvachko.
new ca565ef HDFS-13925. Unit Test for transitioning between different
states. Contributed by Sherwood Zheng.
new 1189761 HDFS-13924. [SBN read] Handle BlockMissingException when
reading from observer. Contributed by Chao Sun.
new 4ce7f9f HDFS-14016. [SBN read] ObserverReadProxyProvider should
enable observer read by default. Contributed by Chen Liang.
new 683daed HDFS-14035. NN status discovery does not leverage delegation
token. Contributed by Chen Liang.
new 96cdd13 HDFS-14017. [SBN read]
ObserverReadProxyProviderWithIPFailover should work with HA configuration.
Contributed by Chen Liang.
new 6916ee7 HDFS-14067. [SBN read] Allow manual failover between standby
and observer. Contributed by Chao Sun.
new 54a1c66 HDFS-14094. [SBN read] Fix the order of logging arguments in
ObserverReadProxyProvider. Contributed by Ayush Saxena.
new 8769e6f HDFS-14120. [SBN read] ORFPP should also clone DT for the
virtual IP. Contributed by Chen Liang.
new b8df864 HDFS-14131. [SBN read] Create user guide for Consistent Reads
from Observer feature. Contributed by Chao Sun.
new 82f68a4 HDFS-14142. Move ipfailover config key out of
HdfsClientConfigKeys. Contributed by Chen Liang.
new 7b1e3c4 HDFS-13873. [SBN read] ObserverNode should reject read
requests when it is too far behind. Contributed by Konstantin Shvachko.
new 2e7610a HDFS-14138. [SBN read] Description errors in the comparison
logic of transaction ID. Contributed by xiangheng.
new 8b8ec65 HDFS-14146. [SBN read] Handle exceptions from and prevent
handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
new 6e19a94 HDFS-14116. [SBN read] Fix class cast error in
NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
new 9aecfdc HDFS-14149. [SBN read] Fix annotations on new
interfaces/classes for SBN reads. Contributed by Chao Sun.
new 451b8b8 HDFS-14160. [SBN read] ObserverReadInvocationHandler should
implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
new 9c1c73a HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in
user guide. Contributed by Chao Sun.
new 3806685 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN
reads. Contributed by Konstantin V Shvachko.
new 45cd309 HDFS-14250. [SBN read]. msync should always direct to active
NameNode to get latest stateID. Contributed by Chao Sun.
new 2598282 HDFS-14279. [SBN read] Fix race condition in
ObserverReadProxyProvider. Contributed by Erik Krogen.
new 16a4043 HDFS-14272. [SBN read] Make ObserverReadProxyProvider
initialize its state ID against the active NN on startup. Contributed by Erik
Krogen.
new 11fee2d HDFS-14211. [SBN Read]. Add a configurable flag to enable
always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
new 32d80d2 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to
gracefully handle StandbyException when fetching HAServiceState. Contributed by
Erik Krogen.
new b530301 HDFS-14537. Journaled Edits Cache is not cleared when
formatting the JN. Contributed by Ranith Sardar.
The 50 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/ha/FailoverController.java | 2 +-
.../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++
.../org/apache/hadoop/ha/HAServiceProtocol.java | 18 +
.../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 +
.../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 +
.../HAServiceProtocolClientSideTranslatorPB.java | 16 +
.../HAServiceProtocolServerSideTranslatorPB.java | 20 +
.../hadoop/io/retry/RetryInvocationHandler.java | 5 +
.../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++
.../main/java/org/apache/hadoop/ipc/Client.java | 40 +-
.../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +-
.../hadoop/ipc/ObserverRetryOnActiveException.java | 37 ++
.../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +-
.../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++-
.../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +-
.../main/java/org/apache/hadoop/ipc/Server.java | 147 ++++++-
.../org/apache/hadoop/ipc/WritableRpcEngine.java | 47 ++-
.../org/apache/hadoop/log/LogThrottlingHelper.java | 34 ++
.../java/org/apache/hadoop/util/ProtoUtil.java | 13 +
.../src/main/proto/HAServiceProtocol.proto | 20 +
.../hadoop-common/src/main/proto/RpcHeader.proto | 2 +
.../hadoop-common/src/site/markdown/Metrics.md | 5 +
.../java/org/apache/hadoop/ha/DummyHAService.java | 18 +-
.../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 +
.../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +-
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 ++++
.../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++
.../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +-
.../hadoop/hdfs/protocol/ClientProtocol.java | 65 +++
.../ClientNamenodeProtocolTranslatorPB.java | 37 ++
.../ha/AbstractNNFailoverProxyProvider.java | 19 +-
.../server/namenode/ha/ClientHAProxyFactory.java | 12 +
.../namenode/ha/IPFailoverProxyProvider.java | 2 +-
.../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++
.../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++
.../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++
.../src/main/proto/ClientNamenodeProtocol.proto | 18 +
.../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 +++++
.../resolver/FederationNamenodeServiceState.java | 3 +
.../server/federation/router/RouterRpcServer.java | 14 +-
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 +
...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++
.../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 +
.../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 +
.../hdfs/qjournal/client/AsyncLoggerSet.java | 14 +
.../hdfs/qjournal/client/IPCLoggerChannel.java | 14 +
.../hdfs/qjournal/client/QuorumJournalManager.java | 137 +++++-
.../hdfs/qjournal/client/QuorumOutputStream.java | 13 +-
.../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +-
.../QJournalProtocolServerSideTranslatorPB.java | 14 +
.../protocolPB/QJournalProtocolTranslatorPB.java | 20 +
.../hadoop/hdfs/qjournal/server/Journal.java | 75 +++-
.../hdfs/qjournal/server/JournalMetrics.java | 20 +-
.../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 +
.../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++
.../hdfs/server/common/HdfsServerConstants.java | 4 +-
.../hdfs/server/datanode/BPServiceActor.java | 2 +-
.../server/namenode/EditLogFileInputStream.java | 44 ++
.../hdfs/server/namenode/FSEditLogLoader.java | 47 ++-
.../hadoop/hdfs/server/namenode/FSImage.java | 19 +-
.../hadoop/hdfs/server/namenode/FSNamesystem.java | 31 +-
.../hdfs/server/namenode/GlobalStateIdContext.java | 158 +++++++
.../hadoop/hdfs/server/namenode/NameNode.java | 41 +-
.../hdfs/server/namenode/NameNodeRpcServer.java | 23 +-
.../namenode/RedundantEditLogInputStream.java | 15 +-
.../hdfs/server/namenode/ha/EditLogTailer.java | 8 +-
.../hdfs/server/namenode/ha/StandbyState.java | 19 +-
.../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 +
.../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 +
.../src/main/proto/QJournalProtocol.proto | 18 +
.../src/main/resources/hdfs-default.xml | 16 +-
.../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 +
.../site/markdown/HDFSHighAvailabilityWithQJM.md | 42 +-
.../src/site/markdown/ObserverNameNode.md | 231 ++++++++++
.../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +-
.../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +-
.../hdfs/TestStateAlignmentContextWithHA.java | 347 +++++++++++++++
.../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +-
.../qjournal/client/TestQuorumJournalManager.java | 143 +++++++
.../client/TestQuorumJournalManagerUnit.java | 103 ++++-
.../hdfs/qjournal/server/JournalTestUtil.java | 48 +++
.../hadoop/hdfs/qjournal/server/TestJournal.java | 46 ++
.../qjournal/server/TestJournaledEditsCache.java | 257 +++++++++++
.../server/namenode/NNThroughputBenchmark.java | 6 +-
.../hdfs/server/namenode/NameNodeAdapter.java | 13 +
.../namenode/TestEditLogFileInputStream.java | 18 +
.../hdfs/server/namenode/TestFSEditLogLoader.java | 48 +++
.../hadoop/hdfs/server/namenode/TestFsck.java | 2 +-
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 143 ++++++-
.../namenode/ha/TestConsistentReadsObserver.java | 385 +++++++++++++++++
.../namenode/ha/TestDelegationTokensWithHA.java | 46 ++
.../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++
.../hdfs/server/namenode/ha/TestObserverNode.java | 364 ++++++++++++++++
.../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++
.../namenode/ha/TestStandbyInProgressTail.java | 167 ++++++--
.../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +-
.../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++
.../yarn/server/resourcemanager/AdminService.java | 7 +
98 files changed, 5937 insertions(+), 165 deletions(-)
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java
create mode 100644
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
create mode 100644
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
