This is an automated email from the ASF dual-hosted git repository. cliang pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 6aa76ea YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) new 44d8b43 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. new ccad9ce HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. new c2585f7 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. new 849e1f0 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. new d7e0011 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. new d502e48 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. new eacf432 HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. new db9cec6 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. new f847983 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. new c02f4cc HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. new 13e8692 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. new ac0cdb0 HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. new 9976b7b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. new 0f811b0 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. new b41997c HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. new 60f0e4df HDFS-13767. Add msync server implementation. Contributed by Chen Liang. new ad80383 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. new be8beda HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. new 34b05a2 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. new 613c9e4 HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. new ea2d862 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. new 3273736 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. new 894f095 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. new 56af83c HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. new 28b11b0 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. new 7b425c8 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. new ca565ef HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. new 1189761 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. new 4ce7f9f HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. new 683daed HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. new 96cdd13 HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. new 6916ee7 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. new 54a1c66 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. new 8769e6f HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. new b8df864 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. new 82f68a4 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. new 7b1e3c4 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. new 2e7610a HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. new 8b8ec65 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. new 6e19a94 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. new 9aecfdc HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. new 451b8b8 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. new 9c1c73a HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. new 3806685 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. new 45cd309 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. new 2598282 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. new 16a4043 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. new 11fee2d HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. new 32d80d2 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. new b530301 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. The 50 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/hadoop/ha/FailoverController.java | 2 +- .../main/java/org/apache/hadoop/ha/HAAdmin.java | 43 ++ .../org/apache/hadoop/ha/HAServiceProtocol.java | 18 + .../apache/hadoop/ha/HAServiceProtocolHelper.java | 9 + .../java/org/apache/hadoop/ha/HAServiceTarget.java | 7 + .../HAServiceProtocolClientSideTranslatorPB.java | 16 + .../HAServiceProtocolServerSideTranslatorPB.java | 20 + .../hadoop/io/retry/RetryInvocationHandler.java | 5 + .../org/apache/hadoop/ipc/AlignmentContext.java | 96 +++++ .../main/java/org/apache/hadoop/ipc/Client.java | 40 +- .../java/org/apache/hadoop/ipc/ExternalCall.java | 3 +- .../hadoop/ipc/ObserverRetryOnActiveException.java | 37 ++ .../org/apache/hadoop/ipc/ProtobufRpcEngine.java | 24 +- .../src/main/java/org/apache/hadoop/ipc/RPC.java | 49 ++- .../main/java/org/apache/hadoop/ipc/RpcEngine.java | 8 +- .../main/java/org/apache/hadoop/ipc/Server.java | 147 ++++++- .../org/apache/hadoop/ipc/WritableRpcEngine.java | 47 ++- .../org/apache/hadoop/log/LogThrottlingHelper.java | 34 ++ .../java/org/apache/hadoop/util/ProtoUtil.java | 13 + .../src/main/proto/HAServiceProtocol.proto | 20 + .../hadoop-common/src/main/proto/RpcHeader.proto | 2 + .../hadoop-common/src/site/markdown/Metrics.md | 5 + .../java/org/apache/hadoop/ha/DummyHAService.java | 18 +- .../java/org/apache/hadoop/ha/MiniZKFCCluster.java | 4 + .../test/java/org/apache/hadoop/ipc/TestRPC.java | 8 +- .../org/apache/hadoop/hdfs/ClientGSIContext.java | 90 ++++ .../java/org/apache/hadoop/hdfs/DFSClient.java | 27 ++ .../apache/hadoop/hdfs/NameNodeProxiesClient.java | 23 +- .../hadoop/hdfs/protocol/ClientProtocol.java | 65 +++ .../ClientNamenodeProtocolTranslatorPB.java | 37 ++ .../ha/AbstractNNFailoverProxyProvider.java | 19 +- .../server/namenode/ha/ClientHAProxyFactory.java | 12 + .../namenode/ha/IPFailoverProxyProvider.java | 2 +- .../namenode/ha/ObserverReadProxyProvider.java | 470 +++++++++++++++++++++ .../ObserverReadProxyProviderWithIPFailover.java | 142 +++++++ .../hadoop/hdfs/server/namenode/ha/ReadOnly.java | 57 +++ .../src/main/proto/ClientNamenodeProtocol.proto | 18 + .../apache/hadoop/hdfs/protocol/TestReadOnly.java | 103 +++++ .../resolver/FederationNamenodeServiceState.java | 3 + .../server/federation/router/RouterRpcServer.java | 14 +- .../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 3 + ...ientNamenodeProtocolServerSideTranslatorPB.java | 49 +++ .../apache/hadoop/hdfs/protocolPB/PBHelper.java | 4 + .../hadoop/hdfs/qjournal/client/AsyncLogger.java | 7 + .../hdfs/qjournal/client/AsyncLoggerSet.java | 14 + .../hdfs/qjournal/client/IPCLoggerChannel.java | 14 + .../hdfs/qjournal/client/QuorumJournalManager.java | 137 +++++- .../hdfs/qjournal/client/QuorumOutputStream.java | 13 +- .../hdfs/qjournal/protocol/QJournalProtocol.java | 24 +- .../QJournalProtocolServerSideTranslatorPB.java | 14 + .../protocolPB/QJournalProtocolTranslatorPB.java | 20 + .../hadoop/hdfs/qjournal/server/Journal.java | 75 +++- .../hdfs/qjournal/server/JournalMetrics.java | 20 +- .../hdfs/qjournal/server/JournalNodeRpcServer.java | 8 + .../hdfs/qjournal/server/JournaledEditsCache.java | 417 ++++++++++++++++++ .../hdfs/server/common/HdfsServerConstants.java | 4 +- .../hdfs/server/datanode/BPServiceActor.java | 2 +- .../server/namenode/EditLogFileInputStream.java | 44 ++ .../hdfs/server/namenode/FSEditLogLoader.java | 47 ++- .../hadoop/hdfs/server/namenode/FSImage.java | 19 +- .../hadoop/hdfs/server/namenode/FSNamesystem.java | 31 +- .../hdfs/server/namenode/GlobalStateIdContext.java | 158 +++++++ .../hadoop/hdfs/server/namenode/NameNode.java | 41 +- .../hdfs/server/namenode/NameNodeRpcServer.java | 23 +- .../namenode/RedundantEditLogInputStream.java | 15 +- .../hdfs/server/namenode/ha/EditLogTailer.java | 8 +- .../hdfs/server/namenode/ha/StandbyState.java | 19 +- .../hadoop/hdfs/tools/NNHAServiceTarget.java | 5 + .../hadoop-hdfs/src/main/proto/HdfsServer.proto | 1 + .../src/main/proto/QJournalProtocol.proto | 18 + .../src/main/resources/hdfs-default.xml | 16 +- .../hadoop-hdfs/src/site/markdown/HDFSCommands.md | 2 + .../site/markdown/HDFSHighAvailabilityWithQJM.md | 42 +- .../src/site/markdown/ObserverNameNode.md | 231 ++++++++++ .../java/org/apache/hadoop/hdfs/DFSTestUtil.java | 12 +- .../org/apache/hadoop/hdfs/MiniDFSCluster.java | 16 +- .../hdfs/TestStateAlignmentContextWithHA.java | 347 +++++++++++++++ .../hadoop/hdfs/qjournal/MiniQJMHACluster.java | 4 +- .../qjournal/client/TestQuorumJournalManager.java | 143 +++++++ .../client/TestQuorumJournalManagerUnit.java | 103 ++++- .../hdfs/qjournal/server/JournalTestUtil.java | 48 +++ .../hadoop/hdfs/qjournal/server/TestJournal.java | 46 ++ .../qjournal/server/TestJournaledEditsCache.java | 257 +++++++++++ .../server/namenode/NNThroughputBenchmark.java | 6 +- .../hdfs/server/namenode/NameNodeAdapter.java | 13 + .../namenode/TestEditLogFileInputStream.java | 18 + .../hdfs/server/namenode/TestFSEditLogLoader.java | 48 +++ .../hadoop/hdfs/server/namenode/TestFsck.java | 2 +- .../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 143 ++++++- .../namenode/ha/TestConsistentReadsObserver.java | 385 +++++++++++++++++ .../namenode/ha/TestDelegationTokensWithHA.java | 46 ++ .../server/namenode/ha/TestMultiObserverNode.java | 159 +++++++ .../hdfs/server/namenode/ha/TestObserverNode.java | 364 ++++++++++++++++ .../namenode/ha/TestObserverReadProxyProvider.java | 403 ++++++++++++++++++ .../namenode/ha/TestStandbyInProgressTail.java | 167 ++++++-- .../apache/hadoop/hdfs/tools/TestDFSHAAdmin.java | 19 +- .../hdfs/tools/TestDFSHAAdminMiniCluster.java | 44 ++ .../yarn/server/resourcemanager/AdminService.java | 7 + 98 files changed, 5937 insertions(+), 165 deletions(-) create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java create mode 100644 hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ObserverRetryOnActiveException.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProvider.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ObserverReadProxyProviderWithIPFailover.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ReadOnly.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/protocol/TestReadOnly.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournaledEditsCache.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContextWithHA.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/JournalTestUtil.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournaledEditsCache.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestConsistentReadsObserver.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestMultiObserverNode.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java create mode 100644 hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverReadProxyProvider.java --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org