This is an automated email from the ASF dual-hosted git repository.
avijayan pushed a change to branch HDDS-3698-nonrolling-upgrade
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from cdb1471 HDDS-5170. Race condition in NodestateManager#addNode allows
datanodes with lower MLV to be used in pipelines. (#2204)
add abf38f4 HDDS-4988. Cancel duplicate PR workflows (#2053)
add a170a28 HDDS-5071. Fix duration of sub-ca certs. (#2127)
add 6b57765 HDDS-5052. [SCM HA Security] Handle leader changes between
SCMInfo and getSCMSigned Cert in OM/SCM. (#2100)
add 4f76d03 HDDS-4969. Extract check dependency installation from Github
Actions workflow (#2029)
add 9c92855 HDDS-4840. Make datanode db profile configurable with
existing hdds.d… (#1955)
add 9de5d0f HDDS-5057. Remove backward direction dependency between
HDDS->Ozone (#2106)
add 1a7745e HDDS-5004. Upgrade Jersey2 dependency (#2065)
add b83c7b6 HDDS-5038. Create and update ZH translation of SCM-HA.md and
OM-HA.md in doc (#2092)
add ccd046e HDDS-5077. Include dev-support in ozone source distribution
(#2134)
add 8ea5be9 HDDS-5050. Add retry policy for ratis requests in SCM HA.
(#2116)
add c8594d1 HDDS-5053. RocksDB block cache capacity is wrongly configured
(#2137)
add fd308b7 HDDS-4980. Bump jaeger-client to 1.6.0 (#2045)
add aa44905 HDDS-5074. Bump Guava version (#2131)
add f1be299 HDDS-5098. Disable intermittent
TestSCMNodeManager#testScmDetectStaleAndDeadNode
add 5bebe62 HDDS-4842. Add timestamp to Revoked Certs table in SCM DB
(#2145)
add 89a2b52 HDDS-5075. NPE during secure SCM initialization with HA code
updated to an already existing cluster. (#2146)
add 3a09f6b HDDS-4642. SCM security protocol support for query CRLs and
latest CRL id for OM and Datanode. (#2030)
add 54ea4c7 HDDS-5080. enable s3 test suite for secure-ha (#2147)
add e8d001f HDDS-4888. Merge basic CI checks (#1984)
add 8a80c80 HDDS-5090. make Decommission work under SCM HA. (#2148)
add 6da41ca HDDS-5110. TestStorageContainerManagerHttpServer fails in CI
(#2159)
add 014d0a2 HDDS-5075. [SCM HA Security] Remove code of not starting
ozone services when Security is enabled on SCM HA cluster (#2132)
add 8badca9 HDDS-5084. Include HISTORY.md/SECURITY.md/CONTRIBUTING.md in
the release artifacts. (#2140)
add 7afb1f8 HDDS-5091. Add project separation and first stable release to
the HISTORY.md (#2149)
add 724d96c HDDS-5026. Cancel failing PR workflow runs (#2129)
add cda430a HDDS-5087. Ozone RPC client leaks KeyProvider instances.
(#2144)
add 8036536 HDDS-5117. CRLInfo should include CRL Sequence ID (#2161)
add 664837b HDDS-5121. Releasing 1.1.0 from "ozone-1.1" branch. Updating
version on Master.
add 24ecd22 HDDS-5121. Remove duplicate SNAPSHOT from version (#2163)
add 8cdabec HDDS-5116. Secure datanode/OM may exit if it cannot connect
to SCM. (#2162)
add 1761426 HDDS-4986. Read failure because of unhealthy container.
(#2152)
add 057f0a0 HDDS-5061. Improve block commit (#2108)
add c89f08e HDDS-5068. build errors due to missing JAXB-runtime jar
dependency for openjdk 11 se standalone (#2153)
add 9912b69 HDDS-4195. Fix Recon after HDDS-4133. (#2103)
add f2be003 HDDS-5125. Only test ozonesecure with SCM Ratis disabled
(#2168)
add 4e45726 HDDS-5122. SCM Reinitialization can end up leaking Ratis
Segmented RaftLogWorker threads (#2167)
add f2ff682 HDDS-5083. Bump version of common-compress (#2139)
add fb3dee8 HDDS-5047. Refactor Pipeline to use ReplicationConfig instead
of factor/type (#2096)
add 2c183b2 HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT"
(#2166)
add bef180e HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine.
(#2155)
add 4c313b8 HDDS-5131. Use timeout in github actions (#2176)
add 6af26ca HDDS-4889. Add simple CI check for docs (#2156)
add 7f2dc04 HDDS-5127. Fix getServiceList when SCM HA is enabled (#2173)
add 1e1eefa HDDS-5051. Ensure failover to suggested leader if any for
NotLeaderException. (#2141)
add 6f9c3f2 HDDS-5060. [SCM HA Security] Make InterSCM grpc channel
secure. (#2187)
add bddc873 HDDS-4515. Datanodes should be able to persist and load CRL
(#2181)
add ce29843 HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT
(#2184)
add bfb6be8 HDDS-5152. Fix Suggested leader in Client. (#2189)
add bccc439 HDDS-5147. Intermittent test failure in
TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (#2188)
add a920f25 HDDS-5153. Decommissioning a dead node should complete
immediately (#2190)
add 66a411b HDDS-4585. Support bucket acl operation in S3g (#1701)
add 5dd0943 HDDS-5144. Create github check to alert when dependency tree
is changed (#2177)
add 10312fb HDDS-4983. Display key offset for each block in command key
info (#2051)
add 799c38e Revert "HDDS-5144. Create github check to alert when
dependency tree is changed (#2177)"
add dcac0f1 HDDS-5166. Remove duplicate assignment of OZONE_OPTS for
freon and sh (#2195)
add 48b84c5 HDDS-2212. Genconf tool should generate config files for
secure clust… (#1788)
new 253cc55 Merge remote-tracking branch 'upstream/master' into
HDDS-3698-nonrolling-upgrade-master-merge2
new 1686e56 Fix compilation errors afte merge Update javassist in recon
pom Fix changes introduced in merge that failed TestSCMNodeManager upgrade
tests Fix checkstyle Fix intermittent test failure
TestSCMNodeManager#testSetNodeOpStateAndCommandFired after merge Skip scm init
default layout version in TestOzoneConfigurationFields
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../{close-pending.yaml => cancel-ci.yaml} | 33 +-
.github/workflows/post-commit.yml | 181 ++---
.gitignore | 2 +
HISTORY.md | 4 +-
hadoop-hdds/client/pom.xml | 4 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 7 +
.../hadoop/hdds/scm/storage/BlockInputStream.java | 7 +-
.../storage/DummyBlockInputStreamWithRetry.java | 7 +-
hadoop-hdds/common/pom.xml | 12 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 3 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 104 ++-
.../hdds/ratis/ServerNotLeaderException.java | 113 +++
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 -
.../hadoop/hdds/scm/ha/NonRetriableException.java} | 13 +-
.../scm/ha/RetriableWithNoFailoverException.java} | 11 +-
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 129 +++-
.../org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java | 5 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 67 +-
.../hadoop/hdds/security/x509/SecurityConfig.java | 13 +-
.../hadoop/hdds/security/x509/crl}/CRLCodec.java | 2 +-
.../hadoop/hdds/security/x509/crl/CRLInfo.java | 25 +-
.../hdds/security/x509/crl}/CRLInfoCodec.java | 3 +-
.../hdds/security/x509/crl}/package-info.java | 9 +-
.../org/apache/hadoop/hdds/utils/db/Codec.java | 0
.../hadoop/hdds/{fs => utils/db}/package-info.java | 4 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 8 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 12 +
.../org/apache/hadoop/ozone/common/Storage.java | 12 +
.../ozone/container/common/helpers/BlockData.java | 2 +-
.../common/src/main/resources/ozone-default.xml | 154 +++--
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 22 +-
.../hdds/ratis/TestServerNotLeaderException.java | 82 +++
.../hadoop/hdds/scm/pipeline/MockPipeline.java | 12 +-
hadoop-hdds/config/pom.xml | 4 +-
hadoop-hdds/container-service/pom.xml | 12 +-
.../hdds/datanode/metadata/CRLDBDefinition.java | 99 +++
.../hdds/datanode/metadata/DatanodeCRLStore.java | 84 +++
.../datanode/metadata/DatanodeCRLStoreImpl.java | 127 ++++
.../hdds/datanode/metadata}/package-info.java | 4 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 23 +-
.../common/statemachine/DatanodeStateMachine.java | 2 +-
.../common/states/datanode/InitDatanodeState.java | 4 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 58 +-
.../container/keyvalue/helpers/BlockUtils.java | 27 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 16 +-
.../container/metadata/AbstractDatanodeStore.java | 26 +-
.../metadata/TestDatanodeCRLStoreImpl.java | 127 ++++
.../hdds/datanode/metadata}/package-info.java | 4 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 6 +-
.../container/common/helpers/TestBlockData.java | 2 +-
.../common/impl/TestContainerDataYaml.java | 13 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 31 +-
.../container/keyvalue/TestKeyValueContainer.java | 43 +-
.../TestKeyValueHandlerWithUnhealthyContainer.java | 18 +-
.../container/ozoneimpl/TestContainerReader.java | 1 +
hadoop-hdds/docs/content/feature/OM-HA.zh.md | 13 +-
hadoop-hdds/docs/content/feature/SCM-HA.md | 2 +-
hadoop-hdds/docs/content/feature/SCM-HA.zh.md | 162 +++++
hadoop-hdds/docs/pom.xml | 4 +-
hadoop-hdds/framework/pom.xml | 4 +-
.../java/org/apache/hadoop/hdds}/ExitManager.java | 2 +-
.../java/org/apache/hadoop/hdds}/NodeDetails.java | 2 +-
.../hadoop/hdds/{freon => }/package-info.java | 4 +-
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 15 +
.../SCMSecurityProtocolClientSideTranslatorPB.java | 37 +
.../hadoop/hdds/scm/metadata/SCMMetadataStore.java | 11 +
.../scm/protocol/ScmBlockLocationProtocol.java | 25 +-
...lockLocationProtocolClientSideTranslatorPB.java | 53 +-
...inerLocationProtocolClientSideTranslatorPB.java | 1 +
.../SCMBlockLocationFailoverProxyProvider.java | 59 +-
.../hadoop/hdds/scm/proxy/SCMClientConfig.java | 27 +-
.../SCMContainerLocationFailoverProxyProvider.java | 69 +-
.../SCMSecurityProtocolFailoverProxyProvider.java | 79 ++-
.../CRLInfo.java => certificate/CertInfo.java} | 97 +--
.../certificate/authority/CertificateServer.java | 15 +
.../certificate/authority/CertificateStore.java | 28 +
.../certificate/authority/DefaultApprover.java | 2 +-
.../certificate/authority/DefaultCAServer.java | 28 +-
.../certificate/authority/DefaultCRLApprover.java | 2 +-
.../x509/certificate/client/CertificateClient.java | 17 +
.../client/DefaultCertificateClient.java | 38 +-
.../x509/{crl => certificate}/package-info.java | 6 +-
.../java/org/apache/hadoop/hdds/utils/HAUtils.java | 25 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 98 ++-
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 4 +-
.../x509/certificate/authority/MockCAStore.java | 18 +
.../certificate/authority/TestDefaultCAServer.java | 10 +
.../x509/certificate/utils/TestCRLCodec.java | 1 +
hadoop-hdds/hadoop-dependency-client/pom.xml | 4 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 4 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 4 +-
hadoop-hdds/interface-admin/pom.xml | 4 +-
hadoop-hdds/interface-client/pom.xml | 4 +-
.../interface-client/src/main/proto/hdds.proto | 9 +
hadoop-hdds/interface-server/pom.xml | 4 +-
.../src/main/proto/ScmServerSecurityProtocol.proto | 29 +
hadoop-hdds/pom.xml | 10 +-
hadoop-hdds/server-scm/pom.xml | 8 +-
.../apache/hadoop/hdds/scm/block/BlockManager.java | 9 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 45 +-
.../hdds/scm/container/ContainerManagerImpl.java | 18 +-
.../hdds/scm/container/ContainerManagerV2.java | 9 +-
.../hdds/scm/container/ContainerStateManager.java | 31 +-
.../hdds/scm/container/SCMContainerManager.java | 6 +-
.../apache/hadoop/hdds/scm/ha/HASecurityUtils.java | 39 +-
.../hadoop/hdds/scm/ha/InterSCMGrpcClient.java | 28 +-
.../hdds/scm/ha/InterSCMGrpcProtocolService.java | 23 +
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 18 +
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 5 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 19 +-
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 3 +-
.../apache/hadoop/hdds/scm/ha/SCMNodeDetails.java | 6 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 48 +-
.../hadoop/hdds/scm/ha/SCMSnapshotDownloader.java | 2 +-
.../hadoop/hdds/scm/ha/SCMSnapshotProvider.java | 36 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 29 +-
.../{CRLInfoCodec.java => CertInfoCodec.java} | 22 +-
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 19 +-
.../hdds/scm/metadata/SCMMetadataStoreImpl.java | 14 +-
.../hdds/scm/node/NodeDecommissionManager.java | 57 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 62 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 41 +-
.../scm/pipeline/BackgroundPipelineCreatorV2.java | 38 +-
.../hadoop/hdds/scm/pipeline/PipelineFactory.java | 24 +-
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 39 +-
.../hdds/scm/pipeline/PipelineManagerV2Impl.java | 74 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 18 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 29 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 12 +-
.../hdds/scm/pipeline/PipelineStateManager.java | 34 +-
.../scm/pipeline/PipelineStateManagerV2Impl.java | 35 +-
.../hadoop/hdds/scm/pipeline/PipelineStateMap.java | 163 +----
.../hdds/scm/pipeline/RatisPipelineProvider.java | 54 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 14 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 74 +-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 4 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 27 +-
.../hadoop/hdds/scm/pipeline/StateManager.java | 42 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 83 ++-
...lockLocationProtocolServerSideTranslatorPB.java | 20 +-
...inerLocationProtocolServerSideTranslatorPB.java | 9 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 6 +-
.../safemode/OneReplicaPipelineSafeModeRule.java | 20 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 15 +-
.../hadoop/hdds/scm/server/SCMCertStore.java | 106 ++-
.../hdds/scm/server/SCMClientProtocolServer.java | 12 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 13 +-
.../hadoop/hdds/scm/server/SCMStorageConfig.java | 19 +-
.../hdds/scm/server/StorageContainerManager.java | 117 ++--
.../scm/server/upgrade/SCMUpgradeFinalizer.java | 8 +-
.../apache/hadoop/hdds/scm/TestHddsServerUtil.java | 31 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 21 +-
.../scm/TestStorageContainerManagerHttpServer.java | 2 +
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 21 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 95 +--
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 12 +-
.../container/TestCloseContainerEventHandler.java | 10 +-
.../scm/container/TestContainerManagerImpl.java | 18 +-
.../scm/container/TestContainerStateManager.java | 15 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 7 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 67 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 52 +-
.../hdds/scm/pipeline/MockPipelineManager.java | 53 +-
.../scm/pipeline/MockRatisPipelineProvider.java | 21 +-
.../TestPipelineDatanodesIntersection.java | 13 +-
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 83 +--
.../scm/pipeline/TestPipelinePlacementPolicy.java | 10 +-
.../scm/pipeline/TestPipelineStateManager.java | 101 +--
.../scm/pipeline/TestRatisPipelineProvider.java | 100 +--
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 71 +-
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 6 +
.../scm/pipeline/TestSimplePipelineProvider.java | 28 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 26 +-
.../TestOneReplicaPipelineSafeModeRule.java | 13 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 18 +-
.../hadoop/hdds/scm/server/TestSCMCertStore.java | 36 +-
hadoop-hdds/test-utils/pom.xml | 4 +-
hadoop-hdds/tools/pom.xml | 4 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 6 +-
.../scm/cli/pipeline/CreatePipelineSubcommand.java | 3 +-
.../scm/cli/pipeline/ListPipelinesSubcommand.java | 4 +-
hadoop-ozone/client/pom.xml | 4 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 15 +-
.../hadoop/ozone/client/OzoneKeyLocation.java | 15 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 57 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 76 ++-
.../hadoop/ozone/client/TestHddsClientUtils.java | 150 ++--
hadoop-ozone/common/pom.xml | 4 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 59 +-
.../hadoop/ozone/om/helpers/OzoneAclUtil.java | 20 +
.../ozone/security/acl/IAccessAuthorizer.java | 1 -
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 7 +-
hadoop-ozone/csi/pom.xml | 4 +-
hadoop-ozone/datanode/pom.xml | 16 +-
hadoop-ozone/dev-support/checks/_lib.sh | 142 ++++
hadoop-ozone/dev-support/checks/acceptance.sh | 5 +
hadoop-ozone/dev-support/checks/bats.sh | 5 +
.../dev-support/checks/{bats.sh => docs.sh} | 21 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 12 +-
hadoop-ozone/dev-support/checks/kubernetes.sh | 16 +
.../dist/dev-support/bin/dist-layout-stitching | 3 +
hadoop-ozone/dist/pom.xml | 4 +-
.../dist/src/main/assemblies/ozone-src.xml | 7 +
.../src/main/compose/compatibility/docker-config | 1 +
.../dist/src/main/compose/ozone-csi/docker-config | 1 +
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../src/main/compose/ozone-om-ha/docker-config | 1 +
.../src/main/compose/ozone-topology/docker-config | 1 +
.../dist/src/main/compose/ozone/docker-config | 1 +
.../src/main/compose/ozoneblockade/docker-config | 1 +
.../src/main/compose/ozones3-haproxy/docker-config | 1 +
.../src/main/compose/ozonescripts/docker-config | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 26 +
.../src/main/compose/ozonesecure-ha/docker-config | 2 +-
.../dist/src/main/compose/ozonesecure-ha/test.sh | 1 +
.../src/main/compose/ozonesecure-mr/docker-config | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 1 -
.../src/main/compose/ozonesecure/docker-config | 1 +
.../dist/src/main/compose/ozonesecure/test.sh | 51 +-
.../dist/src/main/compose/restart/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 1 -
.../compose/upgrade/compose/non-ha/docker-config | 1 +
.../dist/src/main/compose/xcompat/docker-config | 1 +
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 15 +-
....activation-activation.txt => LICENSE-CDDL.txt} | 1 -
...E-jakarta.validation-jakarta.validation-api.txt | 37 -
.../LICENSE-jakarta.ws.rs-jakarta.ws.rs-api.md | 637 +++++++++++++++++
.../LICENSE-javax.ws.rs-javax.ws.rs-api.txt | 759 ---------------------
.../NOTICE-jakarta.ws.rs-jakarta.ws.rs-api.md | 61 ++
.../src/main/smoketest/admincli/pipeline.robot | 7 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 2 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 2 -
.../fault-injection-test/mini-chaos-tests/pom.xml | 4 +-
.../fault-injection-test/network-tests/pom.xml | 2 +-
.../network-tests/src/test/compose/docker-config | 1 +
hadoop-ozone/fault-injection-test/pom.xml | 4 +-
hadoop-ozone/insight/pom.xml | 12 +-
hadoop-ozone/integration-test/pom.xml | 4 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 1 -
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 11 +-
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 10 +-
.../apache/hadoop/hdds/scm/TestSCMSnapshot.java | 6 +-
.../metrics/TestSCMContainerManagerMetrics.java | 9 +-
.../hdds/scm/pipeline/TestLeaderChoosePolicy.java | 25 +-
.../hdds/scm/pipeline/TestNode2PipelineMap.java | 9 +-
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 7 +-
.../hdds/scm/pipeline/TestPipelineClose.java | 10 +-
.../TestRatisPipelineCreateAndDestroy.java | 30 +-
.../hadoop/hdds/scm/pipeline/TestSCMRestart.java | 16 +-
.../safemode/TestSCMSafeModeWithPipelineRules.java | 15 +-
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 23 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 15 +-
.../apache/hadoop/ozone/TestDelegationToken.java | 1 -
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 6 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 10 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 6 +-
.../hadoop/ozone/TestStorageContainerManager.java | 6 +
.../ozone/client/CertificateClientTestImpl.java | 13 +
.../TestContainerStateMachineFailureOnRead.java | 7 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 5 +-
.../client/rpc/TestHybridPipelineOnDatanode.java | 2 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 25 +
.../client/rpc/TestOzoneRpcClientAbstract.java | 120 +++-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 5 +
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 13 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 2 +-
.../ozone/om/TestOzoneManagerConfiguration.java | 1 -
.../ozone/om/TestOzoneManagerRestInterface.java | 3 +-
.../hadoop/ozone/om/TestSecureOzoneManager.java | 4 +-
.../hadoop/ozone/recon/TestReconAsPassiveScm.java | 13 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 5 +-
.../hadoop/ozone/scm/TestFailoverWithSCMHA.java | 147 ++++
.../TestSCMContainerPlacementPolicyMetrics.java | 7 +-
.../ozone/scm/TestSCMInstallSnapshotWithHA.java | 87 +--
.../org/apache/hadoop/ozone/scm/TestSCMMXBean.java | 9 +-
.../hadoop/ozone/scm/TestXceiverClientGrpc.java | 7 +-
.../ozone/scm/pipeline/TestSCMPipelineMetrics.java | 11 +-
hadoop-ozone/interface-client/pom.xml | 4 +-
hadoop-ozone/interface-storage/pom.xml | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 4 +-
.../java/org/apache/hadoop/ozone/om/OMStorage.java | 17 -
.../org/apache/hadoop/ozone/om/OzoneManager.java | 81 ++-
.../hadoop/ozone/om/OzoneManagerStarter.java | 4 -
.../apache/hadoop/ozone/om/ha/OMNodeDetails.java | 2 +-
.../S3MultipartUploadCompleteRequest.java | 2 +-
.../ozone/om/ScmBlockLocationTestingClient.java | 12 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 9 +-
.../ozone/om/TestOzoneManagerHttpServer.java | 2 +
.../ozone/om/request/TestOMRequestUtils.java | 54 +-
.../om/request/key/TestOMKeyCommitRequest.java | 78 ++-
.../ozone/om/request/key/TestOMKeyRequest.java | 6 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 4 +-
hadoop-ozone/ozonefs-common/pom.xml | 4 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 4 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 4 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 4 +-
hadoop-ozone/pom.xml | 4 +-
hadoop-ozone/recon-codegen/pom.xml | 2 +-
hadoop-ozone/recon/pom.xml | 11 +-
.../org/apache/hadoop/ozone/recon/ReconServer.java | 2 -
.../hadoop/ozone/recon/api/NodeEndpoint.java | 6 +-
.../hadoop/ozone/recon/api/PipelineEndpoint.java | 11 +-
.../ozone/recon/scm/ReconPipelineFactory.java | 9 +-
.../recon/scm/ReconPipelineReportHandler.java | 2 +-
.../scm/ReconStorageContainerManagerFacade.java | 15 +-
.../ozone/recon/OMMetadataManagerTestUtils.java | 9 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 4 +-
.../ozone/recon/scm/TestReconPipelineManager.java | 12 +-
hadoop-ozone/s3gateway/pom.xml | 20 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 283 +++++++-
.../s3/endpoint/ListMultipartUploadsResult.java | 53 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 3 +-
...r.java => PutBucketAclRequestUnmarshaller.java} | 22 +-
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 342 ++++++++++
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 235 +++++++
...artUploadInitiateResponse.java => S3Owner.java} | 55 +-
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 5 +
.../apache/hadoop/ozone/s3/util/RFC1123Util.java | 3 +-
.../hadoop/ozone/client/OzoneBucketStub.java | 23 +
.../hadoop/ozone/client/OzoneVolumeStub.java | 24 +
.../hadoop/ozone/s3/endpoint/TestBucketAcl.java | 266 ++++++++
.../{TestBucketGet.java => TestBucketList.java} | 76 +--
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 6 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 56 +-
.../src/test/resources/groupAccessControlList.xml | 57 ++
.../src/test/resources/userAccessControlList.xml | 42 ++
hadoop-ozone/tools/pom.xml | 12 +-
.../StorageContainerManagersCommandHandler.java | 2 +-
.../apache/hadoop/ozone/debug/ChunkKeyHandler.java | 5 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 3 +-
.../hadoop/ozone/freon/DatanodeChunkGenerator.java | 3 +-
.../hadoop/ozone/freon/DatanodeChunkValidator.java | 5 +-
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 6 +-
.../GenerateOzoneRequiredConfigurations.java | 44 +-
.../ozone/genesis/BenchMarkContainerStateMap.java | 92 +--
.../ozone/genesis/BenchMarkOzoneManager.java | 5 +-
.../apache/hadoop/ozone/genesis/BenchMarkSCM.java | 7 +-
.../apache/hadoop/ozone/genesis/GenesisUtil.java | 5 +-
.../TestGenerateOzoneRequiredConfigurations.java | 52 +-
pom.xml | 42 +-
341 files changed, 7276 insertions(+), 3445 deletions(-)
copy .github/workflows/{close-pending.yaml => cancel-ci.yaml} (50%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ServerNotLeaderException.java
copy
hadoop-hdds/common/src/{test/java/org/apache/hadoop/hdds/scm/ha/package-info.java
=> main/java/org/apache/hadoop/hdds/scm/ha/NonRetriableException.java} (79%)
copy
hadoop-hdds/common/src/{test/java/org/apache/hadoop/hdds/scm/ha/package-info.java
=>
main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java}
(81%)
rename
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils
=>
common/src/main/java/org/apache/hadoop/hdds/security/x509/crl}/CRLCodec.java
(99%)
copy hadoop-hdds/{framework =>
common}/src/main/java/org/apache/hadoop/hdds/security/x509/crl/CRLInfo.java
(87%)
copy hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata
=>
common/src/main/java/org/apache/hadoop/hdds/security/x509/crl}/CRLInfoCodec.java
(94%)
copy hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/utils =>
common/src/main/java/org/apache/hadoop/hdds/security/x509/crl}/package-info.java
(88%)
rename hadoop-hdds/{framework =>
common}/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java (100%)
copy hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/{fs =>
utils/db}/package-info.java (90%)
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestServerNotLeaderException.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/CRLDBDefinition.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/DatanodeCRLStore.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/DatanodeCRLStoreImpl.java
copy hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/fs =>
container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata}/package-info.java
(87%)
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/hdds/datanode/metadata/TestDatanodeCRLStoreImpl.java
copy hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/utils/db =>
container-service/src/test/java/org/apache/hadoop/hdds/datanode/metadata}/package-info.java
(91%)
create mode 100644 hadoop-hdds/docs/content/feature/SCM-HA.zh.md
rename {hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util =>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds}/ExitManager.java
(96%)
rename {hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/ha =>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds}/NodeDetails.java
(98%)
copy hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/{freon =>
}/package-info.java (90%)
rename
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/{crl/CRLInfo.java
=> certificate/CertInfo.java} (56%)
copy
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/x509/{crl
=> certificate}/package-info.java (91%)
rename
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/{CRLInfoCodec.java
=> CertInfoCodec.java} (71%)
create mode 100644 hadoop-ozone/dev-support/checks/_lib.sh
copy hadoop-ozone/dev-support/checks/{bats.sh => docs.sh} (74%)
copy
hadoop-ozone/dist/src/main/license/bin/licenses/{LICENSE-javax.activation-activation.txt
=> LICENSE-CDDL.txt} (99%)
copy LICENSE.txt =>
hadoop-ozone/dist/src/main/license/bin/licenses/LICENSE-jakarta.validation-jakarta.validation-api.txt
(87%)
create mode 100644
hadoop-ozone/dist/src/main/license/bin/licenses/LICENSE-jakarta.ws.rs-jakarta.ws.rs-api.md
delete mode 100644
hadoop-ozone/dist/src/main/license/bin/licenses/LICENSE-javax.ws.rs-javax.ws.rs-api.txt
create mode 100644
hadoop-ozone/dist/src/main/license/bin/licenses/NOTICE-jakarta.ws.rs-jakarta.ws.rs-api.md
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestFailoverWithSCMHA.java
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/{CompleteMultipartUploadRequestUnmarshaller.java
=> PutBucketAclRequestUnmarshaller.java} (80%)
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3Acl.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3BucketAcl.java
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/{MultipartUploadInitiateResponse.java
=> S3Owner.java} (58%)
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketAcl.java
rename
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/{TestBucketGet.java
=> TestBucketList.java} (80%)
create mode 100644
hadoop-ozone/s3gateway/src/test/resources/groupAccessControlList.xml
create mode 100644
hadoop-ozone/s3gateway/src/test/resources/userAccessControlList.xml
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
