This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch HDDS-2939
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 974136d HDDS-4923. [FSO]S3Multipart: Run S3 acceptance test with
prefix layout (#2208)
add 6af26ca HDDS-4889. Add simple CI check for docs (#2156)
add 7f2dc04 HDDS-5127. Fix getServiceList when SCM HA is enabled (#2173)
add 1e1eefa HDDS-5051. Ensure failover to suggested leader if any for
NotLeaderException. (#2141)
add 6f9c3f2 HDDS-5060. [SCM HA Security] Make InterSCM grpc channel
secure. (#2187)
add bddc873 HDDS-4515. Datanodes should be able to persist and load CRL
(#2181)
add ce29843 HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT
(#2184)
add bfb6be8 HDDS-5152. Fix Suggested leader in Client. (#2189)
add bccc439 HDDS-5147. Intermittent test failure in
TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (#2188)
add a920f25 HDDS-5153. Decommissioning a dead node should complete
immediately (#2190)
add 66a411b HDDS-4585. Support bucket acl operation in S3g (#1701)
add 5dd0943 HDDS-5144. Create github check to alert when dependency tree
is changed (#2177)
add 10312fb HDDS-4983. Display key offset for each block in command key
info (#2051)
add 799c38e Revert "HDDS-5144. Create github check to alert when
dependency tree is changed (#2177)"
add dcac0f1 HDDS-5166. Remove duplicate assignment of OZONE_OPTS for
freon and sh (#2195)
add 48b84c5 HDDS-2212. Genconf tool should generate config files for
secure clust… (#1788)
add a3e7f04 HDDS-5182. Acceptance test may exit with 0 in case of error
(#2212)
add 2e07c9f HDDS-5178. Update project information of Contribution
guideline (#2210)
add 8bbbf3f HDDS-5177. Update link of weekly calls in README (#2209)
add 00eba3e HDDS-5144. Create github check to alert when dependency tree
is changed (#2199)
add 142a2dd HDDS-5185. Update commons-io to 2.8.0 (#2215)
add d430ff3 HDDS-5159. Make periodic disk check interval configurable.
(#2193)
add 74496a7 HDDS-5196. Fix typo "vaild" in HeartbeatEndpointTask.java and
RegisterEndpointTask (#2222).
add 2061d7d HDDS-5192. Intermittent failure in TestOzoneRpcClient due to
volume name conflict (#2220)
add 5e8ae99 HDDS-5186. If primordial SCM id is set, a non-HA cluster can
not be initialized. (#2218)
add a16e648 HDDS-5126.Recon should check new containers of a container
report with batch (#2172)
add 2c37a15 HDDS-5183. For AccessControlException do not perform
failover. (#2213)
add 75c5c10 HDDS-5099. Error with unit test for hdds.container-service
TestSchemaOneBackwardsCompatibility (#2183)
add 8e8a01d HDDS-5197. Pass option variables to OZONE_OPTS before adding
default GC opts (#2225)
add 7ba1499 HDDS-5198. Remove JBoss repo definition (#2226)
add 2c4a4de HDDS-5173. Divide snapshot related work into
notifyInstallSnapshotFromLeader and reinitialize for SCMStateMachine. (#2224)
add 49538aa HDDS-5207. Intermittent failure in
TestRatisPipelineProvider#testCreatePipelineWithFactorThree (#2233)
add 3e66778 HDDS-5200. Fix scm roles command if one of the host is
unresolvable. (#2227)
add abe136f HDDS-5203. Allow suppressing deprecation warning for HADOOP_
variables (#2231)
add 248d72d Bump logical release name of Ozone 1.2 (#2238)
add b5cbb65 HDDS-5164. Improve client and server logging. (#2234)
add 7262004 HDDS-5089. On-demand disk checker for hdds volume (#2158)
add 842e6f6 HDDS-4729. Add token support for container admin operations
(#2186)
add 03620da HDDS-5059. Increase number of client retries/ failovers to
OMs (#2171)
add f8a06e0 HDDS-5202. Use scm#checkLeader before processing client
requests . (#2229)
add 6075acb HDDS-5205. Make admin check work for SCM HA cluster (#2236)
add 0f6bb36 HDDS-5040. Append om1,om2,om3 to ozone.administrators config
for secure OM HA docker config (#2245)
add a7c8695 HDDS-5160. Remove some Freon integration tests (#2194)
add 6b24c18 HDDS-4976. Add container replica related commands to debug
(#2070)
add fd71f09 The modificationTime should modify when setAcl success (#2242)
add 4189088 HDDS-5165. OM DB checkpoint servlet not accessible in a
secure cluster (#2200)
add eff4914 HDDS-5208. bump rocksdb version to 6.20.3 (#2235)
add 36778af HDDS-5128. Return specific error messages for S3-SDK (#2169)
add 19f989f HDDS-5176. Increase default block cache capacity for
Datanodes (#2207)
add 9cab560 HDDS-5155. Add revokeCertificate to
SCMSecurityProtocolServer. (#2191)
add 727b258 HDDS-5189. fix markdown files to make them hugo83 compatible
(and valid) (#2232)
add 7a4a93e HDDS-5229. Update copyright year in NOTICE (#2248)
add ac7fc7e HDDS-5082. Create unit (!) test for OzoneClient (#2138)
add 2254abf HDDS-5216. Fix race condition causing SCM failOverProxy which
is causing failover wrongly. (#2247)
add fcc8e92 HDDS-5187. Avoid Maven connection errors in CI (#2237)
add 10a5b39 HDDS-5113. Disable flaky
TestBlockDeletingService#testContainerThrottle
add 91993c1 HDDS-5234.Change default grpc and ratis ports for scm ha
(#2251)
add ea1ed7c HDDS-5238. Disable animal-sniffer maven plugin (#2253)
add 9f6d7f2 HDDS-5167. Add clean option for Freon randomkeys to clean
generated objects (#2198)
add d63dd6b HDDS-5239. Create http Basedir if not exist (#2255)
add 5a50866 HDDS-5209. Datanode hasEnoughSpace check should apply on
volume instead of global DN (#2246)
add 30fe6ea HDDS-5245. Fix OzoneContainer TLS configuration. (#2264)
add 4fd8187 HDDS-5206. Support revoking S3 secret (#2239)
add 08375d7 HDDS-5246. Wait for ever to obtain CA list which is needed
during OM/DN startup (#2266)
add 70ef475 HDDS-5233. SCM subsequent init failed when previous scm init
failed. (#2249)
add 09c2278 HDDS-5257. Avoid SCM call to get CA certs in non-HA from OM.
(#2273)
add e1acbb0 HDDS-5256. Fix fall back of config in SCM HA Cluster (#2272)
add 61a237a HDDS-5261. Delete Useless ozone subcommand (#2275)
new 723cdaa Merge remote-tracking branch 'origin/master' into HDDS-2939
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 33 +-
.gitignore | 3 +
CONTRIBUTING.md | 10 +-
NOTICE.txt | 2 +-
README.md | 2 +-
.../hadoop/hdds/scm/XceiverClientFactory.java | 3 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 8 +-
hadoop-hdds/common/pom.xml | 5 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 3 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 119 ++++---
.../hdds/ratis/ServerNotLeaderException.java | 113 ++++++
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 8 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 3 +-
.../hadoop/hdds/scm/ha/NonRetriableException.java | 30 ++
.../hadoop/hdds/scm/ha/SCMHAConfiguration.java | 4 +-
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 102 +++++-
.../org/apache/hadoop/hdds/scm/ha/SCMNodeInfo.java | 5 +-
.../protocol/StorageContainerLocationProtocol.java | 18 +
.../security/token/ContainerTokenIdentifier.java | 103 ++++++
.../security/token/OzoneBlockTokenIdentifier.java | 146 +++-----
.../security/token/ShortLivedTokenIdentifier.java | 112 ++++++
.../hadoop/hdds/security/x509/SecurityConfig.java | 16 +
.../hadoop/hdds/security/x509/crl}/CRLCodec.java | 2 +-
.../hadoop/hdds/security/x509/crl/CRLInfo.java | 1 -
.../hdds/security/x509/crl}/CRLInfoCodec.java | 3 +-
.../hdds/security/x509/crl/package-info.java | 24 ++
.../org/apache/hadoop/hdds/utils/db/Codec.java | 0
.../apache/hadoop/hdds/utils/db/package-info.java | 22 ++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 6 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 12 +
.../common/src/main/resources/ozone-default.xml | 172 ++++++---
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 22 +-
.../hdds/ratis/TestServerNotLeaderException.java | 82 +++++
.../ozone/container/ContainerTestHelper.java | 45 +--
.../hdds/datanode/metadata/CRLDBDefinition.java | 99 ++++++
.../hdds/datanode/metadata/DatanodeCRLStore.java | 84 +++++
.../datanode/metadata/DatanodeCRLStoreImpl.java | 127 +++++++
.../hdds/datanode/metadata/package-info.java | 22 ++
.../apache/hadoop/ozone/HddsDatanodeService.java | 19 +-
.../container/common/impl/HddsDispatcher.java | 25 +-
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../common/statemachine/DatanodeStateMachine.java | 2 +-
.../common/states/datanode/InitDatanodeState.java | 4 +-
.../states/endpoint/HeartbeatEndpointTask.java | 2 +-
.../states/endpoint/RegisterEndpointTask.java | 2 +-
.../container/common/utils/HddsVolumeUtil.java | 11 +
.../ozone/container/common/volume/HddsVolume.java | 13 +
.../container/common/volume/MutableVolumeSet.java | 28 +-
.../container/keyvalue/KeyValueContainer.java | 6 +
.../container/keyvalue/KeyValueContainerData.java | 2 +
.../ozone/container/keyvalue/KeyValueHandler.java | 3 +-
.../container/keyvalue/helpers/BlockUtils.java | 4 +-
.../container/keyvalue/helpers/ChunkUtils.java | 42 ++-
.../keyvalue/impl/ChunkManagerDispatcher.java | 10 +-
.../keyvalue/impl/ChunkManagerFactory.java | 5 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 32 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 15 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 21 +-
.../metadata/TestDatanodeCRLStoreImpl.java | 127 +++++++
.../hdds/datanode/metadata/package-info.java} | 31 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 6 +-
.../container/common/TestBlockDeletingService.java | 47 ++-
.../TestSchemaOneBackwardsCompatibility.java | 18 +
.../impl/TestContainerDeletionChoosingPolicy.java | 31 +-
.../common/impl/TestContainerPersistence.java | 3 +-
.../statemachine/TestDatanodeConfiguration.java | 15 +
.../container/keyvalue/ChunkLayoutTestInfo.java | 4 +-
.../container/keyvalue/helpers/TestChunkUtils.java | 14 +-
.../keyvalue/impl/TestFilePerChunkStrategy.java | 3 +-
hadoop-hdds/docs/content/interface/ReconApi.md | 390 +++++++++------------
hadoop-hdds/docs/content/interface/ReconApi.zh.md | 356 ++++++++-----------
hadoop-hdds/framework/pom.xml | 6 +
.../hadoop/hdds/protocol/SCMSecurityProtocol.java | 11 +
.../SCMSecurityProtocolClientSideTranslatorPB.java | 14 +
.../hadoop/hdds/scm/protocolPB/OzonePBHelper.java | 82 +++++
...lockLocationProtocolClientSideTranslatorPB.java | 6 -
...inerLocationProtocolClientSideTranslatorPB.java | 64 +++-
.../SCMBlockLocationFailoverProxyProvider.java | 150 +++++---
.../hadoop/hdds/scm/proxy/SCMClientConfig.java | 27 +-
.../SCMContainerLocationFailoverProxyProvider.java | 185 +++++-----
.../SCMSecurityProtocolFailoverProxyProvider.java | 97 +++--
.../hadoop/hdds}/security/OzoneSecretKey.java | 4 +-
.../hadoop/hdds}/security/OzoneSecretManager.java | 2 +-
.../hdds}/security/OzoneSecurityException.java | 2 +-
.../apache/hadoop/hdds/security/package-info.java | 31 +-
.../hdds/security/token/BlockTokenVerifier.java | 149 +++-----
.../security/token/CompositeTokenVerifier.java | 42 ++-
.../token/ContainerTokenSecretManager.java | 48 +++
.../security/token/ContainerTokenVerifier.java | 51 +++
.../hdds/security/token/NoopTokenVerifier.java | 38 +-
.../token/ShortLivedTokenSecretManager.java | 107 ++++++
.../security/token/ShortLivedTokenVerifier.java | 134 +++++++
.../hadoop/hdds/security/token/TokenVerifier.java | 63 +++-
.../certificate/authority/CertificateServer.java | 4 +-
.../certificate/authority/DefaultCAServer.java | 3 +-
.../certificate/authority/DefaultCRLApprover.java | 2 +-
.../x509/certificate/client/CertificateClient.java | 15 +-
.../hadoop/hdds/server/http/BaseHttpServer.java | 2 +
.../hadoop/hdds/utils/DBCheckpointServlet.java | 30 +-
.../java/org/apache/hadoop/hdds/utils/HAUtils.java | 162 +++++----
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 45 +--
.../security/token/TestBlockTokenVerifier.java | 75 ++++
.../security/token/TestContainerTokenVerifier.java | 75 ++++
.../token/TestOzoneBlockTokenIdentifier.java | 2 +-
.../hdds/security/token/TokenVerifierTests.java | 242 +++++++++++++
.../certificate/authority/TestDefaultCAServer.java | 6 +-
.../x509/certificate/utils/TestCRLCodec.java | 1 +
.../src/main/proto/ScmAdminProtocol.proto | 31 ++
.../interface-client/src/main/proto/hdds.proto | 16 +
.../src/main/proto/ScmServerProtocol.proto | 1 +
.../src/main/proto/ScmServerSecurityProtocol.proto | 28 ++
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 16 +-
.../java/org/apache/hadoop/hdds/scm/ScmUtils.java | 51 ++-
.../scm/container/ContainerStateManagerImpl.java | 11 +-
.../hadoop/hdds/scm/ha/InterSCMGrpcClient.java | 30 +-
.../hdds/scm/ha/InterSCMGrpcProtocolService.java | 22 ++
.../hadoop/hdds/scm/ha/MockSCMHAManager.java | 16 +-
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 18 +
.../apache/hadoop/hdds/scm/ha/SCMHAManager.java | 29 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 192 +++++-----
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 7 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 53 ++-
.../hadoop/hdds/scm/ha/SCMSnapshotDownloader.java | 5 +-
.../hadoop/hdds/scm/ha/SCMSnapshotProvider.java | 19 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 94 +++--
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 1 +
.../hdds/scm/node/NodeDecommissionManager.java | 31 +-
.../SCMSecurityProtocolServerSideTranslatorPB.java | 33 +-
...lockLocationProtocolServerSideTranslatorPB.java | 19 +-
...inerLocationProtocolServerSideTranslatorPB.java | 58 ++-
.../hdds/scm/server/SCMClientProtocolServer.java | 92 +++--
.../hdds/scm/server/SCMDBCheckpointServlet.java | 5 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 33 +-
.../hdds/scm/server/StorageContainerManager.java | 153 ++++++--
.../apache/hadoop/hdds/scm/TestHddsServerUtil.java | 31 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 21 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 21 +-
.../algorithms/TestContainerPlacementFactory.java | 51 ++-
.../TestSCMContainerPlacementCapacity.java | 37 +-
.../TestSCMContainerPlacementRackAware.java | 105 ++++--
.../TestSCMContainerPlacementRandom.java | 32 +-
.../hadoop/hdds/scm/ha/TestSCMHAConfiguration.java | 76 ++++
.../hdds/scm/node/TestNodeDecommissionManager.java | 33 ++
.../scm/pipeline/TestRatisPipelineProvider.java | 4 +-
.../hdds/scm/cli/ContainerOperationClient.java | 30 +-
.../apache/hadoop/ozone/client/ObjectStore.java | 4 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 15 +-
.../hadoop/ozone/client/OzoneKeyLocation.java | 15 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 57 ++-
.../client/io/BlockOutputStreamEntryPool.java | 2 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 5 +-
.../ozone/client/protocol/ClientProtocol.java | 7 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 50 ++-
.../hadoop/ozone/client/MockBlockAllocator.java} | 36 +-
.../hadoop/ozone/client/MockDatanodeStorage.java | 73 ++++
.../hadoop/ozone/client/MockOmTransport.java | 276 +++++++++++++++
.../ozone/client/MockXceiverClientFactory.java | 72 ++++
.../hadoop/ozone/client/MockXceiverClientSpi.java | 180 ++++++++++
.../ozone/client/SinglePipelineBlockAllocator.java | 90 +++++
.../hadoop/ozone/client/TestHddsClientUtils.java | 150 ++++----
.../hadoop/ozone/client/TestOzoneClient.java | 174 +++++++++
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 1 +
.../org/apache/hadoop/ozone/audit/OMAction.java | 3 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 39 +--
.../hadoop/ozone/om/helpers/OzoneAclUtil.java | 20 ++
.../ozone/om/protocol/OzoneManagerProtocol.java | 8 +
...OzoneManagerProtocolClientSideTranslatorPB.java | 11 +
.../ozone/security/acl/IAccessAuthorizer.java | 1 -
hadoop-ozone/dev-support/checks/_lib.sh | 28 ++
hadoop-ozone/dev-support/checks/build.sh | 2 +-
hadoop-ozone/dev-support/checks/dependency.sh | 50 +++
.../dev-support/checks/{build.sh => docs.sh} | 27 +-
hadoop-ozone/dev-support/checks/junit.sh | 2 +-
.../src/main/compose/compatibility/docker-config | 1 +
.../dist/src/main/compose/ozone-csi/docker-config | 1 +
.../src/main/compose/ozone-ha/docker-compose.yaml | 10 +-
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../src/main/compose/ozone-om-ha/docker-config | 1 +
.../src/main/compose/ozone-topology/docker-config | 1 +
.../dist/src/main/compose/ozone/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 1 +
.../src/main/compose/ozoneblockade/docker-config | 1 +
.../src/main/compose/ozones3-haproxy/docker-config | 1 +
.../src/main/compose/ozonescripts/docker-config | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 16 +-
.../src/main/compose/ozonesecure-ha/docker-config | 4 +-
.../dist/src/main/compose/ozonesecure-ha/test.sh | 5 +
.../src/main/compose/ozonesecure-mr/docker-config | 2 +
.../src/main/compose/ozonesecure/docker-config | 2 +
.../dist/src/main/compose/restart/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
.../dist/src/main/compose/upgrade/docker-config | 1 +
.../dist/src/main/compose/xcompat/docker-config | 1 +
hadoop-ozone/dist/src/main/license/bin/NOTICE.txt | 2 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 235 +++++++++++++
.../test.sh => license/update-jar-report.sh} | 28 +-
.../src/main/smoketest/admincli/container.robot | 14 +-
.../src/main/smoketest/admincli/datanode.robot | 6 +-
.../src/main/smoketest/admincli/pipeline.robot | 7 +-
.../smoketest/admincli/replicationmanager.robot | 7 +-
.../src/main/smoketest/admincli/safemode.robot | 6 +-
.../dist/src/main/smoketest/admincli/scmha.robot | 2 +-
.../dist/src/main/smoketest/cli/envvars.robot | 15 +
.../dist/src/main/smoketest/commonlib.robot | 11 +
.../dist/src/main/smoketest/freon/remove.robot | 43 +++
hadoop-ozone/dist/src/shell/ozone/ozone | 7 +-
.../dist/src/shell/ozone/ozone-functions.sh | 3 +-
.../network-tests/src/test/compose/docker-config | 1 +
.../hdds/scm/TestSCMDbCheckpointServlet.java | 9 +-
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 12 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 7 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 33 ++
.../hadoop/ozone/TestOzoneConfigurationFields.java | 5 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 50 ++-
.../hadoop/ozone/TestStorageContainerManager.java | 32 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 3 +-
.../TestContainerStateMachineFailureOnRead.java | 6 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 10 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 24 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 12 -
.../ozone/client/rpc/TestWatchForCommit.java | 3 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 11 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 151 ++++----
.../server/TestSecureContainerServer.java | 183 +++++-----
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 243 +++++++++++++
.../TestDatanodeHddsVolumeFailureDetection.java | 289 +++++++++++++++
.../ozone/freon/TestFreonWithDatanodeRestart.java | 126 -------
.../ozone/freon/TestHadoopDirTreeGenerator.java | 2 +-
.../ozone/freon/TestHadoopNestedDirGenerator.java | 5 +-
.../ozone/freon/TestOzoneClientKeyGenerator.java | 94 -----
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 19 +
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 274 +++++++++------
.../ozone/om/TestOzoneManagerRestInterface.java | 3 +-
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 2 +-
.../hadoop/ozone/scm/TestFailoverWithSCMHA.java | 147 ++++++++
.../ozone/scm/TestSCMInstallSnapshotWithHA.java | 73 ++--
.../src/main/proto/OmClientProtocol.proto | 12 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 8 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 27 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 117 +++++--
.../hadoop/ozone/om/S3SecretManagerImpl.java | 4 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 3 +
.../om/request/bucket/OMBucketDeleteRequest.java | 2 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 9 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 2 +-
.../om/request/s3/security/S3GetSecretRequest.java | 11 +-
.../request/s3/security/S3RevokeSecretRequest.java | 155 ++++++++
.../s3/security/S3RevokeSecretResponse.java | 57 +++
...OzoneManagerProtocolServerSideTranslatorPB.java | 27 +-
.../security/OzoneBlockTokenSecretManager.java | 86 +----
.../OzoneDelegationTokenSecretManager.java | 1 +
.../security/TestOzoneBlockTokenSecretManager.java | 285 ++++++++-------
hadoop-ozone/ozonefs-hadoop2/pom.xml | 10 -
hadoop-ozone/ozonefs-hadoop3/pom.xml | 10 -
hadoop-ozone/ozonefs/pom.xml | 10 -
.../ozone/recon/scm/ReconContainerManager.java | 107 ++++--
.../recon/scm/ReconContainerReportHandler.java | 24 +-
.../ReconIncrementalContainerReportHandler.java | 2 +-
.../recon/spi/StorageContainerServiceProvider.java | 8 +
.../impl/StorageContainerServiceProviderImpl.java | 6 +
.../ozone/recon/api/TestContainerEndpoint.java | 2 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 8 +
.../scm/AbstractReconContainerManagerTest.java | 30 ++
.../ozone/recon/scm/TestReconContainerManager.java | 50 ++-
...TestReconIncrementalContainerReportHandler.java | 3 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 17 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 283 +++++++++++++--
.../s3/endpoint/ListMultipartUploadsResult.java | 53 +--
.../endpoint/PutBucketAclRequestUnmarshaller.java | 85 +++++
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 342 ++++++++++++++++++
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 235 +++++++++++++
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 72 ++++
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 5 +
.../hadoop/ozone/client/OzoneBucketStub.java | 23 ++
.../hadoop/ozone/client/OzoneVolumeStub.java | 24 ++
.../hadoop/ozone/s3/TestOzoneClientProducer.java | 4 +-
.../hadoop/ozone/s3/endpoint/TestBucketAcl.java | 266 ++++++++++++++
.../{TestBucketGet.java => TestBucketList.java} | 76 ++--
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 6 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 56 ++-
.../src/test/resources/groupAccessControlList.xml | 57 +++
.../src/test/resources/userAccessControlList.xml | 42 +++
.../StorageContainerManagersCommandHandler.java | 2 +-
.../org/apache/hadoop/ozone/debug/OzoneDebug.java | 10 +
.../ContainerCommands.java} | 130 +++----
.../ozone/debug/container/ExportSubcommand.java | 75 ++++
.../ozone/debug/container/InfoSubcommand.java | 53 +++
.../container/ListSubcommand.java} | 53 ++-
.../hadoop/ozone/debug/container/package-info.java | 22 ++
.../hadoop/ozone/freon/BaseFreonGenerator.java | 7 +
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../java/org/apache/hadoop/ozone/freon/Freon.java | 2 +
.../hadoop/ozone/freon/OmBucketGenerator.java | 2 +-
...OmBucketGenerator.java => OmBucketRemover.java} | 36 +-
...etGenerator.java => OzoneClientKeyRemover.java} | 60 ++--
.../hadoop/ozone/freon/RandomKeyGenerator.java | 137 ++++++++
.../apache/hadoop/ozone/freon/S3KeyGenerator.java | 1 +
.../containergenerator/GeneratorDatanode.java | 2 +-
.../GenerateOzoneRequiredConfigurations.java | 44 ++-
.../ozone/genesis/BenchmarkChunkManager.java | 4 +-
.../hadoop/ozone/shell/s3/GetS3SecretHandler.java | 11 +-
...cretHandler.java => RevokeS3SecretHandler.java} | 57 +--
.../org/apache/hadoop/ozone/shell/s3/S3Shell.java | 3 +-
.../TestGenerateOzoneRequiredConfigurations.java | 52 ++-
pom.xml | 44 +--
305 files changed, 10352 insertions(+), 3222 deletions(-)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/ServerNotLeaderException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/NonRetriableException.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/ContainerTokenIdentifier.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/token/ShortLivedTokenIdentifier.java
rename
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/utils
=>
common/src/main/java/org/apache/hadoop/hdds/security/x509/crl}/CRLCodec.java
(99%)
rename hadoop-hdds/{framework =>
common}/src/main/java/org/apache/hadoop/hdds/security/x509/crl/CRLInfo.java
(98%)
rename
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata =>
common/src/main/java/org/apache/hadoop/hdds/security/x509/crl}/CRLInfoCodec.java
(94%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/crl/package-info.java
copy hadoop-hdds/{framework =>
common}/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java (100%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/package-info.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/TestServerNotLeaderException.java
rename hadoop-hdds/{container-service =>
common}/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java
(92%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/CRLDBDefinition.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/DatanodeCRLStore.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/DatanodeCRLStoreImpl.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/hdds/datanode/metadata/package-info.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/hdds/datanode/metadata/TestDatanodeCRLStoreImpl.java
rename
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/utils/db/Codec.java
=>
container-service/src/test/java/org/apache/hadoop/hdds/datanode/metadata/package-info.java}
(52%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/OzonePBHelper.java
rename {hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone =>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds}/security/OzoneSecretKey.java
(97%)
rename {hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone =>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds}/security/OzoneSecretManager.java
(99%)
rename {hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone =>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds}/security/OzoneSecurityException.java
(98%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/OzoneDebug.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/package-info.java
(51%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/OzoneDebug.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/CompositeTokenVerifier.java
(50%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/ContainerTokenSecretManager.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/ContainerTokenVerifier.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/OzoneDebug.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/NoopTokenVerifier.java
(52%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/ShortLivedTokenSecretManager.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/ShortLivedTokenVerifier.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/security/token/TestBlockTokenVerifier.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/security/token/TestContainerTokenVerifier.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/security/token/TokenVerifierTests.java
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/debug/OzoneDebug.java
=> client/src/test/java/org/apache/hadoop/ozone/client/MockBlockAllocator.java}
(52%)
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/MockDatanodeStorage.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/MockOmTransport.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/MockXceiverClientFactory.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/MockXceiverClientSpi.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/SinglePipelineBlockAllocator.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/TestOzoneClient.java
create mode 100755 hadoop-ozone/dev-support/checks/dependency.sh
copy hadoop-ozone/dev-support/checks/{build.sh => docs.sh} (63%)
create mode 100644 hadoop-ozone/dist/src/main/license/jar-report.txt
copy hadoop-ozone/dist/src/main/{compose/ozonesecure-ha/test.sh =>
license/update-jar-report.sh} (54%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/freon/remove.robot
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/DatanodeTestUtils.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/volume/TestDatanodeHddsVolumeFailureDetection.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestFreonWithDatanodeRestart.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOzoneClientKeyGenerator.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestFailoverWithSCMHA.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/S3RevokeSecretRequest.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/security/S3RevokeSecretResponse.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/PutBucketAclRequestUnmarshaller.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3Acl.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3BucketAcl.java
create mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3Owner.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketAcl.java
rename
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/{TestBucketGet.java
=> TestBucketList.java} (80%)
create mode 100644
hadoop-ozone/s3gateway/src/test/resources/groupAccessControlList.xml
create mode 100644
hadoop-ozone/s3gateway/src/test/resources/userAccessControlList.xml
rename
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/{ExportContainer.java
=> container/ContainerCommands.java} (69%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/container/ExportSubcommand.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/container/InfoSubcommand.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/{shell/s3/S3Shell.java
=> debug/container/ListSubcommand.java} (50%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/container/package-info.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/{OmBucketGenerator.java
=> OmBucketRemover.java} (71%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/{OmBucketGenerator.java
=> OzoneClientKeyRemover.java} (62%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/s3/{GetS3SecretHandler.java
=> RevokeS3SecretHandler.java} (52%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
