This is an automated email from the ASF dual-hosted git repository.
nanda pushed a change to branch HDDS-2823
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 8b16a35 HDDS-4133. Use new ContainerManager in SCM. (#1378)
add 536a6e7 HDDS-4461. Reuse compiled binaries in acceptance test (#1588)
add 9c6f805 HDDS-4429. Create unit test for SimpleContainerDownloader.
(#1551)
add b0d54d4 HDDS-4484. Use RaftServerImpl isLeader instead of periodic
leader update logic in OM and isLeaderReady for read/write requests (#1638)
add e970a46 HDDS-4424. Update README with information how to report
security issues (#1548)
add 02ba287 HDDS-4528. Upgrade slf4j to 1.7.30 (#1639)
add fc8b4ec HDDS-4485. [DOC] add the authentication rules of the Ozone
Ranger. (#1603)
add 9094463 HDDS-4473. Reduce number of sortDatanodes RPC calls (#1610)
add fd015cf HDDS-4529. Upgrade snakeyaml to 1.26 (#1640)
add f1e46cb HDDS-4320. Let Ozone input streams implement CanUnbuffer
(#1523)
add 7b6eafd HDDS-4356. SCM is flooded with useless "Deleting blocks"
messages. (#1604)
add 81cf297 HDDS-1982. Extend SCMNodeManager to support decommission and
maintenance states. Contributed by Stephen O'Donnell.
add 0858d83 Merge branch 'trunk' into HDDS-1880-Decom
add c09e0a1 Merge remote-tracking branch 'origin/trunk' into
HDDS-1880-Decom
add 5547d85 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add 1be2b03 Create admin commands and protobuf messages to allow
decommission / recommission and maintenance commands to be sent from the CLI
and update the node status in a skeleton decommission manager
add de928c9 Updates after review comments
add 218afe2 HDDS-2196. Add CLI Commands and Protobuf messages to trigger
decom states
add 4f58c81 HDDS-2197. Extend SCMCLI Topology command to print node
Operational States
add 9aa2282 Merge branch 'master' into HDDS-1880-Decom
add f4e3fd1 HDDS-2349. QueryNode does not respect null values for
opState or state
add a74cbfa Merge branch 'HDDS-1880-Decom' of
https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
add 278699e HDDS-2329. Destroy pipelines on any decommission or
maintenance nodes Signed-off-by: Anu Engineer <[email protected]>
add 88147d0 Merge branch 'master' into HDDS-1880-Decom
add 0c97903 Merged master into branch
add 920b8c5 HDDS-2459. Refactor ReplicationManager to consider
maintenance states
add 92e8d47 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add af2efa5 HDDS-2593. DatanodeAdminMonitor should track under replicated
containers and complete the admin workflow accordingly
add f66c40c HDDS-2728. Remove methods of internal representation from
DatanodeAdminMontor interface (#355)
add ec1df69 HDDS-2607. DeadNodeHandler should not remove replica for a
dead maintenance node (#343)
add 692420f HDDS-2671. Have NodeManager.getNodeStatus throw
NodeNotFoundException (#328)
add 64e25a9 Merge branch 'master' into HDDS-1880-Decom
add 5b228c0 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states
add a251d45 Revert "HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states"
add 265fba4 HDDS-2113. Update JMX metrics for node count in
SCMNodeMetrics for Decommission and Maintenance
add 9506aa9 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states #433
add d902586 HDDS-2592. Add Datanode command to allow the datanode to
persist its admin state (#521)
add 1e431b2 Merge branch 'master' into HDDS-1880-Decom
add 637853d Address issues cause by merge.
add 6fc7904 HDDS-3361. Remove ContainerReplica states representing
decomission and maintenance (#789)
add 8cbcefe merge master into HDDS-1880-Decom
add 2300b9f Merge Master into HDDS-1880-Decom (additional fixes)
add f43a370 HDDS-4300. Removed unneeded class DatanodeAdminNodeDetails
(#1465)
add 5ab169a Merge master into decom branch
add f74d73a HDDS-4322. Add integration tests for Decommission and resolve
issues detected by the tests. (#1484)
add 2504fab Merge master into HDDS-1880-Decom
add 39f5dc1 Removed unused variable detected by findbugs
add d327b7b HDDS-4324. DatanodeAdminMonitor no longers needs maintenance
end time to be passed (#1500)
add 42d53bc HDDS-4323. Add integration tests for putting nodes into
maintenance and fix any issues uncovered in the tests (#1501)
add f64476c HDDS-4340. Add Operational State to the datanode list command
(#1491)
add 18ed003 Merge branch 'master' into HDDS-1880-Decom
add 4c5f21b Merge branch 'master' into HDDS-1880-Decom
add 92369fc Merge Decommission branch into master
add 2c41192 Trigger build
add d903a98 merged master into branch
add 21c387d Address intermittent test failures
add d00bc1d Merge branch 'master' into HDDS-1880-Merge
add 20202f6 Trigger CI
add c296f78 Trigger CI
add cc5eebe Trigger CI
add d811ed7 Trigger build
add 062ad25 Trigger CI
add ca2e991 Merge branch 'master' into HDDS-1880-Merge
add 43f03eb Address compile issues after merging master into
HDDS-1880-Decom
add 3280f3c Merge branch 'HDDS-1880-Decom'
add e6007b1 HDDS-4543. Failed to list keys when there are two bucket
names with same prefix. (#1653)
add 5eae96c HDDS-4547. Upgrade aws-java-sdk to 1.11.901 (#1654)
add 8387887 HDDS-4535. Use fixed thread pool for closed container
replication (#1644)
add db10bae HDDS-4549. Fix typos in documents (#1655)
add e7cb0fb HDDS-4396. Ozone TLP - update documents (#1661)
add 87b97fa HDDS-4447. SCMBlockDeletingService should handle
ContainerNotFoundException (#1643)
add 404aa6b HDDS-4555. Delete a duplicate character in ZH Docs homepage
(#1666)
add c215b11 HDDS-4444. Remove KeyManagerImpl#refreshPipeline because it
is the same as refresh(). (#1593)
add 4669043 HDDS-4329. Expose Ratis retry config cache in OM. (#1659)
add 9ae2a3c HDDS-4537. Remove refreshPipeline in listKeys. (#1660)
add cdc2c13 HDDS-4566. inconsistencies Port describtion in Monitoring
with Prometheus document (#1678)
add 5471432 HDDS-4554. numKey metrics goes negative after intermediate
directory deletion. (#1665)
add 7eefe24 HDDS-4559. Avoid using hard coding uft-8 charset (#1673)
add ef5fc49 HDDS-4499. Upgrade Spring Framework to more recent 5.2 (#1616)
add cd92b6d HDDS-4572. Simplify inequality condition (#1681)
add 1e57d9e HDDS-4265. Refactor OzoneQuota to make it easy to support
more quota … (#1647)
add d02659c HDDS-4491. Deprecate raft.server.rpcslowness.timeout config
key (#1624)
add 9d9db48 HDDS-4526. Remove false-positive error logs from LeaseManager
(#1637)
add 6677200 HDDS-4451.Handle start & stop of Trash Emptier thread when
node becomes leader/follower. (#1615)
add 19dd94d HDDS-4313. Create thread-local instance of FileSystem in
HadoopFsGenerator (#1479)
add f30aba7 HDDS-4503. Provide info on block size via FileSystem (#1619)
add 8ded841 HDDS-4505. Increase default value for SCM heartbeat timeout
to 5s. (#1634)
add e0c75ce HDDS-4579. Add acceptance test for Ozone Client Key Validator
(#1691)
add 1f5a965 HDDS-4272. Volume namespace: add usedNamespace and update it
when create and delete bucket (#1445)
add e0c8556 HDDS-4584. Coverage not updated since TLP (#1698)
add e0ce757 HDDS-4581. Cleanup usage of volumeArgs in KeyRequests. (#1693)
add 0152eb5 HDDS-4404. Datanode can go OOM when a Recon or SCM Server is
very slow in processing reports (#1601)
add 327c148 HDDS-4013. FLAKY-UT:
TestWatchForCommit#testWatchForCommitForGroupMismatchException (#1697)
add 81b75fd HDDS-4580. Datanode can be stuck in leader not ready state
after restart (#1690)
add dfd2aaf HDDS-4576. TestDefaultCertificateClient misuses chars param
of random() (#1684)
add 329bcc4 HDDS-4582 Update `quota in counts` to `quota in namespace`
(#1696)
add 0957975 HDDS-4519. Return forbidden instead of interval server error
from s3g… (#1642)
new 2c3aed3 Merge branch 'master' into HDDS-2823
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 94 +--
HISTORY.md | 4 +-
README.md | 1 +
SECURITY.md | 23 +
.../hadoop/hdds/scm/storage/BlockInputStream.java | 111 +++-
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 67 +-
.../hdds/scm/storage/DummyBlockInputStream.java | 31 +-
.../storage/DummyBlockInputStreamWithRetry.java | 4 +-
.../hdds/scm/storage/DummyChunkInputStream.java | 24 +-
.../hdds/scm/storage/TestBlockInputStream.java | 112 +++-
.../hdds/scm/storage/TestChunkInputStream.java | 62 +-
.../org/apache/hadoop/hdds/client/OzoneQuota.java | 135 ++--
.../org/apache/hadoop/hdds/client/QuotaList.java | 67 ++
.../hadoop/hdds/conf/OzoneConfiguration.java | 14 +
.../hadoop/hdds/protocol/DatanodeDetails.java | 101 ++-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 7 +-
.../apache/hadoop/hdds/scm/client/ScmClient.java | 45 +-
.../protocol/StorageContainerLocationProtocol.java | 18 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +-
.../apache/hadoop/ozone/lease/LeaseManager.java | 38 +-
.../common/src/main/resources/ozone-default.xml | 24 +-
.../hadoop/hdds/protocol/MockDatanodeDetails.java | 4 +
.../hadoop/hdds/conf/ConfigFileAppender.java | 4 +-
.../container/common/helpers/DatanodeIdYaml.java | 38 +-
.../ozone/container/common/impl/ContainerData.java | 2 +-
.../container/common/impl/ContainerDataYaml.java | 4 +-
.../container/common/report/ReportPublisher.java | 2 +-
.../common/statemachine/DatanodeStateMachine.java | 2 +
.../common/statemachine/StateContext.java | 143 ++++-
.../SetNodeOperationalStateCommandHandler.java | 157 +++++
.../states/endpoint/HeartbeatEndpointTask.java | 28 +-
.../server/ratis/ContainerStateMachine.java | 9 +-
.../replication/ReplicationSupervisor.java | 17 +-
.../replication/SimpleContainerDownloader.java | 80 ++-
.../commands/SetNodeOperationalStateCommand.java | 89 +++
.../common/statemachine/TestStateContext.java | 286 ++++++++-
.../TestCreatePipelineCommandHandler.java | 6 +-
.../container/keyvalue/TestTarContainerPacker.java | 11 +-
.../ReplicationSupervisorScheduling.java | 125 ++++
.../replication/TestSimpleContainerDownloader.java | 127 +++-
.../org.mockito.plugins.MockMaker | 0
hadoop-hdds/docs/README.md | 2 +-
hadoop-hdds/docs/content/_index.md | 2 +-
hadoop-hdds/docs/content/_index.zh.md | 4 +-
hadoop-hdds/docs/content/recipe/Prometheus.md | 6 +-
hadoop-hdds/docs/content/recipe/Prometheus.zh.md | 6 +-
.../docs/content/security/SecurityWithRanger.md | 21 +-
.../docs/content/security/SecurityWithRanger.zh.md | 18 +-
hadoop-hdds/docs/pom.xml | 4 +-
hadoop-hdds/docs/static/ozone-logo-monochrome.svg | 4 +-
.../themes/ozonedoc/layouts/partials/header.html | 4 +-
.../themes/ozonedoc/layouts/partials/navbar.html | 4 +-
...inerLocationProtocolClientSideTranslatorPB.java | 85 ++-
.../hadoop/hdds/server/http/HtmlQuoting.java | 2 +-
.../hadoop/hdds/server/http/HttpServer2.java | 3 +-
.../client/TestDefaultCertificateClient.java | 41 +-
.../apache/hadoop/hdds/server/TestJsonUtils.java | 2 +-
.../src/main/proto/ScmAdminProtocol.proto | 47 +-
.../interface-client/src/main/proto/hdds.proto | 13 +-
.../interface-client/src/main/resources/proto.lock | 8 -
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 8 +
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 4 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 6 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 74 ++-
.../hdds/scm/block/SCMBlockDeletingService.java | 9 +-
.../hdds/scm/container/ContainerReplicaCount.java | 271 ++++++++
.../hdds/scm/container/ReplicationManager.java | 302 ++++++---
.../hdds/scm/container/SCMContainerManager.java | 31 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 6 +
.../hdds/scm/metadata/X509CertificateCodec.java | 6 +-
.../DatanodeAdminMonitor.java} | 20 +-
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 371 +++++++++++
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 50 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 7 +-
...e-info.java => InvalidHostStringException.java} | 31 +-
...ge-info.java => InvalidNodeStateException.java} | 31 +-
.../hadoop/hdds/scm/node/NewNodeHandler.java | 16 +-
.../hdds/scm/node/NodeDecommissionManager.java | 369 +++++++++++
.../apache/hadoop/hdds/scm/node/NodeManager.java | 59 +-
.../hadoop/hdds/scm/node/NodeManagerMXBean.java | 2 +-
.../hadoop/hdds/scm/node/NodeStateManager.java | 317 +++++----
.../apache/hadoop/hdds/scm/node/NodeStatus.java | 206 ++++++
.../hadoop/hdds/scm/node/SCMNodeManager.java | 241 +++++--
.../hadoop/hdds/scm/node/SCMNodeMetrics.java | 100 +--
...Handler.java => StartDatanodeAdminHandler.java} | 24 +-
.../hadoop/hdds/scm/node/states/NodeStateMap.java | 244 +++++--
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 3 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 6 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 1 -
...inerLocationProtocolServerSideTranslatorPB.java | 61 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 97 ++-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 8 +
.../hdds/scm/server/StorageContainerManager.java | 35 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 5 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 77 ++-
.../hdds/scm/container/SimpleMockNodeManager.java | 323 ++++++++++
.../scm/container/TestContainerReportHandler.java | 18 +-
.../hdds/scm/container/TestReplicationManager.java | 285 ++++++++-
.../scm/container/TestUnknownContainerReport.java | 4 +-
.../algorithms/TestContainerPlacementFactory.java | 13 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 18 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../states/TestContainerReplicaCount.java | 465 ++++++++++++++
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hdds/scm/node/TestDatanodeAdminMonitor.java | 530 +++++++++++++++
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 30 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 297 +++++++++
.../hadoop/hdds/scm/node/TestNodeStateManager.java | 308 +++++++++
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 214 +++++--
.../hdds/scm/node/states/TestNodeStateMap.java | 140 ++++
.../TestPipelineDatanodesIntersection.java | 3 +-
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 4 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 9 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 5 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 3 +-
.../placement/TestContainerPlacement.java | 6 +-
.../testutils/ReplicationNodeManagerMock.java | 85 ++-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 77 ++-
hadoop-hdds/tools/pom.xml | 6 +
.../hdds/scm/cli/ContainerOperationClient.java | 35 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 42 +-
.../hdds/scm/cli/datanode/DatanodeCommands.java | 5 +-
.../DecommissionSubCommand.java} | 27 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 48 +-
.../MaintenanceSubCommand.java} | 36 +-
.../RecommissionSubCommand.java} | 31 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 119 ++++
.../org/apache/hadoop/ozone/client/BucketArgs.java | 22 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 48 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 62 +-
.../org/apache/hadoop/ozone/client/VolumeArgs.java | 22 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 11 +-
.../hadoop/ozone/client/io/OzoneInputStream.java | 11 +-
.../ozone/client/protocol/ClientProtocol.java | 12 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 38 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 9 +-
.../hadoop/ozone/om/helpers/OmBucketArgs.java | 28 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 32 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 73 ++-
.../ozone/om/protocol/OzoneManagerProtocol.java | 4 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 6 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 4 +-
.../org/apache/hadoop/ozone/csi/NodeService.java | 5 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 2 +-
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 8 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 32 +-
.../dist/src/main/smoketest/createbucketenv.robot | 2 +-
.../dist/src/main/smoketest/createmrenv.robot | 2 +-
.../src/main/smoketest/debug/ozone-debug.robot | 2 +-
.../dist/src/main/smoketest/freon/freon.robot | 37 --
.../{s3/buckethead.robot => freon/generate.robot} | 25 +-
.../{admincli/scmha.robot => freon/validate.robot} | 13 +-
.../dist/src/main/smoketest/gdpr/gdpr.robot | 2 +-
.../dist/src/main/smoketest/topology/cli.robot | 4 +-
.../fs/contract/AbstractContractUnbufferTest.java | 159 +++++
...Stream.java => TestOzoneFileSystemMetrics.java} | 127 ++--
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 4 +-
...DistCp.java => ITestOzoneContractUnbuffer.java} | 26 +-
....java => ITestRootedOzoneContractUnbuffer.java} | 27 +-
.../TestRatisPipelineCreateAndDestroy.java | 3 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 2 +-
.../hadoop/ozone/TestMiniOzoneHACluster.java | 2 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 3 +-
.../hadoop/ozone/TestStorageContainerManager.java | 5 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 4 +
.../ozone/client/rpc/TestKeyInputStream.java | 210 ++++--
.../client/rpc/TestOzoneAtRestEncryption.java | 20 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 65 +-
.../apache/hadoop/ozone/container/TestHelper.java | 75 ++-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 4 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 14 +
.../ozone/om/TestOzoneManagerHAMetadataOnly.java | 24 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 2 +-
.../hadoop/ozone/scm/TestSCMNodeManagerMXBean.java | 30 +-
.../scm/node/TestDecommissionAndMaintenance.java | 708 +++++++++++++++++++++
.../hadoop/ozone/scm/node/TestQueryNode.java | 67 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 2 -
.../src/test/resources/contract/ozone.xml | 5 +
.../src/main/proto/OmClientProtocol.proto | 10 +-
.../apache/hadoop/ozone/om/KeyDeletingService.java | 2 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 114 ++--
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 4 +
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 3 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 62 +-
.../org/apache/hadoop/ozone/om/OzoneTrash.java} | 32 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 42 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 179 ++----
.../om/ratis/OzoneManagerRatisServerConfig.java | 54 ++
.../ozone/om/ratis/OzoneManagerStateMachine.java | 1 -
.../om/request/bucket/OMBucketCreateRequest.java | 29 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 17 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 17 +-
.../om/request/file/OMDirectoryCreateRequest.java | 13 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 11 +-
.../om/request/key/OMAllocateBlockRequest.java | 6 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 6 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 12 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 5 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 25 -
.../ozone/om/request/key/OMKeysDeleteRequest.java | 5 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 7 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 14 +-
.../om/response/bucket/OMBucketCreateResponse.java | 17 +
.../om/response/bucket/OMBucketDeleteResponse.java | 18 +
.../om/response/file/OMFileCreateResponse.java | 5 +-
.../om/response/key/OMAllocateBlockResponse.java | 9 +-
.../ozone/om/response/key/OMKeyCommitResponse.java | 7 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 10 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 7 +-
.../om/response/key/OMKeysDeleteResponse.java | 8 +-
.../multipart/S3MultipartUploadAbortResponse.java | 8 +-
.../S3MultipartUploadCommitPartResponse.java | 8 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 57 +-
.../protocolPB/OzoneManagerRequestHandler.java | 2 +-
.../hadoop/ozone/security/AWSV4AuthValidator.java | 9 +-
.../apache/hadoop/ozone/om/TestChunkStreams.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 119 +++-
.../hadoop/ozone/om/TestOmMetadataManager.java | 17 +-
.../ozone/om/request/TestOMRequestUtils.java | 14 +-
.../bucket/TestOMBucketSetPropertyRequest.java | 2 +-
.../request/file/TestOMDirectoryCreateRequest.java | 2 +-
.../volume/TestOMVolumeSetQuotaRequest.java | 20 +-
.../response/key/TestOMAllocateBlockResponse.java | 11 +-
.../om/response/key/TestOMKeyCommitResponse.java | 7 +-
.../om/response/key/TestOMKeyCreateResponse.java | 7 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 10 +-
.../om/response/key/TestOMKeysDeleteResponse.java | 9 +-
.../s3/multipart/TestS3MultipartResponse.java | 6 +-
.../TestS3MultipartUploadAbortResponse.java | 18 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 2 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 9 +
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 2 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 9 +
.../hadoop/fs/ozone/CapableOzoneFSInputStream.java | 3 +-
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 2 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 10 +-
.../hadoop/fs/ozone/OzoneStreamCapabilities.java | 9 +
.../hadoop/fs/ozone/TestBasicOzoneFileSystems.java | 89 +++
hadoop-ozone/pom.xml | 2 +-
.../ozone/recon/api/ClusterStateEndpoint.java | 5 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 7 +-
.../hadoop/ozone/s3/AWSSignatureProcessor.java | 6 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 2 +-
.../apache/hadoop/ozone/s3/SignatureProcessor.java | 2 -
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 49 +-
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 29 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 23 +-
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 4 +
.../hadoop/ozone/client/ObjectStoreStub.java | 2 +-
.../hadoop/ozone/client/OzoneVolumeStub.java | 5 +-
.../ozone/s3/TestSignedChunksInputStream.java | 12 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 268 ++++++++
.../hadoop/ozone/freon/BaseFreonGenerator.java | 11 +-
.../hadoop/ozone/freon/HadoopFsGenerator.java | 45 +-
.../ozone/genesis/BenchMarkMetadataStoreReads.java | 8 +-
.../genesis/BenchMarkMetadataStoreWrites.java | 6 +-
.../ozone/genesis/BenchMarkRocksDbStore.java | 8 +-
.../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 2 -
.../hadoop/ozone/shell/ClearSpaceQuotaOptions.java | 2 +-
.../hadoop/ozone/shell/SetSpaceQuotaOptions.java | 8 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 4 +-
.../hadoop/ozone/shell/bucket/SetQuotaHandler.java | 8 +-
.../ozone/shell/volume/CreateVolumeHandler.java | 4 +-
.../hadoop/ozone/shell/volume/SetQuotaHandler.java | 8 +-
pom.xml | 6 +-
tools/fault-injection-service/README.md | 2 +-
269 files changed, 10375 insertions(+), 2187 deletions(-)
create mode 100644 SECURITY.md
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/QuotaList.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/SetNodeOperationalStateCommandHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/SetNodeOperationalStateCommand.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisorScheduling.java
copy {hadoop-ozone/recon =>
hadoop-hdds/container-service}/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
(100%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaCount.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/{pipeline/InsufficientDatanodesException.java
=> node/DatanodeAdminMonitor.java} (64%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeAdminMonitorImpl.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidHostStringException.java} (68%)
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidNodeStateException.java} (68%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStatus.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{StaleNodeHandler.java
=> StartDatanodeAdminHandler.java} (76%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/SimpleMockNodeManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/states/TestContainerReplicaCount.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDatanodeAdminMonitor.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/states/TestNodeStateMap.java
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{pipeline/ClosePipelineSubcommand.java
=> datanode/DecommissionSubCommand.java} (70%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/MaintenanceSubCommand.java} (62%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/RecommissionSubCommand.java} (69%)
create mode 100644
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/datanode/TestListInfoSubcommand.java
delete mode 100644 hadoop-ozone/dist/src/main/smoketest/freon/freon.robot
copy hadoop-ozone/dist/src/main/smoketest/{s3/buckethead.robot =>
freon/generate.robot} (55%)
copy hadoop-ozone/dist/src/main/smoketest/{admincli/scmha.robot =>
freon/validate.robot} (73%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/contract/AbstractContractUnbufferTest.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/{TestOzoneFSInputStream.java
=> TestOzoneFileSystemMetrics.java} (58%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/{ITestOzoneContractDistCp.java
=> ITestOzoneContractUnbuffer.java} (72%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/{ITestRootedOzoneContractGetFileStatus.java
=> ITestRootedOzoneContractUnbuffer.java} (77%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/node/TestDecommissionAndMaintenance.java
copy
hadoop-ozone/{s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/package-info.java
=> ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneTrash.java}
(59%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisServerConfig.java
create mode 100644
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestBasicOzoneFileSystems.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestPermissionCheck.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
