This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch HDDS-2939
in repository https://gitbox.apache.org/repos/asf/ozone.git.
discard 186c580 HDDS-4358: Delete : make delete an atomic operation (#1607)
omit 28f2bbc HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
omit 8bd8e7a HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
omit 4124a34 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
omit fbd5707 HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
add 517358b HDDS-4488. Open RocksDB read only when loading containers at
Datanode startup (#1605)
add 49cd3ec HDDS-4432. Update Ratis version to latest snapshot. (#1586)
add fd879be HDDS-4476. Improve the ZH translation of the HA.md in doc.
(#1597)
add f71fc12 HDDS-4417. Simplify Ozone client code with configuration
object -- addendum (#1581)
add 6cc4a43 HDDS-4468. Fix Goofys listBucket large than 1000 objects will
stuck forever (#1595)
add 51ffc82 HDDS-4497. Recon File Size Count task throws SQL Exception.
(#1612)
add 1b2f2ef HDDS-3689. Add various profiles to MiniOzoneChaosCluster to
run different modes. (#1420)
add a9ff68a HDDS-4492. CLI flag --quota should default to 'spaceQuota' to
preserve backward compatibility. (#1609)
add 4b69f08 HDDS-4501. Reload OM State fail should terminate OM for any
exceptions. (#1622)
add 1a304ba HDDS-4392. [DOC] Add Recon architecture to docs (#1602)
add 54cca0b HDDS-4308. Fix issue with quota update (#1489)
add a4cd12c HDDS-4471. GrpcOutputStream length can overflow (#1617)
add fdb373f HDDS-4487. SCM can avoid using RETRIABLE_DATANODE_COMMAND for
datanode deletion commands. (#1621)
add d83ec1a HDDS-4481. With HA OM can send deletion blocks to SCM
multiple times. (#1608)
add 1235430 HDDS-4512. Remove unused netty3 transitive dependency (#1627)
add 43fdd71 HDDS-4370. Datanode deletion service can avoid storing
deleted blocks. (#1620)
add 5704341 HDDS-3363. Intermittent failure in testContainerImportExport
(#1618)
add 143f076 HDDS-4510. SCM can avoid creating
RetriableDatanodeEventWatcher for deletion command ACK (#1626)
add 130ba4d HDDS-4511: Avoiding StaleNodeHandler to take effect in
TestDeleteWithSlowFollower. (#1625)
add 536a6e7 HDDS-4461. Reuse compiled binaries in acceptance test (#1588)
add 9c6f805 HDDS-4429. Create unit test for SimpleContainerDownloader.
(#1551)
add b0d54d4 HDDS-4484. Use RaftServerImpl isLeader instead of periodic
leader update logic in OM and isLeaderReady for read/write requests (#1638)
add e970a46 HDDS-4424. Update README with information how to report
security issues (#1548)
add 02ba287 HDDS-4528. Upgrade slf4j to 1.7.30 (#1639)
add fc8b4ec HDDS-4485. [DOC] add the authentication rules of the Ozone
Ranger. (#1603)
add 9094463 HDDS-4473. Reduce number of sortDatanodes RPC calls (#1610)
add fd015cf HDDS-4529. Upgrade snakeyaml to 1.26 (#1640)
add f1e46cb HDDS-4320. Let Ozone input streams implement CanUnbuffer
(#1523)
add 7b6eafd HDDS-4356. SCM is flooded with useless "Deleting blocks"
messages. (#1604)
add 81cf297 HDDS-1982. Extend SCMNodeManager to support decommission and
maintenance states. Contributed by Stephen O'Donnell.
add 0858d83 Merge branch 'trunk' into HDDS-1880-Decom
add c09e0a1 Merge remote-tracking branch 'origin/trunk' into
HDDS-1880-Decom
add 5547d85 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add 1be2b03 Create admin commands and protobuf messages to allow
decommission / recommission and maintenance commands to be sent from the CLI
and update the node status in a skeleton decommission manager
add de928c9 Updates after review comments
add 218afe2 HDDS-2196. Add CLI Commands and Protobuf messages to trigger
decom states
add 4f58c81 HDDS-2197. Extend SCMCLI Topology command to print node
Operational States
add 9aa2282 Merge branch 'master' into HDDS-1880-Decom
add f4e3fd1 HDDS-2349. QueryNode does not respect null values for
opState or state
add a74cbfa Merge branch 'HDDS-1880-Decom' of
https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
add 278699e HDDS-2329. Destroy pipelines on any decommission or
maintenance nodes Signed-off-by: Anu Engineer <[email protected]>
add 88147d0 Merge branch 'master' into HDDS-1880-Decom
add 0c97903 Merged master into branch
add 920b8c5 HDDS-2459. Refactor ReplicationManager to consider
maintenance states
add 92e8d47 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add af2efa5 HDDS-2593. DatanodeAdminMonitor should track under replicated
containers and complete the admin workflow accordingly
add f66c40c HDDS-2728. Remove methods of internal representation from
DatanodeAdminMontor interface (#355)
add ec1df69 HDDS-2607. DeadNodeHandler should not remove replica for a
dead maintenance node (#343)
add 692420f HDDS-2671. Have NodeManager.getNodeStatus throw
NodeNotFoundException (#328)
add 64e25a9 Merge branch 'master' into HDDS-1880-Decom
add 5b228c0 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states
add a251d45 Revert "HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states"
add 265fba4 HDDS-2113. Update JMX metrics for node count in
SCMNodeMetrics for Decommission and Maintenance
add 9506aa9 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states #433
add d902586 HDDS-2592. Add Datanode command to allow the datanode to
persist its admin state (#521)
add 1e431b2 Merge branch 'master' into HDDS-1880-Decom
add 637853d Address issues cause by merge.
add 6fc7904 HDDS-3361. Remove ContainerReplica states representing
decomission and maintenance (#789)
add 8cbcefe merge master into HDDS-1880-Decom
add 2300b9f Merge Master into HDDS-1880-Decom (additional fixes)
add f43a370 HDDS-4300. Removed unneeded class DatanodeAdminNodeDetails
(#1465)
add 5ab169a Merge master into decom branch
add f74d73a HDDS-4322. Add integration tests for Decommission and resolve
issues detected by the tests. (#1484)
add 2504fab Merge master into HDDS-1880-Decom
add 39f5dc1 Removed unused variable detected by findbugs
add d327b7b HDDS-4324. DatanodeAdminMonitor no longers needs maintenance
end time to be passed (#1500)
add 42d53bc HDDS-4323. Add integration tests for putting nodes into
maintenance and fix any issues uncovered in the tests (#1501)
add f64476c HDDS-4340. Add Operational State to the datanode list command
(#1491)
add 18ed003 Merge branch 'master' into HDDS-1880-Decom
add 4c5f21b Merge branch 'master' into HDDS-1880-Decom
add 92369fc Merge Decommission branch into master
add 2c41192 Trigger build
add d903a98 merged master into branch
add 21c387d Address intermittent test failures
add d00bc1d Merge branch 'master' into HDDS-1880-Merge
add 20202f6 Trigger CI
add c296f78 Trigger CI
add cc5eebe Trigger CI
add d811ed7 Trigger build
add 062ad25 Trigger CI
add ca2e991 Merge branch 'master' into HDDS-1880-Merge
add 43f03eb Address compile issues after merging master into
HDDS-1880-Decom
add 3280f3c Merge branch 'HDDS-1880-Decom'
add e6007b1 HDDS-4543. Failed to list keys when there are two bucket
names with same prefix. (#1653)
add 5eae96c HDDS-4547. Upgrade aws-java-sdk to 1.11.901 (#1654)
add 8387887 HDDS-4535. Use fixed thread pool for closed container
replication (#1644)
add db10bae HDDS-4549. Fix typos in documents (#1655)
add e7cb0fb HDDS-4396. Ozone TLP - update documents (#1661)
new 4b2c562 HDDS-2949: mkdir : store directory entries in a separate
table (#1404)
new 1f9c9e8 HDDS-4266: CreateFile : store parent dir entries into
DirTable and file entry into separate FileTable (#1473)
new d086b57 HDDS-4332: ListFileStatus - do lookup in directory and file
tables (#1503)
new d919785 HDDS-4357: Rename : make rename an atomic ops by updating key
path entry in dir/file table (#1557)
new 2611af0 HDDS-4358: Delete : make delete an atomic operation (#1607)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (186c580)
\
N -- N -- N refs/heads/HDDS-2939 (2611af0)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 90 +--
HISTORY.md | 4 +-
README.md | 1 +
SECURITY.md | 23 +
hadoop-hdds/client/pom.xml | 5 -
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 5 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 12 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 111 +++-
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 67 +-
.../hdds/scm/storage/DummyBlockInputStream.java | 31 +-
.../storage/DummyBlockInputStreamWithRetry.java | 4 +-
.../hdds/scm/storage/DummyChunkInputStream.java | 24 +-
.../hdds/scm/storage/TestBlockInputStream.java | 112 +++-
.../hdds/scm/storage/TestChunkInputStream.java | 62 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 101 ++-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 12 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
.../apache/hadoop/hdds/scm/client/ScmClient.java | 45 +-
.../protocol/StorageContainerLocationProtocol.java | 18 +-
.../common/src/main/resources/ozone-default.xml | 22 +-
.../hadoop/hdds/protocol/MockDatanodeDetails.java | 4 +
hadoop-hdds/container-service/pom.xml | 11 +-
.../container/common/helpers/DatanodeIdYaml.java | 35 +
.../container/common/impl/ContainerDataYaml.java | 4 +-
.../common/statemachine/DatanodeStateMachine.java | 2 +
.../CreatePipelineCommandHandler.java | 2 +-
.../SetNodeOperationalStateCommandHandler.java | 157 +++++
.../states/endpoint/HeartbeatEndpointTask.java | 12 +
.../server/ratis/ContainerStateMachine.java | 4 +-
.../container/common/utils/ContainerCache.java | 2 +-
.../ozone/container/common/volume/HddsVolume.java | 4 +-
.../container/keyvalue/KeyValueContainer.java | 3 +
.../container/keyvalue/helpers/BlockUtils.java | 12 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 7 +-
.../background/BlockDeletingService.java | 19 +-
.../container/metadata/AbstractDatanodeStore.java | 6 +-
.../metadata/DatanodeStoreSchemaOneImpl.java | 7 +-
.../metadata/DatanodeStoreSchemaTwoImpl.java | 7 +-
.../container/replication/GrpcOutputStream.java | 2 +-
.../replication/ReplicationSupervisor.java | 17 +-
.../replication/SimpleContainerDownloader.java | 80 ++-
.../commands/SetNodeOperationalStateCommand.java | 89 +++
.../container/common/TestBlockDeletingService.java | 326 +++++-----
.../ozone/container/common/TestContainerCache.java | 2 +-
.../TestSchemaOneBackwardsCompatibility.java | 83 ++-
.../TestCreatePipelineCommandHandler.java | 13 +-
.../container/keyvalue/TestKeyValueContainer.java | 1 -
.../ReplicationSupervisorScheduling.java | 125 ++++
.../replication/TestSimpleContainerDownloader.java | 127 +++-
hadoop-hdds/docs/README.md | 2 +-
hadoop-hdds/docs/content/_index.md | 2 +-
hadoop-hdds/docs/content/_index.zh.md | 2 +-
hadoop-hdds/docs/content/concept/OzoneManager.md | 4 +-
hadoop-hdds/docs/content/concept/Recon.md | 163 +++++
.../docs/content/concept/ReconHighLevelDesign.png | Bin 0 -> 239168 bytes
hadoop-hdds/docs/content/concept/ReconOmDesign.png | Bin 0 -> 162797 bytes
.../docs/content/concept/ReconScmDesign.png | Bin 0 -> 181628 bytes
.../content/concept/StorageContainerManager.md | 5 +-
hadoop-hdds/docs/content/feature/HA.zh.md | 6 +-
hadoop-hdds/docs/content/feature/Quota.md | 9 +-
hadoop-hdds/docs/content/feature/Quota.zh.md | 8 +-
hadoop-hdds/docs/content/feature/Recon.md | 26 +-
hadoop-hdds/docs/content/interface/ReconApi.md | 511 +++++++++++++++
.../docs/content/security/SecurityWithRanger.md | 21 +-
.../docs/content/security/SecurityWithRanger.zh.md | 18 +-
hadoop-hdds/docs/pom.xml | 4 +-
hadoop-hdds/docs/static/ozone-logo-monochrome.svg | 4 +-
.../themes/ozonedoc/layouts/partials/header.html | 4 +-
.../themes/ozonedoc/layouts/partials/navbar.html | 4 +-
hadoop-hdds/framework/pom.xml | 20 +-
.../hdds/conf/DatanodeRatisServerConfig.java | 2 +-
...inerLocationProtocolClientSideTranslatorPB.java | 85 ++-
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 9 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 14 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 6 +
hadoop-hdds/hadoop-dependency-test/pom.xml | 12 +
.../src/main/proto/ScmAdminProtocol.proto | 47 +-
.../interface-client/src/main/proto/hdds.proto | 13 +-
.../interface-client/src/main/resources/proto.lock | 8 -
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 8 +
hadoop-hdds/pom.xml | 3 +
hadoop-hdds/server-scm/pom.xml | 6 +
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 4 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 6 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 28 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 11 +-
.../hdds/scm/container/ContainerReplicaCount.java | 271 ++++++++
.../hdds/scm/container/ReplicationManager.java | 304 ++++++---
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 6 +
.../DatanodeAdminMonitor.java} | 20 +-
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 371 +++++++++++
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 50 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 7 +-
...e-info.java => InvalidHostStringException.java} | 31 +-
...ge-info.java => InvalidNodeStateException.java} | 31 +-
.../hadoop/hdds/scm/node/NewNodeHandler.java | 20 +
.../hdds/scm/node/NodeDecommissionManager.java | 369 +++++++++++
.../apache/hadoop/hdds/scm/node/NodeManager.java | 59 +-
.../hadoop/hdds/scm/node/NodeManagerMXBean.java | 2 +-
.../hadoop/hdds/scm/node/NodeStateManager.java | 317 +++++----
.../apache/hadoop/hdds/scm/node/NodeStatus.java | 206 ++++++
.../hadoop/hdds/scm/node/SCMNodeManager.java | 241 +++++--
.../hadoop/hdds/scm/node/SCMNodeMetrics.java | 100 +--
...Handler.java => StartDatanodeAdminHandler.java} | 24 +-
.../hadoop/hdds/scm/node/states/NodeStateMap.java | 244 +++++--
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 3 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineUtils.java | 4 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 1 -
...inerLocationProtocolServerSideTranslatorPB.java | 61 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 97 ++-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 8 +
.../hdds/scm/server/StorageContainerManager.java | 44 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 5 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 77 ++-
.../hdds/scm/container/SimpleMockNodeManager.java | 323 ++++++++++
.../scm/container/TestContainerReportHandler.java | 18 +-
.../hdds/scm/container/TestReplicationManager.java | 285 ++++++++-
.../scm/container/TestUnknownContainerReport.java | 4 +-
.../algorithms/TestContainerPlacementFactory.java | 13 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 18 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../states/TestContainerReplicaCount.java | 465 ++++++++++++++
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hdds/scm/node/TestDatanodeAdminMonitor.java | 530 +++++++++++++++
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 29 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 297 +++++++++
.../hadoop/hdds/scm/node/TestNodeStateManager.java | 308 +++++++++
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 214 +++++--
.../hdds/scm/node/states/TestNodeStateMap.java | 140 ++++
.../TestPipelineDatanodesIntersection.java | 3 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 9 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 5 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 3 +-
.../placement/TestContainerPlacement.java | 6 +-
.../testutils/ReplicationNodeManagerMock.java | 85 ++-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 77 ++-
hadoop-hdds/tools/pom.xml | 6 +
.../hdds/scm/cli/ContainerOperationClient.java | 35 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 42 +-
.../hdds/scm/cli/datanode/DatanodeCommands.java | 5 +-
.../DecommissionSubCommand.java} | 27 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 48 +-
.../MaintenanceSubCommand.java} | 36 +-
.../RecommissionSubCommand.java} | 31 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 119 ++++
.../apache/hadoop/ozone/client/OzoneVolume.java | 17 -
.../ozone/client/io/BlockOutputStreamEntry.java | 4 -
.../hadoop/ozone/client/io/KeyInputStream.java | 11 +-
.../hadoop/ozone/client/io/OzoneInputStream.java | 11 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 12 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 7 -
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 22 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 33 +-
hadoop-ozone/datanode/pom.xml | 2 +-
.../dist/src/main/smoketest/topology/cli.robot | 4 +-
.../mini-chaos-tests/src/test/bin/start-chaos.sh | 2 +-
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 5 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 8 +-
.../apache/hadoop/ozone/OzoneChaosCluster.java} | 37 +-
.../ozone/TestAllMiniChaosOzoneCluster.java} | 32 +-
.../ozone/TestDatanodeMiniChaosOzoneCluster.java} | 34 +-
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 100 ++-
.../TestOzoneManagerMiniChaosOzoneCluster.java} | 38 +-
.../hadoop/ozone/failure/FailureManager.java | 3 +-
.../org/apache/hadoop/ozone/failure/Failures.java | 13 +
.../ozone/loadgenerators/AgedDirLoadGenerator.java | 1 -
.../ozone/loadgenerators/AgedLoadGenerator.java | 6 +-
.../loadgenerators/FilesystemLoadGenerator.java | 1 -
.../{utils => loadgenerators}/LoadBucket.java | 2 +-
.../hadoop/ozone/loadgenerators/LoadGenerator.java | 17 +
.../loadgenerators/NestedDirLoadGenerator.java | 1 -
.../loadgenerators/RandomDirLoadGenerator.java | 1 -
.../ozone/loadgenerators/RandomLoadGenerator.java | 1 -
.../loadgenerators/ReadOnlyLoadGenerator.java | 1 -
.../apache/hadoop/ozone/utils/TestProbability.java | 43 --
.../src/test/resources/log4j.properties | 2 +-
hadoop-ozone/integration-test/pom.xml | 6 +
.../fs/contract/AbstractContractUnbufferTest.java | 159 +++++
...DistCp.java => ITestOzoneContractUnbuffer.java} | 26 +-
....java => ITestRootedOzoneContractUnbuffer.java} | 27 +-
.../TestRatisPipelineCreateAndDestroy.java | 3 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 2 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 3 +-
.../hadoop/ozone/TestMiniOzoneHACluster.java | 2 +-
.../hadoop/ozone/TestStorageContainerManager.java | 5 +-
.../ozone/client/rpc/TestBlockOutputStream.java | 12 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 11 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 7 +
.../ozone/client/rpc/TestKeyInputStream.java | 210 ++++--
.../rpc/TestOzoneClientRetriesOnException.java | 13 +-
.../ozone/client/rpc/TestOzoneRpcClient.java | 2 -
.../client/rpc/TestOzoneRpcClientAbstract.java | 296 +--------
.../client/rpc/TestOzoneRpcClientWithRatis.java | 2 -
.../apache/hadoop/ozone/container/TestHelper.java | 70 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 47 ++
.../ozone/om/TestOzoneManagerHAKeyDeletion.java | 77 +++
.../ozone/om/TestOzoneManagerHAWithData.java | 37 --
.../ozone/recon/TestReconWithOzoneManagerHA.java | 2 +-
.../hadoop/ozone/scm/TestSCMNodeManagerMXBean.java | 30 +-
.../scm/node/TestDecommissionAndMaintenance.java | 708 +++++++++++++++++++++
.../hadoop/ozone/scm/node/TestQueryNode.java | 67 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 42 ++
.../src/test/resources/contract/ozone.xml | 5 +
.../src/main/proto/OmClientProtocol.proto | 1 -
hadoop-ozone/interface-storage/pom.xml | 14 +-
hadoop-ozone/ozone-manager/pom.xml | 6 +
.../apache/hadoop/ozone/om/KeyDeletingService.java | 21 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 98 ++-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 3 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 28 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 219 ++-----
.../ozone/om/ratis/OzoneManagerStateMachine.java | 3 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 15 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 8 +-
.../om/request/key/OMAllocateBlockRequest.java | 16 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 10 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 7 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 7 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 23 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 7 +-
.../multipart/S3MultipartUploadAbortRequest.java | 5 +-
.../S3MultipartUploadCommitPartRequest.java | 10 +-
.../om/response/key/OMAllocateBlockResponse.java | 4 -
.../ozone/om/response/key/OMKeyCommitResponse.java | 4 -
.../ozone/om/response/key/OMKeyCreateResponse.java | 4 -
.../ozone/om/response/key/OMKeyDeleteResponse.java | 4 -
.../om/response/key/OMKeysDeleteResponse.java | 4 -
.../multipart/S3MultipartUploadAbortResponse.java | 4 -
.../S3MultipartUploadCommitPartResponse.java | 5 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 57 +-
.../apache/hadoop/ozone/om/TestChunkStreams.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 119 +++-
.../hadoop/ozone/om/TestOmMetadataManager.java | 17 +-
.../om/ratis/TestOzoneManagerStateMachine.java | 10 +-
.../bucket/TestOMBucketSetPropertyRequest.java | 3 +-
.../volume/TestOMVolumeSetQuotaRequest.java | 3 +-
.../hadoop/fs/ozone/CapableOzoneFSInputStream.java | 3 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 10 +-
.../hadoop/fs/ozone/OzoneStreamCapabilities.java | 9 +
hadoop-ozone/pom.xml | 7 +
hadoop-ozone/recon/pom.xml | 6 +
.../ozone/recon/api/ClusterStateEndpoint.java | 5 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 7 +-
.../persistence/DefaultDataSourceProvider.java | 6 +
.../recon/persistence/DerbyDataSourceProvider.java | 1 -
.../ozone/recon/tasks/FileSizeCountTask.java | 4 +-
.../hadoop/ozone/recon/tasks/ReconTaskConfig.java | 4 +-
.../persistence/TestReconWithDifferentSqlDBs.java | 5 +
.../ozone/recon/tasks/TestFileSizeCountTask.java | 6 +
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 8 +-
.../ozone/s3/endpoint/ListObjectResponse.java | 11 +
hadoop-ozone/tools/pom.xml | 2 +-
.../freon/FollowerAppendLogEntryGenerator.java | 11 +-
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 20 +-
.../hadoop/ozone/shell/SetSpaceQuotaOptions.java | 3 +-
pom.xml | 14 +-
tools/fault-injection-service/README.md | 2 +-
260 files changed, 10027 insertions(+), 2370 deletions(-)
create mode 100644 SECURITY.md
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/SetNodeOperationalStateCommandHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/SetNodeOperationalStateCommand.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/ReplicationSupervisorScheduling.java
create mode 100644 hadoop-hdds/docs/content/concept/Recon.md
create mode 100644 hadoop-hdds/docs/content/concept/ReconHighLevelDesign.png
create mode 100644 hadoop-hdds/docs/content/concept/ReconOmDesign.png
create mode 100644 hadoop-hdds/docs/content/concept/ReconScmDesign.png
create mode 100644 hadoop-hdds/docs/content/interface/ReconApi.md
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaCount.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/{pipeline/InsufficientDatanodesException.java
=> node/DatanodeAdminMonitor.java} (64%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeAdminMonitorImpl.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidHostStringException.java} (68%)
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidNodeStateException.java} (68%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStatus.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{StaleNodeHandler.java
=> StartDatanodeAdminHandler.java} (75%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/SimpleMockNodeManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/states/TestContainerReplicaCount.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDatanodeAdminMonitor.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/states/TestNodeStateMap.java
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{pipeline/ClosePipelineSubcommand.java
=> datanode/DecommissionSubCommand.java} (70%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/MaintenanceSubCommand.java} (62%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/RecommissionSubCommand.java} (69%)
create mode 100644
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/datanode/TestListInfoSubcommand.java
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/segmentparser/RatisLogParser.java
=>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/OzoneChaosCluster.java}
(60%)
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/segmentparser/GenericRatisLogParser.java
=>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestAllMiniChaosOzoneCluster.java}
(64%)
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/segmentparser/GenericRatisLogParser.java
=>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestDatanodeMiniChaosOzoneCluster.java}
(61%)
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/segmentparser/GenericRatisLogParser.java
=>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/TestOzoneManagerMiniChaosOzoneCluster.java}
(54%)
rename
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/{utils
=> loadgenerators}/LoadBucket.java (99%)
delete mode 100644
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/TestProbability.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/contract/AbstractContractUnbufferTest.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/{ITestOzoneContractDistCp.java
=> ITestOzoneContractUnbuffer.java} (72%)
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/rooted/{ITestRootedOzoneContractGetFileStatus.java
=> ITestRootedOzoneContractUnbuffer.java} (77%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerHAKeyDeletion.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/node/TestDecommissionAndMaintenance.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
