This is an automated email from the ASF dual-hosted git repository.
sodonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 7b6eafd HDDS-4356. SCM is flooded with useless "Deleting blocks"
messages. (#1604)
add 81cf297 HDDS-1982. Extend SCMNodeManager to support decommission and
maintenance states. Contributed by Stephen O'Donnell.
add 0858d83 Merge branch 'trunk' into HDDS-1880-Decom
add c09e0a1 Merge remote-tracking branch 'origin/trunk' into
HDDS-1880-Decom
add 5547d85 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add 1be2b03 Create admin commands and protobuf messages to allow
decommission / recommission and maintenance commands to be sent from the CLI
and update the node status in a skeleton decommission manager
add de928c9 Updates after review comments
add 218afe2 HDDS-2196. Add CLI Commands and Protobuf messages to trigger
decom states
add 4f58c81 HDDS-2197. Extend SCMCLI Topology command to print node
Operational States
add 9aa2282 Merge branch 'master' into HDDS-1880-Decom
add f4e3fd1 HDDS-2349. QueryNode does not respect null values for
opState or state
add a74cbfa Merge branch 'HDDS-1880-Decom' of
https://git-wip-us.apache.org/repos/asf/hadoop-ozone into HDDS-1880-Decom
add 278699e HDDS-2329. Destroy pipelines on any decommission or
maintenance nodes Signed-off-by: Anu Engineer <[email protected]>
add 88147d0 Merge branch 'master' into HDDS-1880-Decom
add 0c97903 Merged master into branch
add 920b8c5 HDDS-2459. Refactor ReplicationManager to consider
maintenance states
add 92e8d47 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
add af2efa5 HDDS-2593. DatanodeAdminMonitor should track under replicated
containers and complete the admin workflow accordingly
add f66c40c HDDS-2728. Remove methods of internal representation from
DatanodeAdminMontor interface (#355)
add ec1df69 HDDS-2607. DeadNodeHandler should not remove replica for a
dead maintenance node (#343)
add 692420f HDDS-2671. Have NodeManager.getNodeStatus throw
NodeNotFoundException (#328)
add 64e25a9 Merge branch 'master' into HDDS-1880-Decom
add 5b228c0 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states
add a251d45 Revert "HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states"
add 265fba4 HDDS-2113. Update JMX metrics for node count in
SCMNodeMetrics for Decommission and Maintenance
add 9506aa9 HDDS-2860. Cluster disk space metrics should reflect
decommission and maintenance states #433
add d902586 HDDS-2592. Add Datanode command to allow the datanode to
persist its admin state (#521)
add 1e431b2 Merge branch 'master' into HDDS-1880-Decom
add 637853d Address issues cause by merge.
add 6fc7904 HDDS-3361. Remove ContainerReplica states representing
decomission and maintenance (#789)
add 8cbcefe merge master into HDDS-1880-Decom
add 2300b9f Merge Master into HDDS-1880-Decom (additional fixes)
add f43a370 HDDS-4300. Removed unneeded class DatanodeAdminNodeDetails
(#1465)
add 5ab169a Merge master into decom branch
add f74d73a HDDS-4322. Add integration tests for Decommission and resolve
issues detected by the tests. (#1484)
add 2504fab Merge master into HDDS-1880-Decom
add 39f5dc1 Removed unused variable detected by findbugs
add d327b7b HDDS-4324. DatanodeAdminMonitor no longers needs maintenance
end time to be passed (#1500)
add 42d53bc HDDS-4323. Add integration tests for putting nodes into
maintenance and fix any issues uncovered in the tests (#1501)
add f64476c HDDS-4340. Add Operational State to the datanode list command
(#1491)
add 18ed003 Merge branch 'master' into HDDS-1880-Decom
add 4c5f21b Merge branch 'master' into HDDS-1880-Decom
add 92369fc Merge Decommission branch into master
add 2c41192 Trigger build
add d903a98 merged master into branch
add 21c387d Address intermittent test failures
add d00bc1d Merge branch 'master' into HDDS-1880-Merge
add 20202f6 Trigger CI
add c296f78 Trigger CI
add cc5eebe Trigger CI
add d811ed7 Trigger build
add 062ad25 Trigger CI
add ca2e991 Merge branch 'master' into HDDS-1880-Merge
new 43f03eb Address compile issues after merging master into
HDDS-1880-Decom
new 3280f3c Merge branch 'HDDS-1880-Decom'
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/hdds/protocol/DatanodeDetails.java | 101 ++-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
.../apache/hadoop/hdds/scm/client/ScmClient.java | 45 +-
.../protocol/StorageContainerLocationProtocol.java | 18 +-
.../common/src/main/resources/ozone-default.xml | 11 +
.../hadoop/hdds/protocol/MockDatanodeDetails.java | 4 +
.../container/common/helpers/DatanodeIdYaml.java | 35 +
.../common/statemachine/DatanodeStateMachine.java | 2 +
.../SetNodeOperationalStateCommandHandler.java | 157 +++++
.../states/endpoint/HeartbeatEndpointTask.java | 12 +
.../commands/SetNodeOperationalStateCommand.java | 89 +++
...inerLocationProtocolClientSideTranslatorPB.java | 85 ++-
.../src/main/proto/ScmAdminProtocol.proto | 47 +-
.../interface-client/src/main/proto/hdds.proto | 13 +-
.../interface-client/src/main/resources/proto.lock | 8 -
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 8 +
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 4 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 9 +-
.../hdds/scm/container/ContainerReplicaCount.java | 271 ++++++++
.../hdds/scm/container/ReplicationManager.java | 304 ++++++---
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 6 +
.../DatanodeAdminMonitor.java} | 20 +-
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 371 +++++++++++
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 50 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 7 +-
...e-info.java => InvalidHostStringException.java} | 31 +-
...ge-info.java => InvalidNodeStateException.java} | 31 +-
.../hadoop/hdds/scm/node/NewNodeHandler.java | 20 +
.../hdds/scm/node/NodeDecommissionManager.java | 369 +++++++++++
.../apache/hadoop/hdds/scm/node/NodeManager.java | 59 +-
.../hadoop/hdds/scm/node/NodeManagerMXBean.java | 2 +-
.../hadoop/hdds/scm/node/NodeStateManager.java | 317 +++++----
.../apache/hadoop/hdds/scm/node/NodeStatus.java | 206 ++++++
.../hadoop/hdds/scm/node/SCMNodeManager.java | 241 +++++--
.../hadoop/hdds/scm/node/SCMNodeMetrics.java | 100 +--
...Handler.java => StartDatanodeAdminHandler.java} | 24 +-
.../hadoop/hdds/scm/node/states/NodeStateMap.java | 244 +++++--
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 3 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 6 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 6 +-
.../hdds/scm/pipeline/SimplePipelineProvider.java | 1 -
...inerLocationProtocolServerSideTranslatorPB.java | 61 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 97 ++-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 8 +
.../hdds/scm/server/StorageContainerManager.java | 35 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 5 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 77 ++-
.../hdds/scm/container/SimpleMockNodeManager.java | 323 ++++++++++
.../scm/container/TestContainerReportHandler.java | 18 +-
.../hdds/scm/container/TestReplicationManager.java | 285 ++++++++-
.../scm/container/TestUnknownContainerReport.java | 4 +-
.../algorithms/TestContainerPlacementFactory.java | 13 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 18 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../states/TestContainerReplicaCount.java | 465 ++++++++++++++
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hdds/scm/node/TestDatanodeAdminMonitor.java | 530 +++++++++++++++
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 29 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 297 +++++++++
.../hadoop/hdds/scm/node/TestNodeStateManager.java | 308 +++++++++
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 214 +++++--
.../hdds/scm/node/states/TestNodeStateMap.java | 140 ++++
.../TestPipelineDatanodesIntersection.java | 3 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 9 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 5 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 3 +-
.../placement/TestContainerPlacement.java | 6 +-
.../testutils/ReplicationNodeManagerMock.java | 85 ++-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 77 ++-
hadoop-hdds/tools/pom.xml | 6 +
.../hdds/scm/cli/ContainerOperationClient.java | 35 +-
.../hadoop/hdds/scm/cli/TopologySubcommand.java | 42 +-
.../hdds/scm/cli/datanode/DatanodeCommands.java | 5 +-
.../DecommissionSubCommand.java} | 27 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 48 +-
.../MaintenanceSubCommand.java} | 36 +-
.../RecommissionSubCommand.java} | 31 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 119 ++++
.../dist/src/main/smoketest/topology/cli.robot | 4 +-
.../TestRatisPipelineCreateAndDestroy.java | 3 +-
.../hadoop/ozone/TestStorageContainerManager.java | 5 +-
.../ozone/client/rpc/TestKeyInputStream.java | 21 +-
.../hadoop/ozone/scm/TestSCMNodeManagerMXBean.java | 30 +-
.../scm/node/TestDecommissionAndMaintenance.java | 708 +++++++++++++++++++++
.../hadoop/ozone/scm/node/TestQueryNode.java | 67 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 4 +-
.../ozone/recon/api/ClusterStateEndpoint.java | 5 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 7 +-
89 files changed, 6890 insertions(+), 780 deletions(-)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/SetNodeOperationalStateCommandHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/SetNodeOperationalStateCommand.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerReplicaCount.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/{pipeline/InsufficientDatanodesException.java
=> node/DatanodeAdminMonitor.java} (64%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/DatanodeAdminMonitorImpl.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidHostStringException.java} (68%)
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{package-info.java
=> InvalidNodeStateException.java} (68%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStatus.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/{StaleNodeHandler.java
=> StartDatanodeAdminHandler.java} (75%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/SimpleMockNodeManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/states/TestContainerReplicaCount.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDatanodeAdminMonitor.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeDecommissionManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/states/TestNodeStateMap.java
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{pipeline/ClosePipelineSubcommand.java
=> datanode/DecommissionSubCommand.java} (70%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/MaintenanceSubCommand.java} (62%)
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/{container/CloseSubcommand.java
=> datanode/RecommissionSubCommand.java} (69%)
create mode 100644
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/datanode/TestListInfoSubcommand.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/node/TestDecommissionAndMaintenance.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
