This is an automated email from the ASF dual-hosted git repository. elek pushed a change to branch HDDS-1880-Decom in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from 920b8c5 HDDS-2459. Refactor ReplicationManager to consider maintenance states add 678e4bc HDDS-2454. Improve OM HA robot tests. (#136) add 46a0875 HDDS-2510. Use isEmpty() to check whether the collection is empty or not in Ozone Manager module. (#258) add 2185ac7 HDDS-2591. No tailMap needed for startIndex 0 in ContainerSet#listContainer (#254) add 52579a2 HDDS-2595. Update Ratis version to latest snapshot version (#251) add 4a91745 HDDS-2609. Move plain unit tests out of integration tests (#260) add 15b05fa HDDS-2626. Avoid hostname lookup for invalid local IP addresses (#267) add 728755e HDDS-2627. Skip sonar check in forks (#266) add 3de23cc Revert "HDDS-2595. Update Ratis version to latest snapshot version (#251)" (#269) add a637a27 HDDS-2477. TableCache cleanup issue for OM non-HA. (#159) add 4f36984 HDDS-2602 Make auto creation for RATIS ONE pipeline configurable. (#248) add fe7fccf HDDS-2372. Datanode pipeline is failing with NoSuchFileException (#166) add 045e3da HDDS-2635 Minor:Fix typo in HddsConfigKeys and ozone-default. (#274) add ba9db10 HDDS-2632. Fix TestContainerPersistence#testDeleteChunk (#273) add ac93176 HDDS-2633. Let findbugs.sh skip frontend plugin for Recon (#275) add 0ef75b0 HDDS-2528. Sonar : code smell category issues in CommitWatcher (#247) add a73c470 HDDS-2034. Async RATIS pipeline creation and destroy through heartbeat commands add cd892e0 HDDS-2640 Add leaderID information in pipeline list subcommand (#279) add b1ceedd Revert "HDDS-2640 Add leaderID information in pipeline list subcommand (#279)" (#280) add c91916d HDDS-2639. Turning off unit test until review of HDDS-2639 add 72ab0a1 HDDS-2648. Disabling flaky test unil the fix. add cd3f8c9 HDDS-2640 Add leaderID information in pipeline list subcommand (#281) add 33b3c16 HDDS-2628. Make AuditMessage parameters strongly typed (#277) add 5e49d7a HDDS-2505. Fix logic related to SCM address calculation in HddsUtils add 46fd24b HDDS-2444. Remove server side dependencies from ozonefs jar files add 968dec4 HDDS-2619. Fix Sonar issues in ReconTaskControllerImpl add afda1c4 HDDS-2655. Use pre-compiled Pattern in NetUtils#normalize add 7524ee4 HDDS-2658. Insight log level reset does not work add 3480c89 HDDS-2654. Conditionally enable profiling at the kernel level add a25b2b4 HDDS-2639. TestTableCacheImpl is flaky (#278) add f141097 HDDS-2657. Key get command creates the output file even in case of KEY_NOT_FOUND (#290) add 4d42714 HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader (#137) add 87e02a3 HDDS-2573. Handle InterruptedException in KeyOutputStream. (#257) add 9381dd0 HDDS-2637. Handle LeaderNot ready exception in OzoneManager StateMachine and upgrade ratis to latest version. (#276) add 2fb8747 HDDS-2386. Implement incremental ChunkBuffer. (#205) add dbd05b0 HDDS-2620. Fix listMultipartupload API. (#263) add ea6c2c9 HDDS-1722. Use the bindings in ReconSchemaGenerationModule to create Recon SQL tables on startup. (#294) add a3a2bfd HDDS-2571. Handle InterruptedException in SCMPipelineManager (#295) add c9f2b02 HDDS-2656. Prefer execute() over submit() if the returned Future is ignored (#302) add 7b71678 HDDS-2475. Unregister ContainerMetadataScrubberMetrics on thread exit (#286) add 5aa7383 HDDS-2653. Improve executor memory usage in new Freon tests (#284) add f686239 HDDS-2608. Provide command to wait until SCM is out from the safe-mode (#250) add 0ff53ef HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available (#282) add a9990f6 HDDS-2622. Add config parameter for setting limit on total bytes of pending requests in Ratis. (#264) add 8e1e225 HDDS-2667. Prometheus reports invalid metric type add 6c1a9ff Addendum HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available (#282) add b35ad98 HDDS-2506. Remove keyAllocationInfo and replication info from the auditLog (#288) add 4f1869c HDDS-2565. Handle InterruptedException in VolumeSet add 1998983 HDDS-2600. Move chaos test to org.apache.hadoop.ozone.chaos package. (#307) add 2f72319 HDDS-1873. Add API to get last completed times for every Recon task. (#233) add e359e66 HDDS-2666. Suppress loader constraint violation message in TestOzoneFileSystemWithMocks (#303) add 1af4f48 HDDS-2530. Sonar : refactor verifyResourceName in HddsClientUtils to fix Sonar errors (#293) add d18aa99 HDDS-2629. Ozone CLI: CreationTime/modifyTime of volume/bucket/key in… (#272) add 9b07c34 HDDS-2677. Acceptance test may fail despite success status (#312) add 1e18347 HDDS-2570. Handle InterruptedException in ProfileServlet. (#308) add 9593b25 HDDS-1987. Fix listStatus API (#108) add f69cb1e HDDS-2636. Refresh pipeline information in OzoneManager lookupFile call. (#301) add 76ad638 HDDS-2678. Add thread name to log pattern (#315) add e54b9be HDDS-2662. Update gRPC and datanode protobuf version in Ozone add db4a6e6 HDDS-2681. Add leak detection memory flags to MiniOzoneChaosCluster add 5eef651 HDDS-1991. Remove RatisClient in OM HA. (#319) add 1503ab8 HDDS-2692. Seek to file end throws EOF Exception. (#326) add 588ff1e HDDS-2689. OMException NOT_A_FILE missing space in the exception message. (#327) add 91a6477 HDDS-2687. Fix sonar issues in package org.apache.hadoop.ozone.recon.api. (#325) add f3abb4c HDDS-2668. Sonar : fix issues reported in BlockManagerImpl (#317) add 1d56bc2 HDDS-2645. Refactor MiniOzoneChaosCluster to a different package to add filesystem tests. (#292) add 5d9ae67 HDDS-2683. Ratis MaxBuffer should be the same size as the segment size. (#320) add 8bfc726 HDDS-1993. Merge OzoneManagerRequestHandler and OzoneManagerHARequest… (#304) add 022ceb5 HDDS-2693. HddsVolume mixes ChunkLayOutVersion and DataNodeLayoutVersion (#330) add 42a30da HDDS-2105. Merge OzoneClientFactory#getRpcClient functions (#148) add 6b78cc5 HDDS-2389. add toStateMachineLogEntryString provider in Ozone's ContainerStateMachine. (#298) add 51c5e42 HDDS-2532. Sonar : fix issues in OzoneQuota (#311) add 050d58b HDDS-2561. Handle InterruptedException in LeaseManager (#338) add 15f4ed4 HDDS-2704. Source tar file is not created during the relase build (#334) add 10bff97 HDDS-2711. OM does not report JVM metrics (#346) add 6506789 HDDS-2688. Refactor container response builders to hadoop-hdds/common (#324) add 0b1ea25 HDDS-2659. KeyValueHandler#handleCreateContainer should log the exception on container creation failure (#296). add e14f709 HDDS-2588. Consolidate compose environments new 92e8d47 Merge remote-tracking branch 'origin/master' into HDDS-1880-Decom The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/post-commit.yml | 1 + .../hadoop/hdds/scm/client/HddsClientUtils.java | 75 +-- .../hadoop/hdds/scm/storage/BlockOutputStream.java | 3 +- .../apache/hadoop/hdds/scm/storage/BufferPool.java | 4 +- .../hadoop/hdds/scm/storage/CommitWatcher.java | 20 +- hadoop-hdds/common/pom.xml | 9 +- .../org/apache/hadoop/hdds/HddsConfigKeys.java | 10 +- .../java/org/apache/hadoop/hdds/HddsUtils.java | 216 ++++---- .../org/apache/hadoop/hdds/client/OzoneQuota.java | 12 +- .../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 13 +- .../org/apache/hadoop/hdds/scm/net/NetUtils.java | 19 +- .../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 8 +- .../ContainerCommandResponseBuilders.java | 269 +++++++++ .../org/apache/hadoop/hdds/utils/Scheduler.java | 7 +- .../apache/hadoop/hdds/utils/db/DBCheckpoint.java | 12 + .../hadoop/hdds/utils/db/RocksDBCheckpoint.java | 11 + .../org/apache/hadoop/hdds/utils/db/Table.java | 10 +- .../apache/hadoop/hdds/utils/db/TypedTable.java | 5 +- .../hadoop/hdds/utils/db/cache/EpochEntry.java | 2 +- .../hadoop/hdds/utils/db/cache/TableCache.java | 14 +- .../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 58 +- .../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 + .../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 + .../org/apache/hadoop/ozone/OzoneSecurityUtil.java | 53 +- .../apache/hadoop/ozone/audit/AuditMessage.java | 51 +- .../apache/hadoop/ozone/common/ChunkBuffer.java | 49 +- .../common/ChunkBufferImplWithByteBuffer.java | 10 +- .../ozone/common/IncrementalChunkBuffer.java | 267 +++++++++ .../ozone/container/common/helpers/BlockData.java | 5 - .../apache/hadoop/ozone/lease/LeaseManager.java | 4 +- .../apache/hadoop/ozone/web/utils/JsonUtils.java | 15 +- .../common/src/main/resources/ozone-default.xml | 69 +-- .../apache/hadoop/hdds/scm/net/TestNetUtils.java | 30 +- .../hdds/utils/db/TestTypedRDBTableStore.java | 7 +- .../hdds/utils/db/cache/TestTableCacheImpl.java | 244 ++++++++- .../hadoop/ozone/audit/TestOzoneAuditLogger.java | 22 +- .../hadoop/ozone/common/TestChunkBuffer.java | 18 + .../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 117 ++-- .../container/common/helpers/ContainerUtils.java | 85 +-- .../ozone/container/common/impl/ContainerSet.java | 2 +- .../container/common/impl/HddsDispatcher.java | 18 +- .../common/statemachine/DatanodeStateMachine.java | 6 + .../CloseContainerCommandHandler.java | 11 +- .../ClosePipelineCommandHandler.java | 120 ++++ .../commandhandler/CommandHandler.java | 2 +- .../CreatePipelineCommandHandler.java | 135 +++++ .../states/endpoint/HeartbeatEndpointTask.java | 22 + .../common/transport/server/XceiverServerSpi.java | 18 + .../server/ratis/ContainerStateMachine.java | 30 +- .../transport/server/ratis/XceiverServerRatis.java | 78 ++- .../ozone/container/common/volume/HddsVolume.java | 3 +- .../ozone/container/common/volume/VolumeSet.java | 1 + .../ozone/container/keyvalue/KeyValueHandler.java | 124 ++--- .../container/keyvalue/helpers/BlockUtils.java | 88 +-- .../container/keyvalue/helpers/ChunkUtils.java | 62 +-- .../keyvalue/helpers/KeyValueContainerUtil.java | 27 - .../container/keyvalue/helpers/SmallFileUtils.java | 91 --- .../container/keyvalue/impl/BlockManagerImpl.java | 65 +-- .../container/keyvalue/impl/ChunkManagerImpl.java | 43 +- .../container/ozoneimpl/ContainerController.java | 9 + .../ozoneimpl/ContainerDataScrubberMetrics.java | 8 +- .../ozoneimpl/ContainerMetadataScanner.java | 18 +- .../ContainerMetadataScrubberMetrics.java | 8 +- .../replication/ReplicationSupervisor.java | 2 +- ...ainerCommand.java => ClosePipelineCommand.java} | 53 +- .../protocol/commands/CreatePipelineCommand.java | 100 ++++ .../proto/StorageContainerDatanodeProtocol.proto | 23 + .../ozone/container/ContainerTestHelper.java | 325 +---------- .../container/common/helpers/TestBlockData.java | 0 .../impl/TestContainerDeletionChoosingPolicy.java | 0 .../common/impl/TestContainerPersistence.java | 2 +- .../container/common/impl/TestContainerSet.java | 39 +- .../container/common/impl/TestHddsDispatcher.java | 6 +- .../container/common/volume/TestVolumeSet.java | 24 + .../container/keyvalue/TestChunkManagerImpl.java | 3 +- .../container/keyvalue/helpers/TestChunkUtils.java | 39 +- .../ozoneimpl/TestContainerScrubberMetrics.java | 35 +- .../apache/hadoop/hdds/server/BaseHttpServer.java | 19 +- .../apache/hadoop/hdds/server/ProfileServlet.java | 7 +- .../hadoop/hdds/server/PrometheusMetricsSink.java | 5 +- .../org/apache/hadoop/hdds/server/ServerUtils.java | 15 + .../hdds/server/TestPrometheusMetricsSink.java | 2 +- .../hadoop/hdds/scm/block/BlockManagerImpl.java | 3 + .../hdds/scm/container/ContainerStateManager.java | 1 + .../apache/hadoop/hdds/scm/events/SCMEvents.java | 12 +- .../scm/pipeline/BackgroundPipelineCreator.java | 23 +- .../hadoop/hdds/scm/pipeline/PipelineFactory.java | 13 +- .../hadoop/hdds/scm/pipeline/PipelineManager.java | 15 +- .../hadoop/hdds/scm/pipeline/PipelineProvider.java | 2 + .../hdds/scm/pipeline/PipelineReportHandler.java | 54 +- .../hdds/scm/pipeline/PipelineStateManager.java | 4 +- .../hdds/scm/pipeline/RatisPipelineProvider.java | 137 ++--- .../hdds/scm/pipeline/RatisPipelineUtils.java | 103 ---- .../hdds/scm/pipeline/SCMPipelineManager.java | 83 ++- .../hdds/scm/pipeline/SCMPipelineMetrics.java | 10 + .../hdds/scm/pipeline/SimplePipelineProvider.java | 5 + .../scm/safemode/HealthyPipelineSafeModeRule.java | 80 +-- .../safemode/OneReplicaPipelineSafeModeRule.java | 65 +-- .../hdds/scm/safemode/SCMSafeModeManager.java | 16 +- .../hadoop/hdds/scm/safemode/SafeModeHandler.java | 5 +- .../hdds/scm/server/SCMBlockProtocolServer.java | 26 +- .../hdds/scm/server/SCMClientProtocolServer.java | 30 +- .../scm/server/SCMDatanodeHeartbeatDispatcher.java | 1 + .../hdds/scm/server/SCMDatanodeProtocolServer.java | 46 +- .../hdds/scm/server/StorageContainerManager.java | 10 +- .../java/org/apache/hadoop/hdds/scm/TestUtils.java | 12 + .../hadoop/hdds/scm/block/TestBlockManager.java | 44 +- .../container/TestCloseContainerEventHandler.java | 11 +- .../scm/container/TestSCMContainerManager.java | 4 +- .../hdds/scm/node/TestContainerPlacement.java | 2 +- .../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 11 +- .../hadoop/hdds/scm/node/TestSCMNodeManager.java | 9 +- .../scm/pipeline/MockRatisPipelineProvider.java | 10 +- .../scm/pipeline/TestPipelineStateManager.java | 0 .../scm/pipeline/TestRatisPipelineProvider.java | 26 +- .../hdds/scm/pipeline/TestSCMPipelineManager.java | 93 +++- .../scm/pipeline/TestSimplePipelineProvider.java | 0 .../hdds/scm/pipeline}/pipeline/package-info.java | 0 .../safemode/TestHealthyPipelineSafeModeRule.java | 44 +- .../TestOneReplicaPipelineSafeModeRule.java | 36 +- .../hdds/scm/safemode/TestSCMSafeModeManager.java | 50 +- .../hadoop/hdds/scm/cli/SafeModeCommands.java | 1 + .../hdds/scm/cli/SafeModeWaitSubcommand.java | 93 ++++ .../apache/hadoop/ozone/client/OzoneBucket.java | 11 +- .../hadoop/ozone/client/OzoneClientFactory.java | 68 +-- .../org/apache/hadoop/ozone/client/OzoneKey.java | 14 +- .../apache/hadoop/ozone/client/OzoneVolume.java | 9 +- .../hadoop/ozone/client/io/KeyInputStream.java | 2 +- .../hadoop/ozone/client/io/KeyOutputStream.java | 1 + .../hadoop/ozone/client/TestHddsClientUtils.java | 46 ++ hadoop-ozone/common/pom.xml | 8 - .../main/java/org/apache/hadoop/ozone/OmUtils.java | 61 +-- .../org/apache/hadoop/ozone/om/OMConfigKeys.java | 9 - .../apache/hadoop/ozone/om/OMMetadataManager.java | 3 +- .../om/exceptions/OMLeaderNotReadyException.java | 22 +- ...derException.java => OMNotLeaderException.java} | 32 +- .../ozone/om/ha/OMFailoverProxyProvider.java | 2 +- .../hadoop/ozone/om/helpers/OMRatisHelper.java | 43 -- .../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 2 - .../ozone/om/protocol/OzoneManagerHAProtocol.java | 6 +- ...OzoneManagerProtocolClientSideTranslatorPB.java | 51 +- .../java/org/apache/hadoop/ozone/TestOmUtils.java | 65 +-- hadoop-ozone/csi/pom.xml | 11 +- hadoop-ozone/dev-support/checks/findbugs.sh | 8 +- .../dist/dev-support/bin/dist-tar-stitching | 6 +- hadoop-ozone/dist/pom.xml | 39 ++ .../dist/src/main/compose/ozone-mr/common-config | 1 + .../compose/ozone-mr/hadoop27/docker-compose.yaml | 1 + .../compose/ozone-mr/hadoop31/docker-compose.yaml | 1 + .../compose/ozone-mr/hadoop32/docker-compose.yaml | 1 + .../compose/ozone-om-ha-s3/docker-compose.yaml | 1 + .../dist/src/main/compose/ozone-recon/.env | 18 - .../main/compose/ozone-recon/docker-compose.yaml | 65 --- .../src/main/compose/ozone-recon/docker-config | 35 -- .../compose/ozone-topology/docker-compose.yaml | 1 + .../src/main/compose/ozone-topology/docker-config | 1 + hadoop-ozone/dist/src/main/compose/ozone/README.md | 99 ++++ .../src/main/compose/ozone/docker-compose.yaml | 91 +-- .../dist/src/main/compose/ozone/docker-config | 11 +- .../compose/{ozoneperf => ozone}/freon-ockg.yaml | 19 +- .../compose/{ozoneperf => ozone}/freon-rk.yaml | 19 +- .../docker-config => ozone/monitoring.conf} | 15 +- .../dist/src/main/compose/ozone/monitoring.yaml | 56 ++ .../{ozone-recon/test.sh => ozone/profiling.yaml} | 30 +- .../compose/{ozoneperf => ozone}/prometheus.yml | 20 +- .../{ozone-om-ha/startOM.sh => ozone/run.sh} | 13 +- hadoop-ozone/dist/src/main/compose/ozone/test.sh | 9 +- .../src/main/compose/ozoneblockade/docker-config | 2 + hadoop-ozone/dist/src/main/compose/ozoneperf/.env | 18 - .../dist/src/main/compose/ozoneperf/README.md | 74 --- .../src/main/compose/ozoneperf/docker-compose.yaml | 84 --- .../dist/src/main/compose/ozoneperf/docker-config | 36 -- .../dist/src/main/compose/ozoneperf/test.sh | 30 - .../compose/ozones3-haproxy/docker-compose.yaml | 1 + hadoop-ozone/dist/src/main/compose/ozones3/.env | 18 - .../src/main/compose/ozones3/docker-compose.yaml | 58 -- hadoop-ozone/dist/src/main/compose/ozones3/test.sh | 34 -- .../src/main/compose/ozonescripts/docker-config | 3 +- .../compose/ozonesecure-mr/docker-compose.yaml | 1 + .../main/compose/ozonesecure/docker-compose.yaml | 1 + hadoop-ozone/dist/src/main/compose/testlib.sh | 52 +- .../dist/src/main/dockerlibexec/entrypoint.sh | 8 + .../dist/src/main/smoketest/__init__.robot | 3 +- .../dist/src/main/smoketest/basic/basic.robot | 4 +- .../dist/src/main/smoketest/omha/testOMHA.robot | 28 +- hadoop-ozone/dist/src/shell/conf/log4j.properties | 10 +- .../fault-injection-test/mini-chaos-tests}/pom.xml | 37 +- .../mini-chaos-tests/src/test/bin/start-chaos.sh | 46 ++ .../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 6 +- .../hadoop/ozone/MiniOzoneLoadGenerator.java | 107 +--- .../hadoop/ozone/TestMiniChaosOzoneCluster.java | 12 +- .../org/apache/hadoop/ozone/utils/LoadBucket.java | 99 ++++ .../hadoop/ozone/utils}/TestProbability.java | 2 +- .../services/org.apache.hadoop.fs.FileSystem | 0 .../src/test/resources/log4j.properties | 8 +- hadoop-ozone/fault-injection-test/pom.xml | 1 + .../apache/hadoop/ozone/insight/LogSubcommand.java | 31 +- .../integration-test/src/test/bin/start-chaos.sh | 35 -- .../TestContainerStateManagerIntegration.java | 6 +- .../metrics/TestSCMContainerManagerMetrics.java | 3 + .../hdds/scm/pipeline/TestPipelineClose.java | 4 +- .../TestRatisPipelineCreateAndDestroy.java | 29 + .../scm/pipeline/TestRatisPipelineProvider.java | 210 +++++++ .../org/apache/hadoop/ozone/MiniOzoneCluster.java | 2 +- .../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 53 +- .../hadoop/ozone/MiniOzoneHAClusterImpl.java | 24 +- .../hadoop/ozone/TestContainerOperations.java | 2 +- .../TestContainerStateMachineIdempotency.java | 2 +- .../apache/hadoop/ozone/TestMiniOzoneCluster.java | 6 +- .../hadoop/ozone/TestMiniOzoneHACluster.java | 112 ++++ .../hadoop/ozone/TestStorageContainerManager.java | 18 +- .../apache/hadoop/ozone/client/rpc/TestBCSID.java | 3 + .../ozone/client/rpc/TestBlockOutputStream.java | 5 +- .../rpc/TestBlockOutputStreamWithFailures.java | 13 +- .../rpc/TestCloseContainerHandlingByClient.java | 7 +- .../hadoop/ozone/client/rpc/TestCommitWatcher.java | 4 +- .../rpc/TestContainerReplicationEndToEnd.java | 4 +- .../client/rpc/TestContainerStateMachine.java | 10 +- .../rpc/TestContainerStateMachineFailures.java | 18 +- .../client/rpc/TestFailureHandlingByClient.java | 7 +- .../ozone/client/rpc/TestKeyInputStream.java | 7 +- .../rpc/TestMultiBlockWritesWithDnFailures.java | 5 +- .../client/rpc/TestOzoneAtRestEncryption.java | 14 +- .../rpc/TestOzoneClientRetriesOnException.java | 9 +- .../client/rpc/TestOzoneRpcClientAbstract.java | 90 ++- .../ozone/client/rpc/TestSecureOzoneRpcClient.java | 8 +- .../ozone/client/rpc/TestWatchForCommit.java | 7 +- .../ozone/container/TestContainerReplication.java | 2 +- .../apache/hadoop/ozone/container/TestHelper.java | 305 +++++++++++ .../commandhandler/TestBlockDeletion.java | 6 +- .../commandhandler/TestCloseContainerHandler.java | 2 + .../commandhandler/TestDeleteContainerHandler.java | 2 + .../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 10 +- .../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 235 ++++++++ .../org/apache/hadoop/ozone/om/TestKeyPurging.java | 5 +- .../hadoop/ozone/om/TestOMRatisSnapshots.java | 20 +- .../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 34 +- .../apache/hadoop/ozone/om/TestScmSafeMode.java | 2 +- .../hadoop/ozone/ozShell/TestOzoneShellHA.java | 20 +- .../hadoop/ozone/scm/TestContainerSmallFile.java | 2 +- .../scm/TestGetCommittedBlockLengthAndPutKey.java | 2 +- .../org/apache/hadoop/ozone/scm/TestSCMMXBean.java | 2 +- .../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 4 +- hadoop-ozone/ozone-manager/pom.xml | 5 + .../apache/hadoop/ozone/om/KeyDeletingService.java | 2 +- .../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 176 ++++-- .../hadoop/ozone/om/OMDBCheckpointServlet.java | 19 +- .../java/org/apache/hadoop/ozone/om/OMStorage.java | 17 +- .../hadoop/ozone/om/OmMetadataManagerImpl.java | 35 +- .../org/apache/hadoop/ozone/om/OzoneManager.java | 103 +--- .../hadoop/ozone/om/ratis/OMRatisSnapshotInfo.java | 15 +- .../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 16 +- .../ozone/om/ratis/OzoneManagerRatisClient.java | 210 ------- .../ozone/om/ratis/OzoneManagerRatisServer.java | 83 ++- .../ozone/om/ratis/OzoneManagerStateMachine.java | 85 ++- .../hadoop/ozone/om/request/OMClientRequest.java | 6 +- .../hadoop/ozone/om/request/RequestAuditor.java | 3 - .../ozone/om/request/file/OMFileCreateRequest.java | 6 +- .../om/snapshot/OzoneManagerSnapshotProvider.java | 10 + .../protocolPB/OzoneManagerHARequestHandler.java | 39 -- .../OzoneManagerHARequestHandlerImpl.java | 56 -- ...OzoneManagerProtocolServerSideTranslatorPB.java | 27 +- .../protocolPB/OzoneManagerRequestHandler.java | 607 +-------------------- .../hadoop/ozone/protocolPB/RequestHandler.java | 28 +- .../hadoop/ozone/web/ozShell/OzoneAddress.java | 3 +- .../ozone/web/ozShell/keys/GetKeyHandler.java | 4 +- .../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 256 ++++++++- .../org/apache/hadoop/ozone/om/TestOMStorage.java | 91 +++ .../ozone/om/ratis}/TestOMRatisSnapshotInfo.java | 8 +- ...TestOzoneManagerDoubleBufferWithOMResponse.java | 1 + .../om/ratis/TestOzoneManagerRatisServer.java | 29 +- .../ozone/om/request/TestOMRequestUtils.java | 17 + .../security/acl/TestOzoneNativeAuthorizer.java | 20 +- .../ozone/web/ozShell/TestObjectPrinter.java | 6 +- hadoop-ozone/ozonefs-lib-current/pom.xml | 15 - .../fs/ozone/BasicOzoneClientAdapterImpl.java | 2 +- .../hadoop/fs/ozone/TestOzoneFileSystem.java | 12 + .../fs/ozone/TestOzoneFileSystemWithMocks.java | 2 + .../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 48 +- .../hadoop/fs/ozone/TestOzoneFsRenameDir.java | 2 +- .../hadoop/ozone/recon/ReconControllerModule.java | 4 - .../hadoop/ozone/recon/ReconSchemaManager.java | 55 ++ .../org/apache/hadoop/ozone/recon/ReconServer.java | 26 +- .../ozone/recon/api/ContainerKeyService.java | 112 ++-- ...lizationService.java => TaskStatusService.java} | 39 +- .../hadoop/ozone/recon/api/UtilizationService.java | 5 - .../recon/recovery/ReconOMMetadataManager.java | 6 + .../recon/recovery/ReconOmMetadataManagerImpl.java | 11 + .../spi/impl/OzoneManagerServiceProviderImpl.java | 32 +- .../ozone/recon/tasks/ReconTaskController.java | 3 +- .../ozone/recon/tasks/ReconTaskControllerImpl.java | 70 ++- .../ozone/recon/api/TestTaskStatusService.java | 86 +++ .../recon/tasks/TestReconTaskControllerImpl.java | 38 +- hadoop-ozone/s3gateway/pom.xml | 4 + .../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 4 +- .../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 8 +- .../hadoop/ozone/s3/endpoint/RootEndpoint.java | 4 +- .../hadoop/ozone/freon/BaseFreonGenerator.java | 116 ++-- .../hadoop/ozone/freon/RandomKeyGenerator.java | 2 +- .../freon/TestFreonWithDatanodeFastRestart.java | 4 +- .../hadoop/ozone/fsck/TestContainerMapper.java | 2 +- pom.xml | 64 +-- 302 files changed, 6094 insertions(+), 4584 deletions(-) create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java create mode 100644 hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/IncrementalChunkBuffer.java copy hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java => hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetUtils.java (56%) create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/ClosePipelineCommandHandler.java create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CreatePipelineCommandHandler.java delete mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/SmallFileUtils.java copy hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/{CloseContainerCommand.java => ClosePipelineCommand.java} (53%) create mode 100644 hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/CreatePipelineCommand.java rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java (65%) rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/helpers/TestBlockData.java (100%) rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java (100%) rename {hadoop-ozone/integration-test => hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java (99%) delete mode 100644 hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineStateManager.java (100%) rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java (78%) rename {hadoop-ozone/integration-test => hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSimplePipelineProvider.java (100%) copy {hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm => hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline}/pipeline/package-info.java (100%) create mode 100644 hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java copy hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java => hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMLeaderNotReadyException.java (65%) rename hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/{NotLeaderException.java => OMNotLeaderException.java} (71%) delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/.env delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/README.md rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-ockg.yaml (71%) rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-rk.yaml (72%) rename hadoop-ozone/dist/src/main/compose/{ozones3/docker-config => ozone/monitoring.conf} (76%) create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/monitoring.yaml rename hadoop-ozone/dist/src/main/compose/{ozone-recon/test.sh => ozone/profiling.yaml} (68%) mode change 100755 => 100644 rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/prometheus.yml (72%) copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/startOM.sh => ozone/run.sh} (76%) delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/.env delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/README.md delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config delete mode 100755 hadoop-ozone/dist/src/main/compose/ozoneperf/test.sh delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/.env delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml delete mode 100755 hadoop-ozone/dist/src/main/compose/ozones3/test.sh copy {hadoop-hdds/tools => hadoop-ozone/fault-injection-test/mini-chaos-tests}/pom.xml (68%) create mode 100755 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/bin/start-chaos.sh rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java (97%) rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java (66%) rename hadoop-ozone/{integration-test => fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java (90%) create mode 100644 hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/LoadBucket.java rename hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/chaos => fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils}/TestProbability.java (97%) copy hadoop-ozone/{tools/src/main => fault-injection-test/mini-chaos-tests/src/test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem (100%) copy hadoop-ozone/{ozonefs => fault-injection-test/mini-chaos-tests}/src/test/resources/log4j.properties (75%) delete mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestHelper.java delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java delete mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMStorage.java rename hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot => ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis}/TestOMRatisSnapshotInfo.java (90%) rename hadoop-ozone/{integration-test => ozone-manager}/src/test/java/org/apache/hadoop/ozone/security/acl/TestOzoneNativeAuthorizer.java (96%) create mode 100644 hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconSchemaManager.java copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{UtilizationService.java => TaskStatusService.java} (60%) create mode 100644 hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTaskStatusService.java --------------------------------------------------------------------- To unsubscribe, e-mail: ozone-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-commits-h...@hadoop.apache.org