This is an automated email from the ASF dual-hosted git repository.
elek pushed a change to branch HDDS-1880-Decom
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from 920b8c5 HDDS-2459. Refactor ReplicationManager to consider
maintenance states
add 678e4bc HDDS-2454. Improve OM HA robot tests. (#136)
add 46a0875 HDDS-2510. Use isEmpty() to check whether the collection is
empty or not in Ozone Manager module. (#258)
add 2185ac7 HDDS-2591. No tailMap needed for startIndex 0 in
ContainerSet#listContainer (#254)
add 52579a2 HDDS-2595. Update Ratis version to latest snapshot version
(#251)
add 4a91745 HDDS-2609. Move plain unit tests out of integration tests
(#260)
add 15b05fa HDDS-2626. Avoid hostname lookup for invalid local IP
addresses (#267)
add 728755e HDDS-2627. Skip sonar check in forks (#266)
add 3de23cc Revert "HDDS-2595. Update Ratis version to latest snapshot
version (#251)" (#269)
add a637a27 HDDS-2477. TableCache cleanup issue for OM non-HA. (#159)
add 4f36984 HDDS-2602 Make auto creation for RATIS ONE pipeline
configurable. (#248)
add fe7fccf HDDS-2372. Datanode pipeline is failing with
NoSuchFileException (#166)
add 045e3da HDDS-2635 Minor:Fix typo in HddsConfigKeys and ozone-default.
(#274)
add ba9db10 HDDS-2632. Fix TestContainerPersistence#testDeleteChunk (#273)
add ac93176 HDDS-2633. Let findbugs.sh skip frontend plugin for Recon
(#275)
add 0ef75b0 HDDS-2528. Sonar : code smell category issues in
CommitWatcher (#247)
add a73c470 HDDS-2034. Async RATIS pipeline creation and destroy through
heartbeat commands
add cd892e0 HDDS-2640 Add leaderID information in pipeline list
subcommand (#279)
add b1ceedd Revert "HDDS-2640 Add leaderID information in pipeline list
subcommand (#279)" (#280)
add c91916d HDDS-2639. Turning off unit test until review of HDDS-2639
add 72ab0a1 HDDS-2648. Disabling flaky test unil the fix.
add cd3f8c9 HDDS-2640 Add leaderID information in pipeline list
subcommand (#281)
add 33b3c16 HDDS-2628. Make AuditMessage parameters strongly typed (#277)
add 5e49d7a HDDS-2505. Fix logic related to SCM address calculation in
HddsUtils
add 46fd24b HDDS-2444. Remove server side dependencies from ozonefs jar
files
add 968dec4 HDDS-2619. Fix Sonar issues in ReconTaskControllerImpl
add afda1c4 HDDS-2655. Use pre-compiled Pattern in NetUtils#normalize
add 7524ee4 HDDS-2658. Insight log level reset does not work
add 3480c89 HDDS-2654. Conditionally enable profiling at the kernel level
add a25b2b4 HDDS-2639. TestTableCacheImpl is flaky (#278)
add f141097 HDDS-2657. Key get command creates the output file even in
case of KEY_NOT_FOUND (#290)
add 4d42714 HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader (#137)
add 87e02a3 HDDS-2573. Handle InterruptedException in KeyOutputStream.
(#257)
add 9381dd0 HDDS-2637. Handle LeaderNot ready exception in OzoneManager
StateMachine and upgrade ratis to latest version. (#276)
add 2fb8747 HDDS-2386. Implement incremental ChunkBuffer. (#205)
add dbd05b0 HDDS-2620. Fix listMultipartupload API. (#263)
add ea6c2c9 HDDS-1722. Use the bindings in ReconSchemaGenerationModule to
create Recon SQL tables on startup. (#294)
add a3a2bfd HDDS-2571. Handle InterruptedException in SCMPipelineManager
(#295)
add c9f2b02 HDDS-2656. Prefer execute() over submit() if the returned
Future is ignored (#302)
add 7b71678 HDDS-2475. Unregister ContainerMetadataScrubberMetrics on
thread exit (#286)
add 5aa7383 HDDS-2653. Improve executor memory usage in new Freon tests
(#284)
add f686239 HDDS-2608. Provide command to wait until SCM is out from the
safe-mode (#250)
add 0ff53ef HDDS-2646. Start acceptance tests only if at least one THREE
pipeline is available (#282)
add a9990f6 HDDS-2622. Add config parameter for setting limit on total
bytes of pending requests in Ratis. (#264)
add 8e1e225 HDDS-2667. Prometheus reports invalid metric type
add 6c1a9ff Addendum HDDS-2646. Start acceptance tests only if at least
one THREE pipeline is available (#282)
add b35ad98 HDDS-2506. Remove keyAllocationInfo and replication info from
the auditLog (#288)
add 4f1869c HDDS-2565. Handle InterruptedException in VolumeSet
add 1998983 HDDS-2600. Move chaos test to org.apache.hadoop.ozone.chaos
package. (#307)
add 2f72319 HDDS-1873. Add API to get last completed times for every
Recon task. (#233)
add e359e66 HDDS-2666. Suppress loader constraint violation message in
TestOzoneFileSystemWithMocks (#303)
add 1af4f48 HDDS-2530. Sonar : refactor verifyResourceName in
HddsClientUtils to fix Sonar errors (#293)
add d18aa99 HDDS-2629. Ozone CLI: CreationTime/modifyTime of
volume/bucket/key in… (#272)
add 9b07c34 HDDS-2677. Acceptance test may fail despite success status
(#312)
add 1e18347 HDDS-2570. Handle InterruptedException in ProfileServlet.
(#308)
add 9593b25 HDDS-1987. Fix listStatus API (#108)
add f69cb1e HDDS-2636. Refresh pipeline information in OzoneManager
lookupFile call. (#301)
add 76ad638 HDDS-2678. Add thread name to log pattern (#315)
add e54b9be HDDS-2662. Update gRPC and datanode protobuf version in Ozone
add db4a6e6 HDDS-2681. Add leak detection memory flags to
MiniOzoneChaosCluster
add 5eef651 HDDS-1991. Remove RatisClient in OM HA. (#319)
add 1503ab8 HDDS-2692. Seek to file end throws EOF Exception. (#326)
add 588ff1e HDDS-2689. OMException NOT_A_FILE missing space in the
exception message. (#327)
add 91a6477 HDDS-2687. Fix sonar issues in package
org.apache.hadoop.ozone.recon.api. (#325)
add f3abb4c HDDS-2668. Sonar : fix issues reported in BlockManagerImpl
(#317)
add 1d56bc2 HDDS-2645. Refactor MiniOzoneChaosCluster to a different
package to add filesystem tests. (#292)
add 5d9ae67 HDDS-2683. Ratis MaxBuffer should be the same size as the
segment size. (#320)
add 8bfc726 HDDS-1993. Merge OzoneManagerRequestHandler and
OzoneManagerHARequest… (#304)
add 022ceb5 HDDS-2693. HddsVolume mixes ChunkLayOutVersion and
DataNodeLayoutVersion (#330)
add 42a30da HDDS-2105. Merge OzoneClientFactory#getRpcClient functions
(#148)
add 6b78cc5 HDDS-2389. add toStateMachineLogEntryString provider in
Ozone's ContainerStateMachine. (#298)
add 51c5e42 HDDS-2532. Sonar : fix issues in OzoneQuota (#311)
add 050d58b HDDS-2561. Handle InterruptedException in LeaseManager (#338)
add 15f4ed4 HDDS-2704. Source tar file is not created during the relase
build (#334)
add 10bff97 HDDS-2711. OM does not report JVM metrics (#346)
add 6506789 HDDS-2688. Refactor container response builders to
hadoop-hdds/common (#324)
add 0b1ea25 HDDS-2659. KeyValueHandler#handleCreateContainer should log
the exception on container creation failure (#296).
add e14f709 HDDS-2588. Consolidate compose environments
new 92e8d47 Merge remote-tracking branch 'origin/master' into
HDDS-1880-Decom
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 1 +
.../hadoop/hdds/scm/client/HddsClientUtils.java | 75 +--
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 3 +-
.../apache/hadoop/hdds/scm/storage/BufferPool.java | 4 +-
.../hadoop/hdds/scm/storage/CommitWatcher.java | 20 +-
hadoop-hdds/common/pom.xml | 9 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 10 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 216 ++++----
.../org/apache/hadoop/hdds/client/OzoneQuota.java | 12 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 13 +-
.../org/apache/hadoop/hdds/scm/net/NetUtils.java | 19 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 8 +-
.../ContainerCommandResponseBuilders.java | 269 +++++++++
.../org/apache/hadoop/hdds/utils/Scheduler.java | 7 +-
.../apache/hadoop/hdds/utils/db/DBCheckpoint.java | 12 +
.../hadoop/hdds/utils/db/RocksDBCheckpoint.java | 11 +
.../org/apache/hadoop/hdds/utils/db/Table.java | 10 +-
.../apache/hadoop/hdds/utils/db/TypedTable.java | 5 +-
.../hadoop/hdds/utils/db/cache/EpochEntry.java | 2 +-
.../hadoop/hdds/utils/db/cache/TableCache.java | 14 +-
.../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 58 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../org/apache/hadoop/ozone/OzoneSecurityUtil.java | 53 +-
.../apache/hadoop/ozone/audit/AuditMessage.java | 51 +-
.../apache/hadoop/ozone/common/ChunkBuffer.java | 49 +-
.../common/ChunkBufferImplWithByteBuffer.java | 10 +-
.../ozone/common/IncrementalChunkBuffer.java | 267 +++++++++
.../ozone/container/common/helpers/BlockData.java | 5 -
.../apache/hadoop/ozone/lease/LeaseManager.java | 4 +-
.../apache/hadoop/ozone/web/utils/JsonUtils.java | 15 +-
.../common/src/main/resources/ozone-default.xml | 69 +--
.../apache/hadoop/hdds/scm/net/TestNetUtils.java | 30 +-
.../hdds/utils/db/TestTypedRDBTableStore.java | 7 +-
.../hdds/utils/db/cache/TestTableCacheImpl.java | 244 ++++++++-
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 22 +-
.../hadoop/ozone/common/TestChunkBuffer.java | 18 +
.../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 117 ++--
.../container/common/helpers/ContainerUtils.java | 85 +--
.../ozone/container/common/impl/ContainerSet.java | 2 +-
.../container/common/impl/HddsDispatcher.java | 18 +-
.../common/statemachine/DatanodeStateMachine.java | 6 +
.../CloseContainerCommandHandler.java | 11 +-
.../ClosePipelineCommandHandler.java | 120 ++++
.../commandhandler/CommandHandler.java | 2 +-
.../CreatePipelineCommandHandler.java | 135 +++++
.../states/endpoint/HeartbeatEndpointTask.java | 22 +
.../common/transport/server/XceiverServerSpi.java | 18 +
.../server/ratis/ContainerStateMachine.java | 30 +-
.../transport/server/ratis/XceiverServerRatis.java | 78 ++-
.../ozone/container/common/volume/HddsVolume.java | 3 +-
.../ozone/container/common/volume/VolumeSet.java | 1 +
.../ozone/container/keyvalue/KeyValueHandler.java | 124 ++---
.../container/keyvalue/helpers/BlockUtils.java | 88 +--
.../container/keyvalue/helpers/ChunkUtils.java | 62 +--
.../keyvalue/helpers/KeyValueContainerUtil.java | 27 -
.../container/keyvalue/helpers/SmallFileUtils.java | 91 ---
.../container/keyvalue/impl/BlockManagerImpl.java | 65 +--
.../container/keyvalue/impl/ChunkManagerImpl.java | 43 +-
.../container/ozoneimpl/ContainerController.java | 9 +
.../ozoneimpl/ContainerDataScrubberMetrics.java | 8 +-
.../ozoneimpl/ContainerMetadataScanner.java | 18 +-
.../ContainerMetadataScrubberMetrics.java | 8 +-
.../replication/ReplicationSupervisor.java | 2 +-
...ainerCommand.java => ClosePipelineCommand.java} | 53 +-
.../protocol/commands/CreatePipelineCommand.java | 100 ++++
.../proto/StorageContainerDatanodeProtocol.proto | 23 +
.../ozone/container/ContainerTestHelper.java | 325 +----------
.../container/common/helpers/TestBlockData.java | 0
.../impl/TestContainerDeletionChoosingPolicy.java | 0
.../common/impl/TestContainerPersistence.java | 2 +-
.../container/common/impl/TestContainerSet.java | 39 +-
.../container/common/impl/TestHddsDispatcher.java | 6 +-
.../container/common/volume/TestVolumeSet.java | 24 +
.../container/keyvalue/TestChunkManagerImpl.java | 3 +-
.../container/keyvalue/helpers/TestChunkUtils.java | 39 +-
.../ozoneimpl/TestContainerScrubberMetrics.java | 35 +-
.../apache/hadoop/hdds/server/BaseHttpServer.java | 19 +-
.../apache/hadoop/hdds/server/ProfileServlet.java | 7 +-
.../hadoop/hdds/server/PrometheusMetricsSink.java | 5 +-
.../org/apache/hadoop/hdds/server/ServerUtils.java | 15 +
.../hdds/server/TestPrometheusMetricsSink.java | 2 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 3 +
.../hdds/scm/container/ContainerStateManager.java | 1 +
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 12 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 23 +-
.../hadoop/hdds/scm/pipeline/PipelineFactory.java | 13 +-
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 15 +-
.../hadoop/hdds/scm/pipeline/PipelineProvider.java | 2 +
.../hdds/scm/pipeline/PipelineReportHandler.java | 54 +-
.../hdds/scm/pipeline/PipelineStateManager.java | 4 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 137 ++---
.../hdds/scm/pipeline/RatisPipelineUtils.java | 103 ----
.../hdds/scm/pipeline/SCMPipelineManager.java | 83 ++-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 10 +
.../hdds/scm/pipeline/SimplePipelineProvider.java | 5 +
.../scm/safemode/HealthyPipelineSafeModeRule.java | 80 +--
.../safemode/OneReplicaPipelineSafeModeRule.java | 65 +--
.../hdds/scm/safemode/SCMSafeModeManager.java | 16 +-
.../hadoop/hdds/scm/safemode/SafeModeHandler.java | 5 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 26 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 30 +-
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 1 +
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 46 +-
.../hdds/scm/server/StorageContainerManager.java | 10 +-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 12 +
.../hadoop/hdds/scm/block/TestBlockManager.java | 44 +-
.../container/TestCloseContainerEventHandler.java | 11 +-
.../scm/container/TestSCMContainerManager.java | 4 +-
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 11 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 9 +-
.../scm/pipeline/MockRatisPipelineProvider.java | 10 +-
.../scm/pipeline/TestPipelineStateManager.java | 0
.../scm/pipeline/TestRatisPipelineProvider.java | 26 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 93 +++-
.../scm/pipeline/TestSimplePipelineProvider.java | 0
.../hdds/scm/pipeline}/pipeline/package-info.java | 0
.../safemode/TestHealthyPipelineSafeModeRule.java | 44 +-
.../TestOneReplicaPipelineSafeModeRule.java | 36 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 50 +-
.../hadoop/hdds/scm/cli/SafeModeCommands.java | 1 +
.../hdds/scm/cli/SafeModeWaitSubcommand.java | 93 ++++
.../apache/hadoop/ozone/client/OzoneBucket.java | 11 +-
.../hadoop/ozone/client/OzoneClientFactory.java | 68 +--
.../org/apache/hadoop/ozone/client/OzoneKey.java | 14 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 9 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 2 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 1 +
.../hadoop/ozone/client/TestHddsClientUtils.java | 46 ++
hadoop-ozone/common/pom.xml | 8 -
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 61 +--
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 9 -
.../apache/hadoop/ozone/om/OMMetadataManager.java | 3 +-
.../om/exceptions/OMLeaderNotReadyException.java | 22 +-
...derException.java => OMNotLeaderException.java} | 32 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 2 +-
.../hadoop/ozone/om/helpers/OMRatisHelper.java | 43 --
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 2 -
.../ozone/om/protocol/OzoneManagerHAProtocol.java | 6 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 51 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 65 +--
hadoop-ozone/csi/pom.xml | 11 +-
hadoop-ozone/dev-support/checks/findbugs.sh | 8 +-
.../dist/dev-support/bin/dist-tar-stitching | 6 +-
hadoop-ozone/dist/pom.xml | 39 ++
.../dist/src/main/compose/ozone-mr/common-config | 1 +
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 1 +
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 1 +
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 1 +
.../compose/ozone-om-ha-s3/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-recon/.env | 18 -
.../main/compose/ozone-recon/docker-compose.yaml | 65 ---
.../src/main/compose/ozone-recon/docker-config | 35 --
.../compose/ozone-topology/docker-compose.yaml | 1 +
.../src/main/compose/ozone-topology/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/ozone/README.md | 99 ++++
.../src/main/compose/ozone/docker-compose.yaml | 91 +--
.../dist/src/main/compose/ozone/docker-config | 11 +-
.../compose/{ozoneperf => ozone}/freon-ockg.yaml | 19 +-
.../compose/{ozoneperf => ozone}/freon-rk.yaml | 19 +-
.../docker-config => ozone/monitoring.conf} | 15 +-
.../dist/src/main/compose/ozone/monitoring.yaml | 56 ++
.../{ozone-recon/test.sh => ozone/profiling.yaml} | 30 +-
.../compose/{ozoneperf => ozone}/prometheus.yml | 20 +-
.../{ozone-om-ha/startOM.sh => ozone/run.sh} | 13 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 9 +-
.../src/main/compose/ozoneblockade/docker-config | 2 +
hadoop-ozone/dist/src/main/compose/ozoneperf/.env | 18 -
.../dist/src/main/compose/ozoneperf/README.md | 74 ---
.../src/main/compose/ozoneperf/docker-compose.yaml | 84 ---
.../dist/src/main/compose/ozoneperf/docker-config | 36 --
.../dist/src/main/compose/ozoneperf/test.sh | 30 -
.../compose/ozones3-haproxy/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/ozones3/.env | 18 -
.../src/main/compose/ozones3/docker-compose.yaml | 58 --
hadoop-ozone/dist/src/main/compose/ozones3/test.sh | 34 --
.../src/main/compose/ozonescripts/docker-config | 3 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 52 +-
.../dist/src/main/dockerlibexec/entrypoint.sh | 8 +
.../dist/src/main/smoketest/__init__.robot | 3 +-
.../dist/src/main/smoketest/basic/basic.robot | 4 +-
.../dist/src/main/smoketest/omha/testOMHA.robot | 28 +-
hadoop-ozone/dist/src/shell/conf/log4j.properties | 10 +-
.../fault-injection-test/mini-chaos-tests}/pom.xml | 37 +-
.../mini-chaos-tests/src/test/bin/start-chaos.sh | 46 ++
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 6 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 107 +---
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 12 +-
.../org/apache/hadoop/ozone/utils/LoadBucket.java | 99 ++++
.../hadoop/ozone/utils}/TestProbability.java | 2 +-
.../services/org.apache.hadoop.fs.FileSystem | 0
.../src/test/resources/log4j.properties | 8 +-
hadoop-ozone/fault-injection-test/pom.xml | 1 +
.../apache/hadoop/ozone/insight/LogSubcommand.java | 31 +-
.../integration-test/src/test/bin/start-chaos.sh | 35 --
.../TestContainerStateManagerIntegration.java | 6 +-
.../metrics/TestSCMContainerManagerMetrics.java | 3 +
.../hdds/scm/pipeline/TestPipelineClose.java | 4 +-
.../TestRatisPipelineCreateAndDestroy.java | 29 +
.../scm/pipeline/TestRatisPipelineProvider.java | 210 +++++++
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 2 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 53 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 24 +-
.../hadoop/ozone/TestContainerOperations.java | 2 +-
.../TestContainerStateMachineIdempotency.java | 2 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 6 +-
.../hadoop/ozone/TestMiniOzoneHACluster.java | 112 ++++
.../hadoop/ozone/TestStorageContainerManager.java | 18 +-
.../apache/hadoop/ozone/client/rpc/TestBCSID.java | 3 +
.../ozone/client/rpc/TestBlockOutputStream.java | 5 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 13 +-
.../rpc/TestCloseContainerHandlingByClient.java | 7 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 4 +-
.../rpc/TestContainerReplicationEndToEnd.java | 4 +-
.../client/rpc/TestContainerStateMachine.java | 10 +-
.../rpc/TestContainerStateMachineFailures.java | 18 +-
.../client/rpc/TestFailureHandlingByClient.java | 7 +-
.../ozone/client/rpc/TestKeyInputStream.java | 7 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 5 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 14 +-
.../rpc/TestOzoneClientRetriesOnException.java | 9 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 90 ++-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 8 +-
.../ozone/client/rpc/TestWatchForCommit.java | 7 +-
.../ozone/container/TestContainerReplication.java | 2 +-
.../apache/hadoop/ozone/container/TestHelper.java | 305 +++++++++++
.../commandhandler/TestBlockDeletion.java | 6 +-
.../commandhandler/TestCloseContainerHandler.java | 2 +
.../commandhandler/TestDeleteContainerHandler.java | 2 +
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 10 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 235 ++++++++
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 5 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 20 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 34 +-
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 2 +-
.../hadoop/ozone/ozShell/TestOzoneShellHA.java | 20 +-
.../hadoop/ozone/scm/TestContainerSmallFile.java | 2 +-
.../scm/TestGetCommittedBlockLengthAndPutKey.java | 2 +-
.../org/apache/hadoop/ozone/scm/TestSCMMXBean.java | 2 +-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 5 +
.../apache/hadoop/ozone/om/KeyDeletingService.java | 2 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 176 ++++--
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 19 +-
.../java/org/apache/hadoop/ozone/om/OMStorage.java | 17 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 35 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 103 +---
.../hadoop/ozone/om/ratis/OMRatisSnapshotInfo.java | 15 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 16 +-
.../ozone/om/ratis/OzoneManagerRatisClient.java | 210 -------
.../ozone/om/ratis/OzoneManagerRatisServer.java | 83 ++-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 85 ++-
.../hadoop/ozone/om/request/OMClientRequest.java | 6 +-
.../hadoop/ozone/om/request/RequestAuditor.java | 3 -
.../ozone/om/request/file/OMFileCreateRequest.java | 6 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 10 +
.../protocolPB/OzoneManagerHARequestHandler.java | 39 --
.../OzoneManagerHARequestHandlerImpl.java | 56 --
...OzoneManagerProtocolServerSideTranslatorPB.java | 27 +-
.../protocolPB/OzoneManagerRequestHandler.java | 607 +--------------------
.../hadoop/ozone/protocolPB/RequestHandler.java | 28 +-
.../hadoop/ozone/web/ozShell/OzoneAddress.java | 3 +-
.../ozone/web/ozShell/keys/GetKeyHandler.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 256 ++++++++-
.../org/apache/hadoop/ozone/om/TestOMStorage.java | 91 +++
.../ozone/om/ratis}/TestOMRatisSnapshotInfo.java | 8 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 1 +
.../om/ratis/TestOzoneManagerRatisServer.java | 29 +-
.../ozone/om/request/TestOMRequestUtils.java | 17 +
.../security/acl/TestOzoneNativeAuthorizer.java | 20 +-
.../ozone/web/ozShell/TestObjectPrinter.java | 6 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 15 -
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 2 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 12 +
.../fs/ozone/TestOzoneFileSystemWithMocks.java | 2 +
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 48 +-
.../hadoop/fs/ozone/TestOzoneFsRenameDir.java | 2 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 4 -
.../hadoop/ozone/recon/ReconSchemaManager.java | 55 ++
.../org/apache/hadoop/ozone/recon/ReconServer.java | 26 +-
.../ozone/recon/api/ContainerKeyService.java | 112 ++--
...lizationService.java => TaskStatusService.java} | 39 +-
.../hadoop/ozone/recon/api/UtilizationService.java | 5 -
.../recon/recovery/ReconOMMetadataManager.java | 6 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 11 +
.../spi/impl/OzoneManagerServiceProviderImpl.java | 32 +-
.../ozone/recon/tasks/ReconTaskController.java | 3 +-
.../ozone/recon/tasks/ReconTaskControllerImpl.java | 70 ++-
.../ozone/recon/api/TestTaskStatusService.java | 86 +++
.../recon/tasks/TestReconTaskControllerImpl.java | 38 +-
hadoop-ozone/s3gateway/pom.xml | 4 +
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 4 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 8 +-
.../hadoop/ozone/s3/endpoint/RootEndpoint.java | 4 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 116 ++--
.../hadoop/ozone/freon/RandomKeyGenerator.java | 2 +-
.../freon/TestFreonWithDatanodeFastRestart.java | 4 +-
.../hadoop/ozone/fsck/TestContainerMapper.java | 2 +-
pom.xml | 64 +--
302 files changed, 6094 insertions(+), 4584 deletions(-)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/IncrementalChunkBuffer.java
copy
hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java
=>
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetUtils.java
(56%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/ClosePipelineCommandHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/CreatePipelineCommandHandler.java
delete mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/SmallFileUtils.java
copy
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/{CloseContainerCommand.java
=> ClosePipelineCommand.java} (53%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/CreatePipelineCommand.java
rename {hadoop-ozone/integration-test =>
hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/ContainerTestHelper.java
(65%)
rename {hadoop-ozone/integration-test =>
hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/helpers/TestBlockData.java
(100%)
rename {hadoop-ozone/integration-test =>
hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerDeletionChoosingPolicy.java
(100%)
rename {hadoop-ozone/integration-test =>
hadoop-hdds/container-service}/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java
(99%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineUtils.java
rename {hadoop-ozone/integration-test =>
hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineStateManager.java
(100%)
rename {hadoop-ozone/integration-test =>
hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
(78%)
rename {hadoop-ozone/integration-test =>
hadoop-hdds/server-scm}/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSimplePipelineProvider.java
(100%)
copy {hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm
=>
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline}/pipeline/package-info.java
(100%)
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java
copy
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java
=>
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMLeaderNotReadyException.java
(65%)
rename
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/{NotLeaderException.java
=> OMNotLeaderException.java} (71%)
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/.env
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/README.md
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-ockg.yaml
(71%)
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-rk.yaml
(72%)
rename hadoop-ozone/dist/src/main/compose/{ozones3/docker-config =>
ozone/monitoring.conf} (76%)
create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/monitoring.yaml
rename hadoop-ozone/dist/src/main/compose/{ozone-recon/test.sh =>
ozone/profiling.yaml} (68%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/prometheus.yml
(72%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/startOM.sh =>
ozone/run.sh} (76%)
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/.env
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/README.md
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
delete mode 100755 hadoop-ozone/dist/src/main/compose/ozoneperf/test.sh
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/.env
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml
delete mode 100755 hadoop-ozone/dist/src/main/compose/ozones3/test.sh
copy {hadoop-hdds/tools =>
hadoop-ozone/fault-injection-test/mini-chaos-tests}/pom.xml (68%)
create mode 100755
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/bin/start-chaos.sh
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
(97%)
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
(66%)
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
(90%)
create mode 100644
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/LoadBucket.java
rename
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/chaos =>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils}/TestProbability.java
(97%)
copy hadoop-ozone/{tools/src/main =>
fault-injection-test/mini-chaos-tests/src/test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
(100%)
copy hadoop-ozone/{ozonefs =>
fault-injection-test/mini-chaos-tests}/src/test/resources/log4j.properties (75%)
delete mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestRatisPipelineProvider.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/TestHelper.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMStorage.java
rename
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot
=>
ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis}/TestOMRatisSnapshotInfo.java
(90%)
rename hadoop-ozone/{integration-test =>
ozone-manager}/src/test/java/org/apache/hadoop/ozone/security/acl/TestOzoneNativeAuthorizer.java
(96%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconSchemaManager.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{UtilizationService.java
=> TaskStatusService.java} (60%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTaskStatusService.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
