This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-1564
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from 1938862 HDDS-1576. Support configure more than one raft log storage
to host m… (#117)
new 9ebcb54 HDDS-2505. Fix logic related to SCM address calculation in
HddsUtils
new 431c85d HDDS-2444. Remove server side dependencies from ozonefs jar
files
new 28dc284 HDDS-2619. Fix Sonar issues in ReconTaskControllerImpl
new 2defafa HDDS-2655. Use pre-compiled Pattern in NetUtils#normalize
new 367846d HDDS-2658. Insight log level reset does not work
new a84a9f4 HDDS-2654. Conditionally enable profiling at the kernel level
new f831950 HDDS-2639. TestTableCacheImpl is flaky (#278)
new ab740dc HDDS-2657. Key get command creates the output file even in
case of KEY_NOT_FOUND (#290)
new bb22b3c HDDS-2455. Implement MiniOzoneHAClusterImpl#getOMLeader (#137)
new 27c384f HDDS-2573. Handle InterruptedException in KeyOutputStream.
(#257)
new cdb9e15 HDDS-2637. Handle LeaderNot ready exception in OzoneManager
StateMachine and upgrade ratis to latest version. (#276)
new 8c07266 HDDS-2386. Implement incremental ChunkBuffer. (#205)
new 4efd8aa HDDS-2620. Fix listMultipartupload API. (#263)
new bf9cbf1 HDDS-1722. Use the bindings in ReconSchemaGenerationModule to
create Recon SQL tables on startup. (#294)
new 13c1db0 HDDS-2571. Handle InterruptedException in SCMPipelineManager
(#295)
new 0543da7 HDDS-2656. Prefer execute() over submit() if the returned
Future is ignored (#302)
new 5c21608 HDDS-2475. Unregister ContainerMetadataScrubberMetrics on
thread exit (#286)
new f0ea1a4 HDDS-2653. Improve executor memory usage in new Freon tests
(#284)
new 8b7d7ef HDDS-2608. Provide command to wait until SCM is out from the
safe-mode (#250)
new 3d1b44d HDDS-2646. Start acceptance tests only if at least one THREE
pipeline is available (#282)
new d903120 HDDS-2622. Add config parameter for setting limit on total
bytes of pending requests in Ratis. (#264)
new d7a4ad4 HDDS-2667. Prometheus reports invalid metric type
new 495700c Addendum HDDS-2646. Start acceptance tests only if at least
one THREE pipeline is available (#282)
new 0b5d319 HDDS-2506. Remove keyAllocationInfo and replication info from
the auditLog (#288)
new bf36ab0 HDDS-2565. Handle InterruptedException in VolumeSet
new 0319d42 HDDS-2600. Move chaos test to org.apache.hadoop.ozone.chaos
package. (#307)
new 74bab02 HDDS-1873. Add API to get last completed times for every
Recon task. (#233)
new f2965ac HDDS-2666. Suppress loader constraint violation message in
TestOzoneFileSystemWithMocks (#303)
new 30b9d58 HDDS-2530. Sonar : refactor verifyResourceName in
HddsClientUtils to fix Sonar errors (#293)
new d89f6d4 HDDS-2629. Ozone CLI: CreationTime/modifyTime of
volume/bucket/key in… (#272)
new 93abc66 HDDS-2677. Acceptance test may fail despite success status
(#312)
new 80975cc HDDS-2570. Handle InterruptedException in ProfileServlet.
(#308)
new 7478168 HDDS-1987. Fix listStatus API (#108)
new 4c046d4 HDDS-2636. Refresh pipeline information in OzoneManager
lookupFile call. (#301)
new 6c5d0fb HDDS-2678. Add thread name to log pattern (#315)
new 7657fbd HDDS-2662. Update gRPC and datanode protobuf version in Ozone
new e306d51 HDDS-2681. Add leak detection memory flags to
MiniOzoneChaosCluster
new 0f8239f HDDS-1991. Remove RatisClient in OM HA. (#319)
new 421337b HDDS-2692. Seek to file end throws EOF Exception. (#326)
new b33b64d HDDS-2689. OMException NOT_A_FILE missing space in the
exception message. (#327)
new 1ecd057 HDDS-2687. Fix sonar issues in package
org.apache.hadoop.ozone.recon.api. (#325)
new 6f743cc HDDS-2668. Sonar : fix issues reported in BlockManagerImpl
(#317)
new b324d4a HDDS-2645. Refactor MiniOzoneChaosCluster to a different
package to add filesystem tests. (#292)
new 1b43a92 HDDS-2683. Ratis MaxBuffer should be the same size as the
segment size. (#320)
new 91cab42 HDDS-1993. Merge OzoneManagerRequestHandler and
OzoneManagerHARequest… (#304)
new d0ec961 HDDS-2693. HddsVolume mixes ChunkLayOutVersion and
DataNodeLayoutVersion (#330)
new 554c311 HDDS-2105. Merge OzoneClientFactory#getRpcClient functions
(#148)
new 14f13c9 HDDS-2389. add toStateMachineLogEntryString provider in
Ozone's ContainerStateMachine. (#298)
new 5a507b4 HDDS-2532. Sonar : fix issues in OzoneQuota (#311)
new db20ab5 HDDS-2561. Handle InterruptedException in LeaseManager (#338)
new 86af50f HDDS-2704. Source tar file is not created during the relase
build (#334)
new 3da6188 HDDS-2711. OM does not report JVM metrics (#346)
new 54deaa0 HDDS-2688. Refactor container response builders to
hadoop-hdds/common (#324)
new db7efc7 HDDS-2659. KeyValueHandler#handleCreateContainer should log
the exception on container creation failure (#296).
new 3799c3e HDDS-2588. Consolidate compose environments
new 5707737 HDDS-2684. Refactor common test utilities to
hadoop-hdds/common (#322)
new 551a585 HDDS-2242. Avoid unnecessary rpc needed to discover the
pipeline leader. (#313)
new 71021e4 HDDS-2553. Sonar: Iterator.next() methods should throw
NoSuchElementException
new c5da69e HDDS-2423. Add the recover-trash command client side code
new f94f8f7 HDDS-2414. Simplify robot tests with removing output greps
new 341785d HDDS-2680. Fix updating lastAppliedIndex in
OzoneManagerStateMachine. (#316)
new c311c99 HDDS-2675. Sonar : fix issues in PipelineStateManager (#347)
new 7b2355a HDDS-2578. Handle InterruptedException in RandomKeyGenerator
(#305)
new 88727f7 HDDS-2539. Sonar: Fix sonar warnings in SCMContainerManager
class. (#332)
new 3b6eb17 HDDS-2729. Enable multilingual Hugo features in ozone docs
(#357)
new 7f9368b HDDS-2751. Wrong number of placeholders in log message (#362)
new eb341bd HDDS-2723. Wrap InputStream with try-with-resource in
TestOzoneFSInputStream. (#361)
new bdb0839 HDDS-2724. Sonar : Fix sonar issues in package
'org.apache.hadoop.ozo… (#354)
new 4753d70 HDDS-2669. Sonar : fix issues in scm Pipeline (#360)
new 3ecb968 HDDS-2752. Unnecessary calls to isNoneEmpty and isAllEmpty
(#364)
new c7caff1 HDDS-2630. NullPointerException in S3g. (#336)
new 8158b12 Revert "HDDS-2704. Source tar file is not created during the
relase build (#334)" (#368)
new 552f0b3 HDDS-2623. Expose SCMDatanodeProtocolServer RPC endpoint
through Recon. (#353)
new bd14928 HDDS-2070. Create insight point to debug one specific
pipeline (#299)
new 27a77e9 HDDS-2722. Let ChunkManager read/write ChunkBuffer instead of
ByteBuffer (#352)
new 3a6fa66 HDDS-2703. OzoneFSInputStream to support ByteBufferReadable
(#345)
new 1331aa1 HDDS-2739. No need to try install awscli before each test
(#359)
new a359ab6 HDDS-2652. Remove unnecessary log statement. (#371)
new 21281c6 HDDS-2709. Maven property skipShade should not skip ozonefs
compilation (#342)
new d318ee2 HDDS-2735. Let GitHub Actions run acceptance check in
parallel (#358)
new 79ea2d9 HDDS-2774. Hadoop 3.1 acceptance test fails with apk command
not found (#374)
new f472e8d HDDS-2776. NPE when stop recon while start recon failed (#376)
new 0d91171 HDDS-2762. Sonar warnings fixed (#373)
new 57d3fe5 HDDS-2755. Compare transactionID and updateID of Volume
operations to avoid replaying transactions (#369)
new f99848d HDDS-1989. Fix ApplyTransaction error handling in
OzoneManagerStateMachine. (#351)
new 7a6b6a1 HDDS-2685. Fix Rename API in BasicOzoneFileSystem (#321)
new a96ff61 HDDS-2704. Source tar file is not created during the relase
build. (#380)
new d5034a0 HDDS-2777. Add bytes read statistics to Ozone FileSystem
implementation (#382)
new 51b8cd1 HDDS-2786. ITestOzoneContractSeek zero byte file failures
(#384)
new 417c958 Revert "Rebase Fix"
new 40da915 HDDS-1564. Rebase fix.
The 91 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 6 -
.github/workflows/pr.yml | 6 -
.../hadoop/hdds/scm/client/HddsClientUtils.java | 75 +--
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 3 +-
.../apache/hadoop/hdds/scm/storage/BufferPool.java | 4 +-
hadoop-hdds/common/pom.xml | 9 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 216 ++++----
.../org/apache/hadoop/hdds/client/OzoneQuota.java | 12 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 13 +-
.../ReconConfigKeys.java} | 27 +-
.../apache/hadoop/hdds/recon}/package-info.java | 4 +-
.../hadoop/hdds/scm/ByteStringConversion.java | 16 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 5 +
.../org/apache/hadoop/hdds/scm/net/NetUtils.java | 19 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 25 +-
.../ContainerCommandResponseBuilders.java | 269 +++++++++
.../certificate/authority/DefaultCAServer.java | 4 +-
.../client/DefaultCertificateClient.java | 2 +-
.../hadoop/hdds/utils/RocksDBStoreMBean.java | 2 +-
.../apache/hadoop/hdds/utils/db/DBCheckpoint.java | 12 +
.../hadoop/hdds/utils/db/RocksDBCheckpoint.java | 11 +
.../hadoop/hdds/utils/db/cache/TableCacheImpl.java | 5 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 5 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +
.../apache/hadoop/ozone/common/ChunkBuffer.java | 88 ++-
.../common/ChunkBufferImplWithByteBuffer.java | 38 +-
.../ozone/common/IncrementalChunkBuffer.java | 287 ++++++++++
.../ozone/container/common/helpers/BlockData.java | 5 -
.../apache/hadoop/ozone/lease/LeaseManager.java | 4 +-
.../apache/hadoop/ozone/web/utils/JsonUtils.java | 15 +-
.../common/src/main/resources/ozone-default.xml | 38 +-
.../hadoop/hdds/conf/SimpleConfiguration.java | 2 +-
.../hadoop/hdds/conf/TestOzoneConfiguration.java | 10 +-
.../hadoop/hdds/protocol/MockDatanodeDetails.java | 125 +++++
.../apache/hadoop/hdds/scm/net/TestNetUtils.java | 30 +-
.../hadoop/hdds/scm/pipeline/MockPipeline.java | 74 +++
.../hadoop/hdds/utils/MockGatheringChannel.java | 74 +++
.../hdds/utils/db/cache/TestTableCacheImpl.java | 62 +--
.../hadoop/ozone/audit/TestOzoneAuditLogger.java | 10 +-
.../apache/hadoop/ozone/audit/package-info.java | 2 +-
.../hadoop/ozone/common/TestChunkBuffer.java | 47 +-
.../{log4j2.properties => auditlog.properties} | 0
hadoop-hdds/container-service/pom.xml | 6 +-
.../org/apache/hadoop/hdds/scm/HddsServerUtil.java | 139 ++---
.../container/common/helpers/ContainerUtils.java | 85 +--
.../container/common/impl/HddsDispatcher.java | 7 +-
.../common/statemachine/EndpointStateMachine.java | 2 +-
.../commandhandler/CommandDispatcher.java | 2 +-
.../server/ratis/ContainerStateMachine.java | 30 +-
.../transport/server/ratis/XceiverServerRatis.java | 42 +-
.../ozone/container/common/volume/HddsVolume.java | 3 +-
.../container/common/volume/HddsVolumeChecker.java | 2 +-
.../ozone/container/common/volume/VolumeSet.java | 1 +
.../ozone/container/keyvalue/KeyValueHandler.java | 149 ++---
.../container/keyvalue/helpers/BlockUtils.java | 88 +--
.../container/keyvalue/helpers/ChunkUtils.java | 26 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 27 -
.../container/keyvalue/helpers/SmallFileUtils.java | 91 ---
.../container/keyvalue/impl/BlockManagerImpl.java | 65 +--
.../keyvalue/impl/ChunkManagerDummyImpl.java | 7 +-
.../container/keyvalue/impl/ChunkManagerImpl.java | 13 +-
.../keyvalue/interfaces/ChunkManager.java | 13 +-
.../container/ozoneimpl/ContainerController.java | 9 +
.../ozoneimpl/ContainerDataScrubberMetrics.java | 8 +-
.../ozoneimpl/ContainerMetadataScanner.java | 18 +-
.../ContainerMetadataScrubberMetrics.java | 8 +-
.../replication/ReplicationSupervisor.java | 2 +-
.../ozone/container/ContainerTestHelper.java | 87 +--
.../common/impl/TestContainerPersistence.java | 33 +-
.../container/common/impl/TestHddsDispatcher.java | 6 +-
.../container/common/volume/TestVolumeSet.java | 24 +
.../container/keyvalue/TestChunkManagerImpl.java | 18 +-
.../container/keyvalue/helpers/TestChunkUtils.java | 13 +-
.../ozoneimpl/TestContainerScrubberMetrics.java | 35 +-
hadoop-hdds/docs/config.yaml | 9 +-
hadoop-hdds/docs/content/_index.md | 8 +-
.../default.md => content/_index.zh.md} | 6 +-
.../default.md => content/concept/Datanodes.zh.md} | 8 +-
.../default.md => content/concept/_index.zh.md} | 8 +-
.../themes/ozonedoc/layouts/_default/section.html | 4 +
.../themes/ozonedoc/layouts/_default/single.html | 8 +-
.../docs/themes/ozonedoc/layouts/index.html | 4 +-
.../ozonedoc/layouts/partials/languages.html} | 10 +-
.../apache/hadoop/hdds/server/BaseHttpServer.java | 19 +-
.../apache/hadoop/hdds/server/ProfileServlet.java | 7 +-
.../hadoop/hdds/server/PrometheusMetricsSink.java | 5 +-
.../org/apache/hadoop/hdds/server/ServerUtils.java | 8 +
.../hdds/server/TestPrometheusMetricsSink.java | 2 +-
hadoop-hdds/pom.xml | 7 +
.../hdds/scm/container/SCMContainerManager.java | 48 +-
.../scm/container/states/ContainerStateMap.java | 2 +-
.../scm/pipeline/BackgroundPipelineCreator.java | 2 +-
.../hdds/scm/pipeline/PipelineActionHandler.java | 2 +-
.../hdds/scm/pipeline/PipelineStateManager.java | 7 +-
.../hdds/scm/pipeline/SCMPipelineManager.java | 3 +-
.../scm/safemode/HealthyPipelineSafeModeRule.java | 2 +-
.../scm/server/OzoneStorageContainerManager.java | 30 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 12 +-
.../hdds/scm/server/SCMSecurityProtocolServer.java | 2 +-
.../hdds/scm/server/StorageContainerManager.java | 9 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 5 +-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 74 +--
.../command/TestCommandStatusReportHandler.java | 3 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 4 +-
.../container/TestCloseContainerEventHandler.java | 4 +-
.../scm/container/TestContainerActionsHandler.java | 4 +-
.../scm/container/TestContainerStateManager.java | 12 +-
.../TestIncrementalContainerReportHandler.java | 20 +-
.../hdds/scm/container/TestReplicationManager.java | 4 +-
.../scm/container/TestSCMContainerManager.java | 5 +-
.../algorithms/TestContainerPlacementFactory.java | 4 +-
.../TestSCMContainerPlacementCapacity.java | 4 +-
.../TestSCMContainerPlacementRackAware.java | 6 +-
.../TestSCMContainerPlacementRandom.java | 4 +-
.../hdds/scm/node/TestContainerPlacement.java | 3 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 19 +-
.../hdds/scm/node/TestNodeReportHandler.java | 3 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 10 +-
.../hadoop/hdds/scm/node/TestStatisticsUpdate.java | 5 +-
.../TestPipelineDatanodesIntersection.java | 2 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 2 +-
.../scm/pipeline/TestPipelineStateManager.java | 9 +-
.../scm/pipeline/TestRatisPipelineProvider.java | 10 +-
.../scm/pipeline/TestSimplePipelineProvider.java | 7 +-
.../scm/server/TestSCMBlockProtocolServer.java | 4 +-
.../server/TestSCMDatanodeHeartbeatDispatcher.java | 8 +-
.../ozone/container/common/TestEndPoint.java | 21 +-
.../hadoop/hdds/scm/cli/SafeModeCommands.java | 1 +
.../hdds/scm/cli/SafeModeWaitSubcommand.java | 93 ++++
.../apache/hadoop/ozone/client/OzoneBucket.java | 11 +-
.../hadoop/ozone/client/OzoneClientFactory.java | 68 +--
.../org/apache/hadoop/ozone/client/OzoneKey.java | 14 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 9 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 9 +-
.../client/io/BlockOutputStreamEntryPool.java | 51 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 12 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 153 +++---
.../ozone/client/protocol/ClientProtocol.java | 13 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 8 +
.../hadoop/ozone/client/TestHddsClientUtils.java | 46 ++
hadoop-ozone/common/pom.xml | 8 -
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 62 +--
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 9 -
.../apache/hadoop/ozone/om/OMMetadataManager.java | 3 +-
.../hadoop/ozone/om/exceptions/OMException.java | 4 +-
.../om/exceptions/OMLeaderNotReadyException.java | 22 +-
...derException.java => OMNotLeaderException.java} | 32 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 2 +-
.../hadoop/ozone/om/helpers/OMRatisHelper.java | 43 --
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 2 -
.../ozone/om/protocol/OzoneManagerHAProtocol.java | 6 +-
.../ozone/om/protocol/OzoneManagerProtocol.java | 15 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 90 ++-
.../src/main/proto/OzoneManagerProtocol.proto | 22 +
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 65 +--
hadoop-ozone/csi/pom.xml | 11 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dist/pom.xml | 39 ++
.../dist/src/main/compose/ozone-mr/common-config | 1 +
.../compose/ozone-mr/hadoop27/docker-compose.yaml | 1 +
.../compose/ozone-mr/hadoop31/docker-compose.yaml | 1 +
.../src/main/compose/ozone-mr/hadoop31/test.sh | 8 +-
.../compose/ozone-mr/hadoop32/docker-compose.yaml | 1 +
.../compose/ozone-om-ha-s3/docker-compose.yaml | 1 +
.../dist/src/main/compose/ozone-recon/.env | 18 -
.../main/compose/ozone-recon/docker-compose.yaml | 65 ---
.../src/main/compose/ozone-recon/docker-config | 35 --
.../compose/ozone-topology/docker-compose.yaml | 1 +
.../src/main/compose/ozone-topology/docker-config | 1 +
hadoop-ozone/dist/src/main/compose/ozone/README.md | 99 ++++
.../src/main/compose/ozone/docker-compose.yaml | 91 +--
.../dist/src/main/compose/ozone/docker-config | 11 +-
.../compose/{ozoneperf => ozone}/freon-ockg.yaml | 19 +-
.../compose/{ozoneperf => ozone}/freon-rk.yaml | 19 +-
.../docker-config => ozone/monitoring.conf} | 15 +-
.../dist/src/main/compose/ozone/monitoring.yaml | 56 ++
.../{ozone-recon/test.sh => ozone/profiling.yaml} | 30 +-
.../compose/{ozoneperf => ozone}/prometheus.yml | 20 +-
.../{ozone-om-ha/startOM.sh => ozone/run.sh} | 13 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 9 +-
.../src/main/compose/ozoneblockade/docker-config | 2 +
hadoop-ozone/dist/src/main/compose/ozoneperf/.env | 18 -
.../dist/src/main/compose/ozoneperf/README.md | 74 ---
.../src/main/compose/ozoneperf/docker-compose.yaml | 84 ---
.../dist/src/main/compose/ozoneperf/docker-config | 36 --
.../dist/src/main/compose/ozoneperf/test.sh | 30 -
.../compose/ozones3-haproxy/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/ozones3/.env | 18 -
.../src/main/compose/ozones3/docker-compose.yaml | 58 --
hadoop-ozone/dist/src/main/compose/ozones3/test.sh | 34 --
.../src/main/compose/ozonescripts/docker-config | 3 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 1 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 53 +-
.../dist/src/main/dockerlibexec/entrypoint.sh | 8 +
.../dist/src/main/smoketest/basic/basic.robot | 4 +-
.../src/main/smoketest/basic/ozone-shell.robot | 18 +-
.../dist/src/main/smoketest/gdpr/gdpr.robot | 16 +-
.../dist/src/main/smoketest/omha/testOMHA.robot | 4 +-
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 18 +-
.../src/main/smoketest/s3/MultipartUpload.robot | 2 +-
.../dist/src/main/smoketest/s3/__init__.robot | 2 +-
.../dist/src/main/smoketest/s3/bucketcreate.robot | 2 +-
.../dist/src/main/smoketest/s3/buckethead.robot | 2 +-
.../dist/src/main/smoketest/s3/bucketlist.robot | 2 +-
.../dist/src/main/smoketest/s3/objectcopy.robot | 2 +-
.../dist/src/main/smoketest/s3/objectdelete.robot | 2 +-
.../src/main/smoketest/s3/objectmultidelete.robot | 2 +-
.../dist/src/main/smoketest/s3/objectputget.robot | 2 +-
hadoop-ozone/dist/src/shell/conf/log4j.properties | 10 +-
.../fault-injection-test/mini-chaos-tests}/pom.xml | 37 +-
.../mini-chaos-tests/src/test/bin/start-chaos.sh | 46 ++
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 6 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 107 +---
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 12 +-
.../org/apache/hadoop/ozone/utils/LoadBucket.java | 99 ++++
.../hadoop/ozone/utils}/TestProbability.java | 2 +-
.../services/org.apache.hadoop.fs.FileSystem | 0
.../src/test/resources/log4j.properties | 8 +-
hadoop-ozone/fault-injection-test/pom.xml | 1 +
.../hadoop/ozone/insight/BaseInsightPoint.java | 5 +-
.../ozone/insight/BaseInsightSubCommand.java | 2 +
.../apache/hadoop/ozone/insight/InsightPoint.java | 3 +-
.../apache/hadoop/ozone/insight/LogSubcommand.java | 33 +-
.../ozone/insight/datanode/RatisInsight.java | 44 +-
.../hadoop/ozone/insight/om/KeyManagerInsight.java | 4 +-
.../hadoop/ozone/insight/om/OmProtocolInsight.java | 3 +-
.../ozone/insight/scm/EventQueueInsight.java | 4 +-
.../ozone/insight/scm/NodeManagerInsight.java | 4 +-
.../ozone/insight/scm/ReplicaManagerInsight.java | 4 +-
.../scm/ScmProtocolBlockLocationInsight.java | 3 +-
.../scm/ScmProtocolContainerLocationInsight.java | 3 +-
.../insight/scm/ScmProtocolDatanodeInsight.java | 3 +-
.../insight/scm/ScmProtocolSecurityInsight.java | 3 +-
.../integration-test/src/test/bin/start-chaos.sh | 35 --
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 129 +++++
.../scm/pipeline/TestRatisPipelineProvider.java | 16 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 10 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 24 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 18 +-
.../hadoop/ozone/TestMiniOzoneHACluster.java | 112 ++++
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 4 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 14 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 90 ++-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 8 +-
.../ozone/container/TestContainerReplication.java | 3 +-
.../transport/server/ratis/TestCSMMetrics.java | 4 +-
.../container/metrics/TestContainerMetrics.java | 7 +-
.../container/ozoneimpl/TestOzoneContainer.java | 15 +-
.../ozoneimpl/TestOzoneContainerRatis.java | 4 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 7 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 7 +-
.../container/server/TestContainerServer.java | 11 +-
.../server/TestSecureContainerServer.java | 8 +-
.../hadoop/ozone/dn/scrubber/TestDataScrubber.java | 8 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 235 ++++++++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 20 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 34 +-
.../hadoop/ozone/ozShell/TestOzoneShellHA.java | 20 +-
.../hadoop/ozone/scm/node/TestSCMNodeMetrics.java | 5 +-
hadoop-ozone/ozone-manager/pom.xml | 6 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 172 ++++--
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 19 +-
.../java/org/apache/hadoop/ozone/om/OMStorage.java | 17 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 35 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 88 +--
.../hadoop/ozone/om/ratis/OMRatisSnapshotInfo.java | 15 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 7 +-
.../ozone/om/ratis/OzoneManagerRatisClient.java | 210 -------
.../ozone/om/ratis/OzoneManagerRatisServer.java | 119 ++--
.../ozone/om/ratis/OzoneManagerRatisSnapshot.java | 10 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 199 ++++++-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 16 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 23 +
.../hadoop/ozone/om/request/RequestAuditor.java | 3 -
.../ozone/om/request/file/OMFileCreateRequest.java | 6 +-
.../request/s3/bucket/S3BucketCreateRequest.java | 18 +-
.../om/request/volume/OMVolumeCreateRequest.java | 26 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 51 +-
.../ozone/om/request/volume/OMVolumeRequest.java | 20 +
.../om/request/volume/OMVolumeSetOwnerRequest.java | 24 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 24 +-
.../hadoop/ozone/om/response/OMClientResponse.java | 29 +-
.../response/file/OMDirectoryCreateResponse.java | 28 +-
.../response/s3/bucket/S3BucketCreateResponse.java | 6 +-
.../S3MultipartUploadCommitPartResponse.java | 71 +--
.../om/response/volume/OMVolumeAclOpResponse.java | 2 -
.../om/response/volume/OMVolumeCreateResponse.java | 38 +-
.../om/response/volume/OMVolumeDeleteResponse.java | 42 +-
.../response/volume/OMVolumeSetOwnerResponse.java | 56 +-
.../response/volume/OMVolumeSetQuotaResponse.java | 26 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 10 +
.../protocolPB/OzoneManagerHARequestHandler.java | 39 --
.../OzoneManagerHARequestHandlerImpl.java | 56 --
...OzoneManagerProtocolServerSideTranslatorPB.java | 27 +-
.../protocolPB/OzoneManagerRequestHandler.java | 607 +--------------------
.../hadoop/ozone/protocolPB/RequestHandler.java | 28 +-
.../hadoop/ozone/web/ozShell/OzoneAddress.java | 3 +-
.../ozone/web/ozShell/keys/GetKeyHandler.java | 4 +-
.../ozone/om/ScmBlockLocationTestingClient.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 256 ++++++++-
.../org/apache/hadoop/ozone/om/TestOMStorage.java | 91 +++
.../apache/hadoop/ozone/om/TestTrashService.java | 125 +++++
.../ozone/om/ratis/TestOMRatisSnapshotInfo.java | 5 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 2 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 3 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 29 +-
.../om/ratis/TestOzoneManagerStateMachine.java | 215 ++++++++
.../ozone/om/request/TestOMRequestUtils.java | 17 +
.../request/volume/TestOMVolumeCreateRequest.java | 51 +-
.../request/volume/TestOMVolumeDeleteRequest.java | 46 +-
.../om/request/volume/TestOMVolumeRequest.java | 30 +
.../volume/TestOMVolumeSetOwnerRequest.java | 31 ++
.../volume/TestOMVolumeSetQuotaRequest.java | 38 +-
.../ozone/om/response/TestOMResponseUtils.java | 2 +-
.../volume/TestOMVolumeCreateResponse.java | 7 +-
.../volume/TestOMVolumeDeleteResponse.java | 13 +-
.../volume/TestOMVolumeSetOwnerResponse.java | 11 +-
.../volume/TestOMVolumeSetQuotaResponse.java | 7 +-
.../ozone/web/ozShell/TestObjectPrinter.java | 6 +-
hadoop-ozone/ozonefs-lib-current/pom.xml | 15 -
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 2 +-
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 36 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 45 +-
.../hadoop/fs/ozone/TestOzoneFSInputStream.java | 72 ++-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 12 +
.../fs/ozone/TestOzoneFileSystemWithMocks.java | 2 +
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 48 +-
.../hadoop/fs/ozone/TestOzoneFsRenameDir.java | 16 +-
.../hadoop/fs/ozone/TestReadWriteStatistics.java | 452 +++++++++++++++
hadoop-ozone/pom.xml | 2 +-
hadoop-ozone/recon/pom.xml | 5 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 6 +-
.../hadoop/ozone/recon/ReconSchemaManager.java | 55 ++
.../org/apache/hadoop/ozone/recon/ReconServer.java | 47 +-
.../ozone/recon/api/ContainerKeyService.java | 112 ++--
...lizationService.java => TaskStatusService.java} | 39 +-
.../hadoop/ozone/recon/api/UtilizationService.java | 5 -
.../recon/recovery/ReconOMMetadataManager.java | 6 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 11 +
.../recon/scm/ReconDatanodeProtocolServer.java | 76 +++
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 29 +-
.../recon/scm/ReconStorageContainerManager.java | 134 +++++
.../recon/{recovery => scm}/package-info.java | 2 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 32 +-
.../ozone/recon/tasks/ContainerKeyMapperTask.java | 8 +-
.../ozone/recon/tasks/FileSizeCountTask.java | 12 +-
.../ozone/recon/tasks/OMDBUpdatesHandler.java | 79 ++-
.../ozone/recon/tasks/ReconTaskController.java | 3 +-
.../ozone/recon/tasks/ReconTaskControllerImpl.java | 70 ++-
.../ozone/recon/api/TestTaskStatusService.java | 86 +++
.../recon/tasks/TestReconTaskControllerImpl.java | 38 +-
hadoop-ozone/s3gateway/pom.xml | 5 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 4 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 8 +-
.../hadoop/ozone/s3/endpoint/RootEndpoint.java | 4 +-
.../hadoop/ozone/s3/exception/OS3Exception.java | 2 +-
.../s3/header/AuthenticationHeaderParser.java | 4 +
.../ozone/s3/header/AuthorizationHeaderV4.java | 18 +-
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 114 ++++
hadoop-ozone/tools/pom.xml | 1 -
.../hadoop/ozone/freon/BaseFreonGenerator.java | 116 ++--
.../hadoop/ozone/freon/RandomKeyGenerator.java | 8 +-
hadoop-ozone/upgrade/pom.xml | 1 -
pom.xml | 71 +--
365 files changed, 7053 insertions(+), 4472 deletions(-)
copy
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/{utils/MetaStoreIterator.java
=> recon/ReconConfigKeys.java} (58%)
copy {hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli =>
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/recon}/package-info.java
(90%)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ContainerCommandResponseBuilders.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/common/IncrementalChunkBuffer.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/protocol/MockDatanodeDetails.java
copy
hadoop-ozone/recon-codegen/src/main/java/org/hadoop/ozone/recon/schema/ReconSchemaDefinition.java
=>
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/net/TestNetUtils.java
(56%)
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockPipeline.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/utils/MockGatheringChannel.java
rename hadoop-hdds/common/src/test/resources/{log4j2.properties =>
auditlog.properties} (100%)
delete mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/SmallFileUtils.java
copy hadoop-hdds/docs/{archetypes/default.md => content/_index.zh.md} (93%)
copy hadoop-hdds/docs/{archetypes/default.md =>
content/concept/Datanodes.zh.md} (88%)
copy hadoop-hdds/docs/{archetypes/default.md => content/concept/_index.zh.md}
(92%)
copy hadoop-hdds/docs/{static/NOTES.md =>
themes/ozonedoc/layouts/partials/languages.html} (77%)
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartUploadList.java
=>
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/OzoneStorageContainerManager.java
(54%)
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/SafeModeWaitSubcommand.java
copy
hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/ConfigurationException.java
=>
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMLeaderNotReadyException.java
(65%)
rename
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/{NotLeaderException.java
=> OMNotLeaderException.java} (71%)
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/.env
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozone-recon/docker-config
create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/README.md
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-ockg.yaml
(71%)
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/freon-rk.yaml
(72%)
rename hadoop-ozone/dist/src/main/compose/{ozones3/docker-config =>
ozone/monitoring.conf} (76%)
create mode 100644 hadoop-ozone/dist/src/main/compose/ozone/monitoring.yaml
rename hadoop-ozone/dist/src/main/compose/{ozone-recon/test.sh =>
ozone/profiling.yaml} (68%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/compose/{ozoneperf => ozone}/prometheus.yml
(72%)
copy hadoop-ozone/dist/src/main/compose/{ozone-om-ha/startOM.sh =>
ozone/run.sh} (76%)
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/.env
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/README.md
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
delete mode 100755 hadoop-ozone/dist/src/main/compose/ozoneperf/test.sh
delete mode 100644 hadoop-ozone/dist/src/main/compose/ozones3/.env
delete mode 100644
hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml
delete mode 100755 hadoop-ozone/dist/src/main/compose/ozones3/test.sh
copy {hadoop-hdds/tools =>
hadoop-ozone/fault-injection-test/mini-chaos-tests}/pom.xml (68%)
create mode 100755
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/bin/start-chaos.sh
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneChaosCluster.java
(97%)
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/MiniOzoneLoadGenerator.java
(66%)
rename hadoop-ozone/{integration-test =>
fault-injection-test/mini-chaos-tests}/src/test/java/org/apache/hadoop/ozone/TestMiniChaosOzoneCluster.java
(90%)
create mode 100644
hadoop-ozone/fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils/LoadBucket.java
rename
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/chaos =>
fault-injection-test/mini-chaos-tests/src/test/java/org/apache/hadoop/ozone/utils}/TestProbability.java
(97%)
copy hadoop-ozone/{tools/src/main =>
fault-injection-test/mini-chaos-tests/src/test}/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
(100%)
copy hadoop-ozone/{ozonefs =>
fault-injection-test/mini-chaos-tests}/src/test/resources/log4j.properties (75%)
delete mode 100755 hadoop-ozone/integration-test/src/test/bin/start-chaos.sh
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestRatisPipelineLeader.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestMiniOzoneHACluster.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisClient.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandler.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerHARequestHandlerImpl.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMStorage.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestTrashService.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java
create mode 100644
hadoop-ozone/ozonefs/src/test/java/org/apache/hadoop/fs/ozone/TestReadWriteStatistics.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconSchemaManager.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/{UtilizationService.java
=> TaskStatusService.java} (60%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconDatanodeProtocolServer.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/PendingDeleteHandler.java
=>
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconNodeManager.java
(54%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManager.java
copy hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/{recovery
=> scm}/package-info.java (95%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestTaskStatusService.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketPut.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
