This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-5713
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 891f2fb83c HDDS-12796. [DiskBalancer] Exclude UNHEALTHY DNs from
status subcommand output (#8286)
add 2a4de14eb8 HDDS-12772. Configure initial heartbeat and first election
time for quicker MiniOzoneCluster startup (#8235)
add 25b7102e30 HDDS-12775. flaky-test-check builds the workflow branch
(#8237)
add bf20540044 HDDS-12458. Refactor DataNodeSafeModeRule to use
NodeManager (#7998)
add c962b98cef HDDS-12404. Grafana dashboard for snapshot metrics (#7954)
add 5733c5556e HDDS-11799. Remove config
hdds.scm.safemode.pipeline-availability.check (#8095)
add a0387afc88 HDDS-12800. Bump aspectj-maven-plugin to 1.14.1 (#8230)
add fa8afa339c HDDS-12741. Recon UI should show full path from
ContainerKeys API response (#8200)
add 189fbdbe4c HDDS-12463. Add perf metrics in SCM for allocateBlock and
openPipeline (#8111)
add 1ebf2652b1 HDDS-12580. Set up Temporary DB for Storing Container Log
Information (#8072)
add a1345f9fdd HDDS-12368. Seek to correct start key in
KeyManagerImpl#getTableEntries (#7925)
add 102ae3fd26 HDDS-8660. Notify ReplicationManager when nodes go dead or
out of service (#7997)
add dc9952e446 HDDS-12746. Reduce visibility of Logger members (#8210)
add f22f32e05d HDDS-12764. NodeDecommissionManager#monitorInterval should
get with ms unit from config (#8250)
add 25dda2d954 HDDS-12408. Create mixin for ContainerID list parameters
(#7970)
add 691383526a HDDS-12519. Generate auto-complete script for Ozone
commands (#8030)
add fbb8706a7b HDDS-12145. Remove unused config
hdds.container.ratis.replication.level (#8252)
add 7ff8ad7a13 HDDS-12755. Redundant declaration in
TestHadoopNestedDirGenerator#spanCheck() (#8218)
add 7062685609 HDDS-12805. Use slf4j for HTTP request log (#8255)
add fdebdacc18 HDDS-12761. Add missing network configuration properties in
ozone-default.xml (#8257)
add 0d6231c2b3 HDDS-12801. SCM should remove pipeline before DN. (#8261)
add 22734a91f2 HDDS-12461. Bump Ranger to 2.6.0 (#8120)
add a21e362b6a HDDS-12547. Container creation and import use the same
VolumeChoosingPolicy (#8090)
add 7dfd8c1a78 HDDS-12060. Replace System.currentTimeMillis() with
Time.monotonicNow() for duration calculation (#8096)
add eed5924251 HDDS-12770. Use ContainerID instead of Long in
CONTAINER_IDS_TABLE. (#8247)
add a92fe59ec4 HDDS-12581. Multi-threaded Log File Parsing with Batch
Updates to DB (#8254)
add e6b9d5a123 HDDS-12825. ReconIncrementalContainerReportHandler is not
synchronized on datanode. (#8272)
add 5292ac57d1 HDDS-12836. Bump junit to 5.12.2 (#8269)
add 9e4da5bac2 HDDS-12821. Update Build from Source user doc. (#8262)
add 3d8644e172 HDDS-12830. Add RocksDatabaseException. (#8277)
add 9b48b6e8c1 HDDS-12838. Bump awssdk to 2.31.21 (#8267)
add a27c1cf3a6 HDDS-12837. Bump vite to 4.5.13 (#8283)
add bb2c63c502 HDDS-12803. OmKeyInfo#isKeyInfoSame should handle object
tags (#8259)
add c7117dcc17 HDDS-11974. Split Container Safemode Rule into Ratis & EC
Container Safemode Rules (#7951)
add f26e2f06a3 HDDS-12771. xcompat fails if run in itself due to undefined
OZONE_CURRENT_VERSION (#8236)
add a95d3389e0 HDDS-12740. Use DatanodeID in HddsTestUtils (#8281)
add b33ed23605 HDDS-12412. Make hdds-common compliant with
FieldDeclarationsShouldBeAtStartOfClass (#7967)
add 62df3069f8 HDDS-12493. Move container upgrade under repair (#8205)
add 8a4deeb1cf HDDS-12827. Move out NodeStateMap.Entry and
ContainerMap.Entry (#8273)
add 55f6924a1f HDDS-12845. Reuse TestDataUtil.createKey in
OzoneRpcClientTests (#8274)
add b91e8e732f HDDS-12518. Auto-compact tables which can tend to be large
in size at intervals (#8260)
add ef8e0da2c1 HDDS-12524. Reuse TestDataUtil.createKey in more tests
(#8270)
add 978dd717f8 HDDS-12806. Replace commons-logging with jcl-over-slf4j
(#8265)
add 3d4b5fdf55 HDDS-6631. Fix typos in output/exception messages (#8294)
add 93613c1cae HDDS-12734. Enable native lib in CI checks (#8190)
add 3c9e1465e9 HDDS-12350. Reduce duplication between OmBucketReadWrite
tests (#8241)
add f6a4a48b0b HDDS-12840. Avoid long containerId in KeyValueContainer
(#8291)
add e8af186de1 HDDS-12152. Stop testing with Hadoop 3.1.2 (#7773)
add e9e149336b HDDS-12704. Add missing audit logs for SCM client
operations (#8251)
add 5c91b44ad1 HDDS-8802. Added pagination support for ListSnapshotDiff
jobs (#8124)
add d8a391558f HDDS-12413. Move field declarations to start of class in
hdds-container-service (#7968)
add e613598768 HDDS-12113. Move HAProxy test to HA environment (#8271)
add 889f3b7de8 HDDS-10284. Move GenericTestUtils#getTempPath to
MiniOzoneCluster (#8300)
add 213c2fe6a0 HDDS-11734. Bump maven-compiler-plugin to 3.14.0 (#8301)
add 205cb2cc4d HDDS-12846. Log DatanodeDetails instead of
DatanodeDetails.getUuidString (#8295)
add 3eed6a6f37 HDDS-12847. Use DatanodeID instead of
DatanodeDetails.getUuidString (#8293)
add 97b35eeddc HDDS-12874. Bump picocli to 4.7.7 (#8303)
add db9076e11e HDDS-12867. Replace hard-coded namespace URL with constant
S3_XML_NAMESPACE (#8299)
add 84b2ff6d66 HDDS-12876. Bump awssdk to 2.31.25 (#8304)
add 3ee57e6052 HDDS-12813. Replace calls to deprecated RandomUtils methods
(#8302)
add 7c23cbdf66 HDDS-12504. Replace calls to deprecated RandomStringUtils
methods (#8306)
add b26d09ec9f Merge remote-tracking branch 'origin/master' into HDDS-5713
No new revisions were added by this update.
Summary of changes:
.github/workflows/ci.yml | 21 +-
.github/workflows/intermittent-test-check.yml | 2 +
dev-support/ci/categorize_basic_checks.sh | 2 +-
dev-support/ci/selective_ci_checks.bats | 26 +-
dev-support/ci/selective_ci_checks.sh | 38 --
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 4 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 10 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 7 +-
.../hdds/scm/storage/BlockDataStreamOutput.java | 2 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 2 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 2 +-
.../apache/hadoop/hdds/scm/storage/BufferPool.java | 2 +-
.../hdds/scm/storage/TestBlockInputStream.java | 8 +-
.../storage/TestBlockOutputStreamCorrectness.java | 2 +-
.../hadoop/hdds/scm/storage/TestBufferPool.java | 6 +-
.../TestECBlockReconstructedStripeInputStream.java | 2 +-
hadoop-hdds/common/pom.xml | 4 +
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 22 +-
.../java/org/apache/hadoop/hdds/HddsIdFactory.java | 5 +-
.../java/org/apache/hadoop/hdds/StringUtils.java | 3 +-
.../apache/hadoop/hdds/cli/AbstractSubcommand.java | 5 +
.../org/apache/hadoop/hdds/cli/GenericCli.java | 3 +-
.../hadoop/hdds/cli/GenericParentCommand.java | 2 +
.../hadoop/hdds/client/ECReplicationConfig.java | 29 +-
.../org/apache/hadoop/hdds/client/OzoneQuota.java | 46 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 41 +-
.../apache/hadoop/hdds/fs/SpaceUsageSource.java | 3 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 19 +-
.../hdds/ratis/ContainerCommandRequestMessage.java | 9 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 14 +
.../hadoop/hdds/ratis/conf/RatisClientConfig.java | 257 +++++-----
.../apache/hadoop/hdds/recon/ReconConfigKeys.java | 16 +-
.../java/org/apache/hadoop/hdds/scm/ScmConfig.java | 13 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 13 +-
.../apache/hadoop/hdds/scm/XceiverClientSpi.java | 5 +-
.../hadoop/hdds/scm/container/ContainerID.java | 8 +-
.../hadoop/hdds/scm/container/ContainerInfo.java | 8 +-
.../scm/container/ReplicationManagerReport.java | 88 ++--
.../apache/hadoop/hdds/scm/net/InnerNodeImpl.java | 25 +-
.../apache/hadoop/hdds/scm/net/NetConstants.java | 7 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 2 +-
.../org/apache/hadoop/hdds/scm/net/NodeSchema.java | 70 +--
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 8 +-
.../hadoop/hdds/scm/pipeline/PipelineID.java | 4 +-
.../ContainerCommandResponseBuilders.java | 2 +-
.../hadoop/hdds/scm/protocolPB/OzonePBHelper.java | 12 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 7 +-
.../hadoop/hdds/security/SecurityConstants.java | 3 +-
.../hdds/security/ssl/ReloadingX509KeyManager.java | 2 +-
.../security/ssl/ReloadingX509TrustManager.java | 9 +-
.../org/apache/hadoop/hdds/server/JsonUtils.java | 2 +-
.../apache/hadoop/hdds/tracing/StringCodec.java | 2 +-
.../hadoop/hdds/utils/BackgroundService.java | 3 +-
.../apache/hadoop/hdds/utils/db/CodecBuffer.java | 52 +-
.../hadoop/hdds/utils/db/DelegatedCodec.java | 30 +-
.../apache/hadoop/hdds/utils/db/Proto2Codec.java | 14 +-
.../apache/hadoop/hdds/utils/db/Proto3Codec.java | 14 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 8 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 23 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 20 +-
.../apache/hadoop/ozone/common/ChecksumCache.java | 2 +-
.../ozone/common/PureJavaCrc32ByteBuffer.java | 13 +-
.../ozone/common/PureJavaCrc32CByteBuffer.java | 13 +-
.../hadoop/ozone/common/utils/BufferUtils.java | 2 +-
.../ozone/container/common/helpers/BlockData.java | 8 +-
.../container/common/helpers/ChunkInfoList.java | 8 +-
.../org/apache/hadoop/ozone/util/CacheMetrics.java | 53 +-
.../hadoop/ozone/util/ShutdownHookManager.java | 19 +-
.../hadoop/ozone/util/StringWithByteString.java | 8 +-
.../org/apache/hadoop/ozone/util/UUIDUtil.java | 10 +-
.../common/src/main/resources/ozone-default.xml | 120 ++++-
.../hadoop/hdds/scm/net/TestNodeSchemaLoader.java | 11 +-
.../hdds/security/x509/CertificateTestUtils.java | 4 +-
.../apache/hadoop/ozone/common/TestChecksum.java | 4 +-
.../ozone/common/TestChecksumByteBuffer.java | 3 +-
.../hadoop/ozone/common/TestChecksumCache.java | 3 -
.../common/TestChecksumImplsComputeSameValues.java | 4 +-
.../apache/hadoop/hdds/conf/TimeDurationUtil.java | 2 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 9 +-
.../apache/hadoop/ozone/HddsPolicyProvider.java | 13 +-
.../common/helpers/CommandHandlerMetrics.java | 47 +-
.../ozone/container/common/impl/ContainerSet.java | 21 +-
.../container/common/impl/HddsDispatcher.java | 3 +-
.../common/impl/OpenContainerBlockMap.java | 21 +-
.../container/common/interfaces/Container.java | 36 +-
.../ozone/container/common/interfaces/Handler.java | 9 +-
.../common/report/ContainerReportPublisher.java | 3 +-
.../common/report/PipelineReportPublisher.java | 3 +-
.../common/statemachine/DatanodeConfiguration.java | 273 +++++-----
.../common/statemachine/DatanodeStateMachine.java | 16 +-
.../common/statemachine/StateContext.java | 8 +-
.../ClosePipelineCommandHandler.java | 7 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 2 +-
.../states/endpoint/HeartbeatEndpointTask.java | 5 +-
.../states/endpoint/VersionEndpointTask.java | 3 +-
.../server/ratis/ContainerStateMachine.java | 61 ++-
.../transport/server/ratis/DispatcherContext.java | 30 +-
.../transport/server/ratis/XceiverServerRatis.java | 57 ++-
.../container/common/utils/DiskCheckUtil.java | 5 +-
.../container/common/utils/HddsVolumeUtil.java | 5 +-
.../volume/CapacityVolumeChoosingPolicy.java | 2 +-
.../volume/RoundRobinVolumeChoosingPolicy.java | 2 +-
.../container/common/volume/StorageVolume.java | 66 ++-
.../common/volume/StorageVolumeChecker.java | 2 +-
.../common/volume/ThrottledAsyncChecker.java | 2 +-
.../common/volume/VolumeChoosingPolicyFactory.java | 11 +-
.../container/keyvalue/KeyValueContainer.java | 169 +++----
.../container/keyvalue/KeyValueContainerCheck.java | 4 +
.../KeyValueContainerMetadataInspector.java | 15 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 22 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 6 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 2 +-
.../keyvalue/impl/KeyValueStreamDataChannel.java | 13 +-
.../container/metadata/AbstractDatanodeStore.java | 2 +-
.../metadata/DatanodeSchemaTwoDBDefinition.java | 12 +-
.../metadata/WitnessedContainerDBDefinition.java | 8 +-
.../metadata/WitnessedContainerMetadataStore.java | 3 +-
.../WitnessedContainerMetadataStoreImpl.java | 7 +-
.../AbstractBackgroundContainerScanner.java | 2 +-
.../ozoneimpl/BackgroundContainerDataScanner.java | 2 +-
.../BackgroundContainerMetadataScanner.java | 2 +-
.../ozoneimpl/OnDemandContainerDataScanner.java | 2 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 25 +-
.../replication/AbstractReplicationTask.java | 24 +-
.../container/replication/ContainerImporter.java | 10 +-
.../replication/CopyContainerCompression.java | 10 +-
.../replication/DownloadAndImportReplicator.java | 2 +-
.../replication/SimpleContainerDownloader.java | 2 +-
.../upgrade/DatanodeSchemaV2FinalizeAction.java | 2 +-
.../upgrade/DatanodeSchemaV3FinalizeAction.java | 2 +-
.../hadoop/ozone/TestHddsSecureDatanodeInit.java | 6 +-
.../ozone/container/common/ContainerTestUtils.java | 21 +-
.../hadoop/ozone/container/common/ScmTestMock.java | 17 +-
.../container/common/TestBlockDeletingService.java | 12 +-
.../container/common/TestDatanodeStateMachine.java | 4 +
.../TestSchemaTwoBackwardsCompatibility.java | 4 +-
.../container/common/helpers/TestBlockData.java | 3 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 2 +-
.../common/impl/TestContainerPersistence.java | 11 +-
.../container/common/impl/TestHddsDispatcher.java | 28 +-
.../container/common/interfaces/TestHandler.java | 4 +-
.../common/volume/TestPeriodicVolumeChecker.java | 2 +-
.../common/volume/TestStorageVolumeChecker.java | 2 +-
.../container/common/volume/TestVolumeSet.java | 4 +-
.../common/volume/TestVolumeSetDiskChecks.java | 10 +-
.../keyvalue/ContainerTestVersionInfo.java | 40 +-
.../container/keyvalue/TestKeyValueContainer.java | 3 +-
.../TestKeyValueContainerIntegrityChecks.java | 2 +-
.../TestKeyValueContainerMarkUnhealthy.java | 2 +-
.../TestKeyValueContainerMetadataInspector.java | 9 +-
.../container/keyvalue/TestKeyValueHandler.java | 32 +-
.../TestKeyValueHandlerWithUnhealthyContainer.java | 4 -
.../keyvalue/impl/CommonChunkManagerTestCases.java | 2 +-
.../impl/TestKeyValueStreamDataChannel.java | 2 +-
.../container/ozoneimpl/TestContainerReader.java | 11 +-
.../ReplicationSupervisorScheduling.java | 5 +-
.../replication/TestContainerImporter.java | 10 +-
.../replication/TestGrpcContainerUploader.java | 2 +-
.../replication/TestReplicationSupervisor.java | 18 +-
.../TestSendContainerRequestHandler.java | 7 +-
.../ozone/container/upgrade/UpgradeTestHelper.java | 3 +-
hadoop-hdds/docs/content/start/FromSource.md | 91 +++-
.../rawcoder/AbstractNativeRawDecoder.java | 2 +-
.../rawcoder/AbstractNativeRawEncoder.java | 2 +-
.../ozone/erasurecode/rawcoder/util/CodecUtil.java | 2 +-
hadoop-hdds/framework/pom.xml | 4 -
.../ReconfigureProtocolClientSideTranslatorPB.java | 2 +-
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 2 +-
.../SCMBlockLocationFailoverProxyProvider.java | 2 +-
.../SCMContainerLocationFailoverProxyProvider.java | 2 +-
.../SCMSecurityProtocolFailoverProxyProvider.java | 2 +-
.../SecretKeyProtocolFailoverProxyProvider.java | 2 +-
.../SingleSecretKeyProtocolProxyProvider.java | 2 +-
.../hadoop/hdds/server/events/EventWatcher.java | 5 +-
.../hadoop/hdds/server/http/HttpRequestLog.java | 106 ----
.../hdds/server/http/HttpRequestLogAppender.java | 62 ---
.../hadoop/hdds/server/http/HttpServer2.java | 24 +-
.../hadoop/hdds/utils/DBCheckpointServlet.java | 2 +-
.../java/org/apache/hadoop/hdds/utils/HAUtils.java | 2 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 48 +-
.../hadoop/hdds/utils/ProtocolMessageMetrics.java | 5 +-
.../hadoop/hdds/utils/db/DBConfigFromFile.java | 10 +-
.../hadoop/hdds/utils/db/DumpFileWriter.java | 46 --
.../hadoop/hdds/utils/db/RDBSstFileLoader.java | 22 +-
.../hadoop/hdds/utils/db/RDBSstFileWriter.java | 26 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 27 +-
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 17 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 157 +++---
.../hadoop/hdds/utils/db/cache/FullTableCache.java | 2 +-
.../hdds/utils/db/cache/PartialTableCache.java | 2 +-
.../apache/hadoop/ozone/common/StorageInfo.java | 2 +-
.../hadoop/ozone/grpc/metrics/GrpcMetrics.java | 2 +-
.../ozone/upgrade/BasicUpgradeFinalizer.java | 7 +-
.../hdds/fs/TestCachingSpaceUsageSource.java | 2 +-
.../hadoop/hdds/fs/TestSpaceUsageFactory.java | 4 +-
.../security/ssl/TestReloadingX509KeyManager.java | 2 +-
.../ssl/TestReloadingX509TrustManager.java | 2 +-
.../security/symmetric/TestManagedSecretKey.java | 6 +-
.../token/TestOzoneBlockTokenIdentifier.java | 2 +-
.../certificate/authority/TestDefaultCAServer.java | 44 +-
.../client/TestDefaultCertificateClient.java | 11 +-
.../client/TestRootCaRotationPoller.java | 6 +-
.../certificate/utils/TestCertificateCodec.java | 6 +-
.../hadoop/hdds/server/http/TestHtmlQuoting.java | 4 +-
.../hdds/server/http/TestHttpRequestLog.java | 51 --
.../server/http/TestHttpRequestLogAppender.java | 40 --
.../hdds/server/http/TestHttpServer2Metrics.java | 8 +-
.../hadoop/hdds/utils/MapBackedTableIterator.java | 83 +++
.../hadoop/hdds/utils/TestRDBSnapshotProvider.java | 4 +-
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 12 +-
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 42 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 52 +-
.../hdds/utils/db/TestTypedRDBTableStore.java | 42 +-
.../hadoop/hdds/utils/db/TestTypedTable.java | 147 ++++++
.../hadoop/hdds/utils/db/cache/TestTableCache.java | 2 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 4 +
.../hdds/utils/db/RocksDatabaseException.java} | 30 +-
.../hdds/utils/db/managed/ManagedRocksDB.java | 15 +-
.../utils/db/managed/ManagedRocksObjectUtils.java | 12 +-
.../apache/hadoop/hdds/utils/db/package-info.java | 4 +-
hadoop-hdds/rocks-native/pom.xml | 1 -
.../apache/hadoop/hdds/utils/NativeConstants.java | 1 +
.../utils/db/managed/ManagedRawSSTFileReader.java | 9 +
.../hadoop/hdds/utils/TestNativeLibraryLoader.java | 5 +-
.../db/managed/TestManagedRawSSTFileIterator.java | 8 +-
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 5 -
.../ozone/rocksdb/util/TestSstFileSetReader.java | 11 +-
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 13 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 7 -
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 2 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 2 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 2 +-
.../SCMDeletedBlockTransactionStatusManager.java | 4 +-
.../container/AbstractContainerReportHandler.java | 25 +-
.../scm/container/CloseContainerEventHandler.java | 2 +-
.../hdds/scm/container/ContainerReportHandler.java | 3 +-
.../IncrementalContainerReportHandler.java | 39 +-
.../balancer/AbstractFindTargetGreedy.java | 12 +-
.../scm/container/balancer/ContainerBalancer.java | 2 +-
.../container/balancer/ContainerBalancerTask.java | 36 +-
.../scm/container/balancer/FindSourceGreedy.java | 7 +-
.../FindTargetGreedyByNetworkTopology.java | 2 +-
.../balancer/FindTargetGreedyByUsageInfo.java | 2 +-
.../hdds/scm/container/balancer/MoveManager.java | 4 +-
.../algorithms/SCMContainerPlacementCapacity.java | 2 +-
.../algorithms/SCMContainerPlacementRackAware.java | 4 +-
.../SCMContainerPlacementRackScatter.java | 4 +-
.../algorithms/SCMContainerPlacementRandom.java | 2 +-
.../placement/metrics/SCMPerformanceMetrics.java | 14 +
.../replication/ECOverReplicationHandler.java | 2 +-
.../replication/ECUnderReplicationHandler.java | 2 +-
.../replication/MisReplicationHandler.java | 2 +-
.../QuasiClosedStuckUnderReplicationHandler.java | 2 +-
.../replication/RatisOverReplicationHandler.java | 2 +-
.../replication/RatisUnderReplicationHandler.java | 2 +-
.../container/replication/ReplicationManager.java | 44 +-
...er.java => ReplicationManagerEventHandler.java} | 27 +-
.../replication/ReplicationManagerUtil.java | 2 +-
.../container/replication/ReplicationQueue.java | 4 +
.../health/ClosedWithUnhealthyReplicasHandler.java | 2 +-
.../health/DeletingContainerHandler.java | 4 +-
.../replication/health/EmptyContainerHandler.java | 4 +-
.../health/MismatchedReplicasHandler.java | 2 +-
.../health/QuasiClosedContainerHandler.java | 2 +-
.../health/QuasiClosedStuckReplicationCheck.java | 2 +-
.../health/RatisReplicationCheckHandler.java | 2 +-
.../RatisUnhealthyReplicationCheckHandler.java | 2 +-
.../health/VulnerableUnhealthyReplicasHandler.java | 2 +-
.../hdds/scm/container/states/ContainerEntry.java | 55 ++
.../scm/container/states/ContainerStateMap.java | 52 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 4 +
.../apache/hadoop/hdds/scm/ha/HASecurityUtils.java | 2 +-
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 2 +
.../hdds/scm/ha/SCMHADBTransactionBufferImpl.java | 2 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 4 -
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 2 +-
.../scm/ha/SCMHATransactionBufferMonitorTask.java | 2 +-
.../apache/hadoop/hdds/scm/ha/SCMNodeDetails.java | 5 -
.../scm/ha/StatefulServiceStateManagerImpl.java | 2 +-
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 17 +-
.../hdds/scm/node/HealthyReadOnlyNodeHandler.java | 2 +-
.../hdds/scm/node/NodeDecommissionManager.java | 23 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 37 +-
.../hadoop/hdds/scm/node/states/DatanodeEntry.java | 61 +++
.../hadoop/hdds/scm/node/states/NodeStateMap.java | 59 +--
.../hdds/scm/pipeline/PipelineActionHandler.java | 4 +-
.../hdds/scm/pipeline/PipelineManagerImpl.java | 10 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 7 +-
.../hdds/scm/pipeline/RatisPipelineProvider.java | 2 +-
.../hdds/scm/pipeline/SCMPipelineMetrics.java | 8 +
.../algorithms/RoundRobinPipelineChoosePolicy.java | 2 +-
.../hdds/scm/safemode/ContainerSafeModeRule.java | 359 -------------
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 10 +-
.../hdds/scm/safemode/ECContainerSafeModeRule.java | 249 +++++++++
.../scm/safemode/HealthyPipelineSafeModeRule.java | 2 +-
.../scm/safemode/RatisContainerSafeModeRule.java | 201 ++++++++
.../hdds/scm/safemode/SCMSafeModeManager.java | 32 +-
.../hadoop/hdds/scm/safemode/SafeModeMetrics.java | 3 +-
.../hdds/scm/safemode/SafeModeRuleFactory.java | 30 +-
.../scm/security/RootCARotationHandlerImpl.java | 2 +-
.../hdds/scm/security/RootCARotationManager.java | 2 +-
.../hdds/scm/security/SecretKeyManagerService.java | 2 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 5 +
.../hdds/scm/server/SCMClientProtocolServer.java | 562 ++++++++++++++-------
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 2 +-
.../hdds/scm/server/StorageContainerManager.java | 8 +-
.../upgrade/FinalizationStateManagerImpl.java | 4 +-
.../ScmOnFinalizeActionForDatanodeSchemaV2.java | 2 +-
.../org/apache/hadoop/ozone/audit/SCMAction.java | 16 +-
.../commands/RetriableDatanodeEventWatcher.java | 2 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 21 +-
.../hadoop/hdds/scm/TestHddsServerUtils.java | 2 +-
.../hdds/scm/TestSCMCommonPlacementPolicy.java | 4 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 2 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 8 +-
.../command/TestCommandStatusReportHandler.java | 3 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 12 +-
.../scm/container/TestContainerManagerImpl.java | 2 +-
.../container/balancer/TestContainerBalancer.java | 6 +-
.../TestContainerBalancerDatanodeNodeLimit.java | 2 +-
.../balancer/TestContainerBalancerTask.java | 2 +-
.../algorithms/TestContainerPlacementFactory.java | 16 +-
.../TestSCMContainerPlacementCapacity.java | 16 +-
.../TestSCMContainerPlacementRackAware.java | 34 +-
.../TestSCMContainerPlacementRackScatter.java | 36 +-
.../TestSCMContainerPlacementRandom.java | 18 +-
.../TestRatisOverReplicationHandler.java | 2 +-
.../replication/TestReplicationManager.java | 60 +++
.../TestReplicationManagerEventHandler.java | 72 +++
.../hadoop/hdds/scm/ha/TestSCMHAMetrics.java | 4 +-
.../hdds/scm/node/TestContainerPlacement.java | 4 +-
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 16 +-
.../hdds/scm/node/TestDiskBalancerManager.java | 11 +-
.../hdds/scm/node/TestNodeReportHandler.java | 4 +-
.../hadoop/hdds/scm/node/TestNodeStateManager.java | 2 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 82 ++-
.../hadoop/hdds/scm/node/TestSCMNodeMetrics.java | 6 +-
.../hdds/scm/node/TestSCMNodeStorageStatMap.java | 5 +-
.../hadoop/hdds/scm/node/TestStatisticsUpdate.java | 4 +-
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 23 +-
.../scm/pipeline/TestPipelinePlacementFactory.java | 4 +-
.../scm/safemode/TestDataNodeSafeModeRule.java | 172 +++++++
.../safemode/TestHealthyPipelineSafeModeRule.java | 19 +-
.../TestOneReplicaPipelineSafeModeRule.java | 14 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 175 +++----
.../hdds/scm/safemode/TestSafeModeRuleFactory.java | 6 +-
.../scm/security/TestRootCARotationManager.java | 16 +-
.../ozone/container/common/TestEndPoint.java | 46 +-
hadoop-hdds/test-utils/pom.xml | 5 -
.../org/apache/ozone/test/GenericTestUtils.java | 46 +-
.../org/apache/ozone/test/LambdaTestUtils.java | 5 +-
.../java/org/apache/ozone/test/tag/Native.java | 37 --
hadoop-hdds/tools/pom.xml | 12 -
.../org/apache/hadoop/hdds/cli/ItemsFromStdin.java | 67 +++
.../hdds/scm/cli/SafeModeWaitSubcommand.java | 5 +-
.../hdds/scm/cli/container/ContainerCommands.java | 2 +-
...ackage-info.java => ContainerIDParameters.java} | 19 +-
.../hdds/scm/cli/container/InfoSubcommand.java | 34 +-
.../hdds/scm/cli/container/UpgradeSubcommand.java | 162 +-----
.../scm/cli/container/upgrade/UpgradeChecker.java | 109 ----
.../scm/cli/container/upgrade/UpgradeManager.java | 172 -------
.../scm/cli/container/upgrade/UpgradeTask.java | 474 -----------------
.../hdds/scm/cli/datanode/HostNameParameters.java | 27 +-
.../cli/container/upgrade/TestUpgradeManager.java | 305 -----------
.../java/org/apache/hadoop/ozone/shell/Shell.java | 22 +-
.../shell/snapshot/ListSnapshotDiffHandler.java | 28 +-
.../apache/hadoop/ozone/client/ObjectStore.java | 76 ++-
.../hadoop/ozone/client/OzoneClientFactory.java | 5 -
.../client/io/BlockDataStreamOutputEntryPool.java | 2 +-
.../ozone/client/io/BlockOutputStreamEntry.java | 2 +-
.../client/io/BlockOutputStreamEntryPool.java | 2 +-
.../hadoop/ozone/client/io/ECKeyOutputStream.java | 2 +-
.../ozone/client/io/KeyDataStreamOutput.java | 2 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 2 +-
.../ozone/client/io/KeyOutputStreamSemaphore.java | 2 +-
.../ozone/client/io/OzoneCryptoInputStream.java | 2 +-
.../ozone/client/protocol/ClientProtocol.java | 18 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 21 +-
.../hadoop/ozone/client/MockDatanodeStorage.java | 2 +-
.../TestBlockOutputStreamIncrementalPutBlock.java | 2 +-
.../TestReplicatedBlockChecksumComputer.java | 4 +-
.../ozone/client/io/TestKeyOutputStream.java | 2 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 2 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 18 +
.../ozone/om/ha/GrpcOMFailoverProxyProvider.java | 2 +-
.../om/ha/HadoopRpcOMFailoverProxyProvider.java | 2 +-
.../ozone/om/ha/OMFailoverProxyProviderBase.java | 2 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 1 +
.../ozone/om/protocol/OzoneManagerProtocol.java | 17 +-
.../ozone/om/protocolPB/GrpcOmTransport.java | 2 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 26 +-
...ponse.java => ListSnapshotDiffJobResponse.java} | 30 +-
.../apache/hadoop/ozone/web/utils/OzoneUtils.java | 4 +-
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 9 +
.../ozone/security/TestGDPRSymmetricKey.java | 4 +-
.../security/TestOzoneDelegationTokenSelector.java | 4 +-
hadoop-ozone/dev-support/checks/native.sh | 23 -
...stKey Metrics.json => Ozone - OM Snapshot.json} | 242 ++-------
.../dist/src/main/compose/common/hadoop-test.sh | 3 +-
.../dist/src/main/compose/common/s3-haproxy.yaml | 6 -
.../dist/src/main/compose/compatibility/test.sh | 2 +
.../src/main/compose/ozone-balancer/test-ec.sh | 2 +
.../src/main/compose/ozone-balancer/test-ratis.sh | 2 +
.../dist/src/main/compose/ozone-csi/test.sh | 2 +
.../dist/src/main/compose/ozone-ha/test-hadoop.sh | 2 +
.../test-haproxy-s3g.sh} | 12 +-
.../dist/src/main/compose/ozone-ha/test.sh | 2 +
.../dist/src/main/compose/ozone-om-prepare/test.sh | 2 +
.../dist/src/main/compose/ozone-topology/test.sh | 2 +
.../dist/src/main/compose/ozone/test-ec.sh | 2 +
.../dist/src/main/compose/ozone/test-failures1.sh | 2 +
.../dist/src/main/compose/ozone/test-failures2.sh | 2 +
.../dist/src/main/compose/ozone/test-hadoop.sh | 2 +
.../src/main/compose/ozone/test-legacy-bucket.sh | 2 +
.../dist/src/main/compose/ozone/test-s3a.sh | 2 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
.../dist/src/main/compose/ozonescripts/test.sh | 2 +
.../compose/ozonesecure-ha/docker-compose.yaml | 125 +----
.../{common => ozonesecure-ha}/s3-haproxy.cfg | 12 +-
.../s3-haproxy.yaml} | 53 +-
.../ozonesecure-ha/s3g-port-forward.yml} | 5 +-
.../compose/ozonesecure-ha/s3g-virtual-host.yaml | 2 +-
.../compose/ozonesecure-ha/scm-decommission.yaml | 6 +-
.../test-haproxy-s3g.sh | 21 +-
.../main/compose/ozonesecure-ha/test-leadership.sh | 2 +
.../compose/ozonesecure-ha/test-om-bootstrap.sh | 2 +
.../src/main/compose/ozonesecure-ha/test-s3a.sh | 3 +
.../ozonesecure-ha/test-s3g-virtual-host.sh | 2 +
.../ozonesecure-ha/test-scm-decommission.sh | 2 +
.../dist/src/main/compose/ozonesecure-ha/test.sh | 4 +-
.../dist/src/main/compose/ozonesecure-mr/test.sh | 2 +
.../ozonesecure/test-certificate-rotation.sh | 2 +
.../dist/src/main/compose/ozonesecure/test-ec.sh | 2 +
.../dist/src/main/compose/ozonesecure/test-fcq.sh | 2 +
.../compose/ozonesecure/test-root-ca-rotation.sh | 2 +
.../src/main/compose/ozonesecure/test-vault.sh | 2 +
.../dist/src/main/compose/ozonesecure/test.sh | 4 +-
hadoop-ozone/dist/src/main/compose/restart/test.sh | 2 +
hadoop-ozone/dist/src/main/compose/test-all.sh | 3 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 42 +-
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 1 +
hadoop-ozone/dist/src/main/compose/xcompat/lib.sh | 6 +-
.../dist/src/main/compose/xcompat/test-new.sh | 2 +
.../dist/src/main/compose/xcompat/test-old.sh | 2 +
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 11 +-
hadoop-ozone/dist/src/main/license/bin/NOTICE.txt | 57 ---
hadoop-ozone/dist/src/main/license/jar-report.txt | 10 +-
.../src/main/smoketest/admincli/container.robot | 1 -
.../src/main/smoketest/snapshot/snapshot-sh.robot | 15 +-
hadoop-ozone/dist/src/shell/conf/log4j.properties | 19 +
hadoop-ozone/dist/src/shell/ozone/ozone | 5 +
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 6 +-
.../hadoop/ozone/MiniOzoneLoadGenerator.java | 2 +-
.../hadoop/ozone/TestMiniChaosOzoneCluster.java | 2 +-
.../hadoop/ozone/failure/FailureManager.java | 4 +-
.../ozone/loadgenerators/AgedDirLoadGenerator.java | 2 +-
.../ozone/loadgenerators/AgedLoadGenerator.java | 4 +-
.../hadoop/ozone/loadgenerators/DataBuffer.java | 2 +-
.../loadgenerators/FilesystemLoadGenerator.java | 2 +-
.../hadoop/ozone/loadgenerators/LoadBucket.java | 2 +-
.../hadoop/ozone/loadgenerators/LoadExecutors.java | 2 +-
.../loadgenerators/NestedDirLoadGenerator.java | 2 +-
.../loadgenerators/RandomDirLoadGenerator.java | 2 +-
.../ozone/loadgenerators/RandomLoadGenerator.java | 2 +-
.../loadgenerators/ReadOnlyLoadGenerator.java | 2 +-
.../service/hadoop/FileSystemAccessService.java | 5 +-
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 8 +-
.../AbstractContractMultipartUploaderTest.java | 2 +-
.../fs/contract/AbstractContractSeekTest.java | 4 +-
.../hadoop/fs/contract/ContractTestUtils.java | 5 +-
.../fs/ozone/AbstractOzoneFileSystemTest.java | 32 +-
.../ozone/AbstractOzoneFileSystemTestWithFSO.java | 4 +-
.../ozone/AbstractRootedOzoneFileSystemTest.java | 77 ++-
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 2 +-
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 7 +-
.../apache/hadoop/fs/ozone/TestHSyncUpgrade.java | 12 +-
.../apache/hadoop/fs/ozone/TestLeaseRecovery.java | 11 +-
.../hadoop/fs/ozone/TestOzoneFSInputStream.java | 8 +-
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 4 +-
.../hadoop/fs/ozone/TestOzoneFileChecksum.java | 4 +-
.../fs/ozone/TestOzoneFileSystemMetrics.java | 6 +-
.../fs/ozone/TestOzoneFileSystemPrefixParser.java | 8 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 6 +-
.../apache/hadoop/hdds/scm/TestCommitInRatis.java | 5 +-
.../hadoop/hdds/scm/TestContainerOperations.java | 2 +-
.../hdds/scm/TestContainerReportWithKeys.java | 8 +-
.../hadoop/hdds/scm/TestFailoverWithSCMHA.java | 11 +-
.../scm/TestGetCommittedBlockLengthAndPutKey.java | 4 +-
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 3 +-
.../hdds/scm/TestSCMInstallSnapshotWithHA.java | 11 +-
.../org/apache/hadoop/hdds/scm/TestSCMMXBean.java | 2 +-
.../hadoop/hdds/scm/TestSCMNodeManagerMXBean.java | 2 +-
.../hdds/scm/TestStorageContainerManager.java | 17 +-
...estStorageContainerManagerHAWithAllRunning.java | 7 +-
.../apache/hadoop/hdds/scm/TestWatchForCommit.java | 14 +-
.../hadoop/hdds/scm/TestXceiverClientGrpc.java | 6 +-
.../TestContainerStateManagerIntegration.java | 2 +-
.../metrics/TestSCMContainerManagerMetrics.java | 2 +-
.../TestReplicationManagerIntegration.java | 263 ++++++++++
.../hdds/scm/pipeline/TestLeaderChoosePolicy.java | 2 +-
.../TestSCMPipelineBytesWrittenMetrics.java | 5 +-
.../safemode/TestSCMSafeModeWithPipelineRules.java | 3 -
.../hdds/scm/storage/TestContainerCommandsEC.java | 6 +-
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 2 +-
.../hadoop/hdds/upgrade/TestScmHAFinalization.java | 5 +-
.../db/managed/TestRocksObjectLeakDetector.java | 2 +-
.../org/apache/hadoop/ozone/TestBlockTokens.java | 4 +-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 16 +-
.../apache/hadoop/ozone/TestDelegationToken.java | 13 +-
.../ozone/TestGetClusterTreeInformation.java | 2 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 3 +-
.../hadoop/ozone/TestMultipartObjectGet.java | 4 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 2 -
.../hadoop/ozone/TestSecureOzoneCluster.java | 34 +-
.../ozone/client/rpc/OzoneRpcClientTests.java | 304 +++++------
.../apache/hadoop/ozone/client/rpc/TestBCSID.java | 13 +-
.../ozone/client/rpc/TestBlockOutputStream.java | 12 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 16 +-
.../ozone/client/rpc/TestECKeyOutputStream.java | 3 +-
.../client/rpc/TestHybridPipelineOnDatanode.java | 21 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 18 +-
.../rpc/TestOzoneClientMultipartUploadWithFSO.java | 2 +-
.../rpc/TestOzoneRpcClientForAclAuditLog.java | 4 +-
.../TestOzoneRpcClientWithKeyLatestVersion.java | 14 +-
.../hadoop/ozone/client/rpc/TestReadRetries.java | 11 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 6 +-
.../ozone/container/TestContainerReplication.java | 30 +-
.../container/TestContainerReportHandling.java | 9 +-
.../TestContainerReportHandlingWithHA.java | 9 +-
.../commandhandler/TestBlockDeletion.java | 28 +-
.../commandhandler/TestDeleteContainerHandler.java | 14 +-
.../container/metrics/TestContainerMetrics.java | 7 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 15 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 6 +-
.../container/server/TestContainerServer.java | 8 +-
.../server/TestSecureContainerServer.java | 11 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 11 +-
.../TestDatanodeHddsVolumeFailureToleration.java | 7 +-
.../hadoop/ozone/freon/OmBucketTestUtils.java | 215 ++++++++
.../ozone/freon/TestHadoopNestedDirGenerator.java | 18 +-
.../ozone/freon/TestOmBucketReadWriteFileOps.java | 186 +------
.../ozone/freon/TestOmBucketReadWriteKeyOps.java | 65 +--
.../hadoop/ozone/om/TestAddRemoveOzoneManager.java | 17 +-
.../org/apache/hadoop/ozone/om/TestBucket.java | 13 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 34 +-
.../hadoop/ozone/om/TestListKeysWithFSO.java | 10 +-
.../hadoop/ozone/om/TestOMBucketLayoutUpgrade.java | 2 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 30 +-
.../hadoop/ozone/om/TestObjectStoreWithFSO.java | 6 +-
.../ozone/om/TestObjectStoreWithLegacyFS.java | 6 +-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 7 +-
.../hadoop/ozone/om/TestOmBlockVersioning.java | 18 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 28 +-
.../ozone/om/TestOzoneManagerHAWithAllRunning.java | 30 +-
.../om/TestOzoneManagerHAWithStoppedNodes.java | 16 +-
.../om/TestOzoneManagerListVolumesSecure.java | 4 +-
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 18 +-
.../hadoop/ozone/om/TestRecursiveAclWithFSO.java | 4 +-
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 10 +-
.../ozone/om/service/TestRangerBGSyncService.java | 6 +-
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 69 +--
.../ozone/om/snapshot/TestOmSnapshotDisabled.java | 6 +-
.../om/snapshot/TestOmSnapshotDisabledRestart.java | 6 +-
.../om/snapshot/TestOmSnapshotFileSystem.java | 6 +-
.../snapshot/TestOmSnapshotFsoWithNativeLib.java | 6 +-
...mSnapshotFsoWithNativeLibWithLinkedBuckets.java | 6 +-
.../om/snapshot/TestOzoneManagerHASnapshot.java | 22 +-
.../om/snapshot/TestOzoneManagerSnapshotAcl.java | 20 +-
.../snapshot/TestOzoneManagerSnapshotProvider.java | 8 +-
.../om/snapshot/TestOzoneSnapshotRestore.java | 4 +-
.../ozone/recon/TestReconAndAdminContainerCLI.java | 11 +-
.../hadoop/ozone/recon/TestReconAsPassiveScm.java | 8 +-
.../hadoop/ozone/recon/TestReconScmSnapshot.java | 6 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 2 +-
.../ozone/recon/TestReconWithOzoneManager.java | 15 +-
.../ozone/reconfig/TestOmReconfiguration.java | 6 +-
.../ozone/reconfig/TestScmReconfiguration.java | 6 +-
.../hadoop/ozone/repair/om/TestFSORepairTool.java | 2 +-
.../ozone/shell/TestDeletedBlocksTxnShell.java | 4 +-
.../shell/TestOzoneContainerUpgradeShell.java | 76 +--
.../hadoop/ozone/shell/TestOzoneTenantShell.java | 10 +-
.../tools/contract/AbstractContractDistCpTest.java | 2 +-
.../src/main/proto/OmClientProtocol.proto | 5 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 31 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 1 +
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 4 +-
.../hadoop/ozone/UniformDatanodesFactory.java | 9 +-
hadoop-ozone/ozone-manager/pom.xml | 34 +-
.../org/apache/hadoop/ozone/om/KeyManager.java | 7 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 46 ++
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 31 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 18 +-
.../hadoop/ozone/om/SstFilteringService.java | 3 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 2 +-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 2 +-
.../ozone/om/multitenant/AuthorizerLockImpl.java | 2 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 4 +-
.../om/ratis_snapshot/OmRatisSnapshotProvider.java | 7 +-
.../om/request/bucket/OMBucketSetOwnerRequest.java | 2 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 2 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 2 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 4 +-
.../om/request/key/OmKeysDeleteRequestWithFSO.java | 2 +-
.../tenant/OMSetRangerServiceVersionRequest.java | 4 -
.../s3/tenant/OMTenantAssignAdminRequest.java | 2 +-
.../tenant/OMTenantAssignUserAccessIdRequest.java | 2 +-
.../request/s3/tenant/OMTenantCreateRequest.java | 2 +-
.../request/s3/tenant/OMTenantDeleteRequest.java | 2 +-
.../s3/tenant/OMTenantRevokeAdminRequest.java | 2 +-
.../tenant/OMTenantRevokeUserAccessIdRequest.java | 2 +-
.../ozone/om/request/upgrade/OMPrepareRequest.java | 5 +-
.../ozone/om/request/util/OmKeyHSyncUtil.java | 2 +-
.../om/request/volume/OMVolumeCreateRequest.java | 6 -
.../response/file/OMDirectoryCreateResponse.java | 2 +-
.../file/OMDirectoryCreateResponseWithFSO.java | 2 +-
.../ozone/om/response/key/OMKeyCreateResponse.java | 2 +-
.../hadoop/ozone/om/service/CompactionService.java | 184 +++++++
.../ozone/om/service/DirectoryDeletingService.java | 2 +-
.../ozone/om/service/OMRangerBGSyncService.java | 2 +-
.../hadoop/ozone/om/service/QuotaRepairTask.java | 5 +-
.../ozone/om/snapshot/SnapshotDiffManager.java | 52 +-
.../protocolPB/OzoneManagerRequestHandler.java | 34 +-
.../hadoop/ozone/security/OMCertificateClient.java | 2 +-
.../OzoneDelegationTokenSecretManager.java | 2 +-
.../hadoop/ozone/om/TestAuthorizerLockImpl.java | 2 +-
.../apache/hadoop/ozone/om/TestChunkStreams.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 214 ++++++++
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 4 +-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 9 +-
.../org/apache/hadoop/ozone/om/TestScmClient.java | 6 +-
.../hadoop/ozone/om/failover/TestOMFailovers.java | 6 +-
.../apache/hadoop/ozone/om/ha/TestOMHAMetrics.java | 4 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 6 +-
.../ozone/om/request/OMRequestTestUtils.java | 2 +-
.../bucket/TestOMBucketSetPropertyRequest.java | 8 +-
.../request/file/TestOMDirectoryCreateRequest.java | 4 +-
.../file/TestOMDirectoryCreateRequestWithFSO.java | 4 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 6 +-
.../volume/TestOMVolumeSetQuotaRequest.java | 8 +-
.../ozone/om/service/TestCompactionService.java | 178 +++++++
.../ozone/om/service/TestKeyDeletingService.java | 2 +-
.../service/TestMultipartUploadCleanupService.java | 6 +-
.../om/service/TestOpenKeyCleanupService.java | 10 +-
.../om/service/TestSnapshotDiffCleanupService.java | 10 +-
.../ozone/om/snapshot/TestSnapshotCache.java | 2 +-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 79 +--
.../ozone/om/snapshot/TestSstFilteringService.java | 2 +-
.../TestOzoneDelegationTokenSecretManager.java | 6 +-
.../ozone/security/TestOzoneTokenIdentifier.java | 8 +-
.../security/acl/TestOzoneNativeAuthorizer.java | 4 +-
.../hadoop/ozone/security/acl/TestParentAcl.java | 4 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 2 +-
.../hadoop/fs/ozone/TestOzoneFSInputStream.java | 2 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 5 +
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 5 +-
.../ozone/recon/fsck/ContainerHealthTask.java | 10 -
.../ReconIncrementalContainerReportHandler.java | 52 +-
.../ozone/recon/scm/ReconNewNodeHandler.java | 3 +-
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 2 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 4 -
.../recon/tasks/ContainerKeyMapperHelper.java | 5 +-
.../ozone/recon/tasks/FileSizeCountTaskHelper.java | 9 +-
.../ozone/recon/tasks/OmTableInsightTask.java | 5 +-
.../upgrade/ReconTaskStatusTableUpgradeAction.java | 2 +-
.../webapps/recon/ozone-recon-web/package.json | 2 +-
.../webapps/recon/ozone-recon-web/pnpm-lock.yaml | 154 +++---
.../ozone-recon-web/src/views/insights/om/om.tsx | 7 +
.../views/missingContainers/missingContainers.tsx | 7 +
.../apache/hadoop/ozone/recon/TestReconUtils.java | 4 +-
.../hadoop/ozone/s3/commontypes/package-info.java | 8 +-
.../endpoint/CompleteMultipartUploadRequest.java | 3 +-
.../endpoint/CompleteMultipartUploadResponse.java | 3 +-
.../ozone/s3/endpoint/CopyObjectResponse.java | 3 +-
.../hadoop/ozone/s3/endpoint/CopyPartResult.java | 3 +-
.../ozone/s3/endpoint/ListBucketResponse.java | 3 +-
.../s3/endpoint/ListMultipartUploadsResult.java | 3 +-
.../ozone/s3/endpoint/ListObjectResponse.java | 4 +-
.../ozone/s3/endpoint/ListPartsResponse.java | 4 +-
.../ozone/s3/endpoint/MultiDeleteRequest.java | 7 +-
.../ozone/s3/endpoint/MultiDeleteResponse.java | 10 +-
.../endpoint/MultipartUploadInitiateResponse.java | 3 +-
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 3 +-
.../apache/hadoop/ozone/s3/endpoint/S3Tagging.java | 3 +-
.../hadoop/ozone/s3/endpoint/package-info.java | 8 +-
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 3 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 11 +-
.../hadoop/ozone/s3/TestS3GatewayAuditLog.java | 2 +-
...CompleteMultipartUploadRequestUnmarshaller.java | 4 +-
.../TestMultiDeleteRequestUnmarshaller.java | 4 +-
.../hadoop/ozone/s3/endpoint/TestObjectHead.java | 2 +-
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 2 +-
.../ozone/s3/endpoint/TestObjectTaggingPut.java | 13 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 3 +-
.../ozone/s3/metrics/TestS3GatewayMetrics.java | 3 +-
hadoop-ozone/tools/pom.xml | 18 +
.../org/apache/hadoop/ozone/admin/OzoneAdmin.java | 1 -
.../hadoop/ozone/admin/om/PrepareSubCommand.java | 5 +-
.../parser/ContainerDatanodeDatabase.java | 237 +++++++++
.../parser/ContainerLogFileParser.java | 221 ++++++++
.../parser/DBConsts.java} | 18 +-
.../containerlog/parser/DatanodeContainerInfo.java | 137 +++++
.../ozone/containerlog/parser}/package-info.java | 4 +-
.../ContainerLogController.java} | 16 +-
.../ozone/debug/container/ContainerLogParser.java | 93 ++++
.../ozone/debug/container}/package-info.java | 5 +-
.../datanode/container/ContainerCommands.java | 4 +
.../apache/hadoop/ozone/debug/ldb/DBScanner.java | 4 +-
.../apache/hadoop/ozone/debug/ldb/ValueSchema.java | 2 +-
.../ozone/freon/AbstractOmBucketReadWriteOps.java | 2 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 9 +-
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 4 +-
.../ozone/freon/ClosedContainerReplicator.java | 6 +-
.../hadoop/ozone/freon/ContentGenerator.java | 2 +-
.../hadoop/ozone/freon/DatanodeBlockPutter.java | 2 +-
.../hadoop/ozone/freon/DatanodeChunkGenerator.java | 2 +-
.../ozone/freon/DatanodeSimulationState.java | 2 +-
.../hadoop/ozone/freon/DatanodeSimulator.java | 10 +-
.../freon/FollowerAppendLogEntryGenerator.java | 4 +-
.../java/org/apache/hadoop/ozone/freon/Freon.java | 2 +-
.../hadoop/ozone/freon/HadoopDirTreeGenerator.java | 4 +-
.../ozone/freon/HadoopNestedDirGenerator.java | 4 +-
.../ozone/freon/LeaderAppendLogEntryGenerator.java | 4 +-
.../freon/OzoneClientKeyReadWriteListOps.java | 2 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 8 +-
.../apache/hadoop/ozone/freon/S3KeyGenerator.java | 2 +-
.../hadoop/ozone/freon/SCMThroughputBenchmark.java | 2 +-
.../freon/containergenerator/GeneratorScm.java | 2 +-
.../org/apache/hadoop/ozone/repair/RepairTool.java | 15 +
.../DatanodeRepair.java} | 13 +-
.../ozone/repair/datanode}/package-info.java | 4 +-
.../schemaupgrade/ContainerUpgradeResult.java | 124 +++++
.../schemaupgrade/UpgradeContainerSchema.java | 506 +++++++++++++++++++
.../datanode/schemaupgrade}/UpgradeUtils.java | 67 ++-
.../schemaupgrade/VolumeUpgradeResult.java | 119 +++++
.../datanode/schemaupgrade}/package-info.java | 2 +-
.../hadoop/ozone/repair/om/FSORepairTool.java | 2 +-
.../apache/hadoop/ozone/utils/AutoCompletion.java | 175 +++++++
.../resources/container-log-db-queries.properties | 24 +
.../apache/hadoop/ozone/debug/TestCheckNative.java | 8 +-
.../ozone/debug/audit/parser/TestAuditParser.java | 2 +-
.../schemaupgrade/TestUpgradeContainerSchema.java | 378 ++++++++++++++
pom.xml | 140 ++---
742 files changed, 9087 insertions(+), 6896 deletions(-)
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLog.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/http/HttpRequestLogAppender.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/DumpFileWriter.java
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLog.java
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/TestHttpRequestLogAppender.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/MapBackedTableIterator.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/db/TestTypedTable.java
rename
hadoop-hdds/{framework/src/main/java/org/apache/hadoop/hdds/utils/db/DumpFileLoader.java
=>
managed-rocksdb/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabaseException.java}
(58%)
copy hadoop-hdds/{framework/src/test =>
managed-rocksdb/src/main}/java/org/apache/hadoop/hdds/utils/db/package-info.java
(95%)
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/{DatanodeCommandCountUpdatedHandler.java
=> ReplicationManagerEventHandler.java} (59%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/states/ContainerEntry.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/DatanodeEntry.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ContainerSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/ECContainerSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/safemode/RatisContainerSafeModeRule.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManagerEventHandler.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/safemode/TestDataNodeSafeModeRule.java
delete mode 100644
hadoop-hdds/test-utils/src/test/java/org/apache/ozone/test/tag/Native.java
create mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/cli/ItemsFromStdin.java
copy
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/{package-info.java
=> ContainerIDParameters.java} (64%)
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/upgrade/UpgradeChecker.java
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/upgrade/UpgradeManager.java
delete mode 100644
hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/upgrade/UpgradeTask.java
delete mode 100644
hadoop-hdds/tools/src/test/java/org/apache/hadoop/hdds/scm/cli/container/upgrade/TestUpgradeManager.java
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/{ListSnapshotResponse.java
=> ListSnapshotDiffJobResponse.java} (54%)
delete mode 100755 hadoop-ozone/dev-support/checks/native.sh
copy hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/{Ozone -
ListKey Metrics.json => Ozone - OM Snapshot.json} (66%)
rename hadoop-ozone/dist/src/main/compose/{ozone/disabled-test-s3-haproxy.sh
=> ozone-ha/test-haproxy-s3g.sh} (75%)
copy hadoop-ozone/dist/src/main/compose/{common =>
ozonesecure-ha}/s3-haproxy.cfg (81%)
rename hadoop-ozone/dist/src/main/compose/{common/s3-haproxy-secure.yaml =>
ozonesecure-ha/s3-haproxy.yaml} (67%)
copy hadoop-ozone/dist/src/main/{smoketest/.env =>
compose/ozonesecure-ha/s3g-port-forward.yml} (94%)
rename hadoop-ozone/dist/src/main/compose/{ozonesecure =>
ozonesecure-ha}/test-haproxy-s3g.sh (84%)
mode change 100755 => 100644
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/container/replication/TestReplicationManagerIntegration.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/OmBucketTestUtils.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/CompactionService.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestCompactionService.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/containerlog/parser/ContainerDatanodeDatabase.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/containerlog/parser/ContainerLogFileParser.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/{debug/audit/parser/common/ParserConsts.java
=> containerlog/parser/DBConsts.java} (62%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/containerlog/parser/DatanodeContainerInfo.java
copy {hadoop-hdds/erasurecode/src/test/java/org/apache/ozone/erasurecode =>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/containerlog/parser}/package-info.java
(88%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/{om/OMDebug.java
=> container/ContainerLogController.java} (76%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/container/ContainerLogParser.java
copy
{hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/security/x509/certificate/client
=>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/container}/package-info.java
(82%)
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/{ldb/LDBRepair.java
=> datanode/DatanodeRepair.java} (74%)
copy {hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/fs =>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode}/package-info.java
(91%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/ContainerUpgradeResult.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/UpgradeContainerSchema.java
rename
{hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/upgrade
=>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade}/UpgradeUtils.java
(51%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/VolumeUpgradeResult.java
rename
{hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/container/upgrade
=>
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade}/package-info.java
(93%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/utils/AutoCompletion.java
create mode 100644
hadoop-ozone/tools/src/main/resources/container-log-db-queries.properties
create mode 100644
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/repair/datanode/schemaupgrade/TestUpgradeContainerSchema.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
