This is an automated email from the ASF dual-hosted git repository.
aswinshakil pushed a change to branch HDDS-10239-container-reconciliation
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 0a53c73de2 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 2072ef09f6 HDDS-13214. populate-cache fails due to unused dependency
(#8594)
add 203c1d35f0 HDDS-13206. Update documentation for Apache Ranger (#8583)
add b46e6b2686 HDDS-13150. Fixed SnapshotLimitCheck when failures occur.
(#8532)
add f62eb8a466 HDDS-13160. Remove SnapshotDirectoryCleaningService and
refactor AbstractDeletingService (#8547)
add 3c9d4d875f HDDS-13103. Correct transaction metrics in
SCMBlockDeletingService. (#8516)
add e7f554497b HDDS-13171. Replace pipelineID if nodes are changed (#8562)
add ba950741b3 HDDS-13107. Support limiting output of `ozone admin
datanode list` (#8595)
add 5787135483 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM
upgrade docs (#8598)
add 516bc9659b HDDS-13148. [Docs] Update Transparent Data Encryption doc.
(#8530)
add 06283866b4 HDDS-13142. Correct SCMPerformanceMetrics for delete
operation. (#8592)
add bbaf71e71e HDDS-13130. Rename all instances of Disk Usage to Namespace
usage (#8571)
add ba1887ca9a HDDS-13241. Fix some potential resource leaks (#8602)
add 8c32363072 HDDS-13050. Update StartFromDockerHub.md. (#8586)
add 752da2be72 HDDS-12760. Intermittent Timeout in
testImportedContainerIsClosed (#8349)
add 6bf009c202 HDDS-12927. metrics and log to indicate datanode crossing
disk limits (#8573)
add 3b5985c29c HDDS-13244. Disable
TestSnapshotDeletingServiceIntegrationTest
add 931bc2d8a9 HDDS-13243. copy-rename-maven-plugin version is missing
(#8605)
add 09d3b2757d HDDS-13244. TestSnapshotDeletingServiceIntegrationTest
should close snapshots after deleting them (#8611)
add 6ff3ad6624 HDDS-12873. Improve ContainerData statistics
synchronization. (#8305)
add bf29f7ffb7 HDDS-13235. The equals/hashCode methods in anonymous
KeyValue classes may not work. (#8607)
add 4a461b2418 HDDS-10490. Intermittent NPE in
TestSnapshotDiffManager#testLoadJobsOnStartUp (#8596)
add 6d4d423814 HDDS-13181. Added metrics for internal Snapshot Operations.
(#8606)
add 1e472174f7 HDDS-13168. Fix error response format in
CheckUploadContentTypeFilter (#8614)
add 5d6fe94891 HDDS-13216. Standardize Container[Replica]NotFoundException
messages (#8599)
add ca1afe8519 HDDS-13257. Remove separate split for shell integration
tests (#8616)
add 0701d6a20a HDDS-13248. Remove `ozone debug replicas verify` option
--output-dir (#8612)
add f31c264e38 HDDS-13045. Implement Immediate Triggering of Heartbeat
when Volume Full (#8590)
add 38a9186d61 HDDS-12637. Increase max buffer size for tar entry
read/write (#8618)
add ad5a507dfa HDDS-13035. SnapshotDeletingService should hold write locks
while purging deleted snapshots (#8554)
add 3ecf3450b3 HDDS-13207. [Docs] Third party systems compatible with
Ozone S3. (#8584)
add fd1330072f HDDS-10775. Support bucket ownership verification (#8558)
add 7e770586bd HDDS-13112. [Docs] OM Bootstrap can also happen when
follower falls behind too much. (#8600)
add a9271131c7 HDDS-13271. [Docs] Minor text updates, reference links.
(#8624)
add 8983a63374 HDDS-13272. Bump junit to 5.13.1 (#8625)
add 3f3ec43ec0 HDDS-13266. `ozone debug checknative` to show OpenSSL lib
(#8623)
add d38484ef31 HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (#8627)
add 36fe8880fb HDDS-13275. Improve CheckNative implementation (#8628)
add ce288b6ed0 HDDS-13265. Simplify the page Access Ozone using HTTPFS
REST API (#8629)
add 77138b884a HDDS-13254. Change table iterator to optionally read key or
value. (#8621)
add 5a3e4e79c3 HDDS-13273. Bump awssdk to 2.31.63 (#8626)
add 7af8c44009 HDDS-11454. Ranger integration for Docker Compose
environment (#8575)
add 3370f42015 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in
hadoop-hdds (#8631)
add ac511ac4ea HDDS-13259. Deletion Progress - Grafana Dashboard (#8617)
add 9bc53b21eb HDDS-11513. All deletion configurations should be
configurable without restart (#8003)
add 90ed60b7c4 HDDS-13279. Skip verifying Apache Ranger binaries in CI
(#8633)
add 052d912444 HDDS-13037. Let container create command support STANDALONE
, RATIS and EC containers (#8559)
add 48aefeaad0 HDDS-13277. [Docs] Native C/C++ Ozone clients (#8630)
add 9958f5bff0 HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to
CVE-2025-48734 (#8646)
add a7fc290c20 HDDS-13236. Change Table methods not to throw IOException.
(#8645)
add 6df3077fe1 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (#8503)
add d9049a2aea HDDS-13220. Change Recon 'Negative usedBytes' message
loglevel to DEBUG (#8648)
add 9d2b4158e7 HDDS-13234. Expired secret key can abort leader OM startup.
(#8601)
add 97262aa6d6 HDDS-13256. Updated OM Snapshot Grafana Dashboard to
reflect metric updates from HDDS-13181. (#8639)
add de683aad88 HDDS-13263. Refactor DB Checkpoint Utilities. (#8620)
add b574709dd6 HDDS-12924. datanode used space calculation optimization
(#8365)
add f713e57b19 HDDS-12482. Avoid using CommonConfigurationKeys (#8647)
add e9c0a45888 HDDS-13262. Simplify key name validation (#8619)
add f3050cffff HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in
SCM/Datanode. (#8638)
add a0af93e210 HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>`
in test (#8657)
add 732985958d HDDS-13023. Container checksum is missing after container
import (#8459)
add 8c1b439d51 HDDS-13296. Integration check always passes due to missing
output (#8662)
add da53b5b4d6 HDDS-13299. Fix failures related to delete (#8665)
new 3c4b3bec36 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add d207c18428 HDDS-13301. Fix TestSchemaOneBackwardsCompatibility (#8666)
add af3a0f052a HDDS-13307. integration (flaky) fails with all tests
passing (#8667)
add e1365a79dc HDDS-13286. Fail stream write when the volume is full.
(#8644)
add db6ed84df3 HDDS-13294. Remove the Table.close() method. (#8658)
add c043be77cf HDDS-13312. Bump maven-patch-plugin to 1.3 (#8673)
add b86fad8b9d HDDS-13316. Bump commons-text to 1.13.1 (#8669)
add 774b3ee78a HDDS-13315. Bump log4j2 to 2.25.0 (#8670)
add 803fa3194e HDDS-13313. Bump maven-javadoc-plugin to 3.11.2 (#8672)
add 05567e6880 HDDS-13317. Table should support empty array/String (#8676)
new e60f22745c Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 2 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 8 +-
.../hdds/scm/client/TestHddsClientUtils.java | 1 +
.../apache/hadoop/hdds/scm/client/ScmClient.java | 3 +
.../hdds/scm/container/ContainerException.java | 9 -
.../scm/container/ContainerNotFoundException.java | 23 +-
.../ContainerReplicaNotFoundException.java | 18 +-
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 58 +-
.../protocol/StorageContainerLocationProtocol.java | 2 +
.../hadoop/hdds/utils/BackgroundService.java | 17 +-
.../apache/hadoop/hdds/utils/db/Proto3Codec.java | 22 +-
.../ozone/container/common/helpers/BlockData.java | 6 +-
.../common/src/main/resources/ozone-default.xml | 10 -
.../hadoop/hdds/scm/pipeline/TestPipeline.java | 44 +-
.../apache/hadoop/hdds/utils/db/CodecTestUtil.java | 29 +
.../ozone/HddsDatanodeClientProtocolServer.java | 4 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 21 +-
.../container/common/helpers/ContainerUtils.java | 15 +
.../common/impl/BlockDeletingService.java | 37 +
.../ozone/container/common/impl/ContainerData.java | 319 ++--
.../container/common/impl/HddsDispatcher.java | 36 +-
.../ContainerDeletionChoosingPolicyTemplate.java | 6 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 6 +-
.../transport/server/ratis/XceiverServerRatis.java | 2 +-
.../ozone/container/common/volume/HddsVolume.java | 71 +
.../container/common/volume/MutableVolumeSet.java | 11 +
.../container/common/volume/StorageVolume.java | 28 +-
.../container/common/volume/VolumeInfoMetrics.java | 29 +
.../ozone/container/common/volume/VolumeUsage.java | 11 +-
.../container/keyvalue/KeyValueContainer.java | 56 +-
.../container/keyvalue/KeyValueContainerData.java | 77 +-
.../KeyValueContainerMetadataInspector.java | 12 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 6 +-
.../container/keyvalue/TarContainerPacker.java | 15 +
.../keyvalue/helpers/KeyValueContainerUtil.java | 96 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 16 +-
.../keyvalue/impl/ChunkManagerDispatcher.java | 2 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 2 +-
.../keyvalue/impl/KeyValueStreamDataChannel.java | 1 +
.../keyvalue/impl/StreamDataChannelBase.java | 5 +
.../statemachine/background/BlockDeletingTask.java | 45 +-
.../container/metadata/AbstractDatanodeStore.java | 18 +-
.../ozone/container/metadata/DatanodeTable.java | 59 +-
.../metadata/SchemaOneChunkInfoListCodec.java | 10 +-
.../metadata/SchemaOneDeletedBlocksTable.java | 69 +-
.../container/ozoneimpl/ContainerController.java | 3 +-
.../ozone/container/ozoneimpl/OzoneContainer.java | 20 +-
.../container/common/TestBlockDeletingService.java | 12 +-
.../common/TestKeyValueContainerData.java | 32 +-
.../TestSchemaOneBackwardsCompatibility.java | 13 +-
.../common/impl/TestContainerPersistence.java | 20 +-
.../container/common/impl/TestHddsDispatcher.java | 64 +-
.../TestCloseContainerCommandHandler.java | 12 +-
.../container/common/volume/TestHddsVolume.java | 54 +
.../container/keyvalue/TestKeyValueContainer.java | 2 +
.../TestKeyValueContainerIntegrityChecks.java | 4 +-
.../container/keyvalue/TestTarContainerPacker.java | 4 +-
.../keyvalue/impl/TestFilePerBlockStrategy.java | 6 +-
.../impl/TestKeyValueStreamDataChannel.java | 31 +
.../container/ozoneimpl/TestOzoneContainer.java | 8 +-
.../replication/TestReplicationSupervisor.java | 3 +-
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 3 +-
hadoop-hdds/docs/content/_index.md | 5 +-
.../content/design/dn-usedspace-calculation.md | 91 ++
hadoop-hdds/docs/content/design/omprepare.md | 11 +-
.../docs/content/design/upgrade-dev-primer.md | 8 +-
hadoop-hdds/docs/content/feature/OM-HA.md | 30 +
hadoop-hdds/docs/content/interface/ReconApi.md | 10 +-
hadoop-hdds/docs/content/interface/ReconApi.zh.md | 4 +-
hadoop-hdds/docs/content/interface/S3.md | 10 +
hadoop-hdds/docs/content/interface/native-cpp.md | 138 ++
.../content/recipe/PythonRequestsOzoneHttpFS.md | 26 +-
hadoop-hdds/docs/content/security/SecuringTDE.md | 150 +-
.../docs/content/security/SecurityWithRanger.md | 4 +-
hadoop-hdds/docs/content/start/OnPrem.md | 7 +
.../docs/content/start/StartFromDockerHub.md | 124 +-
hadoop-hdds/docs/content/start/_index.md | 2 +-
hadoop-hdds/docs/content/start/ozone-recon.png | Bin 0 -> 408165 bytes
hadoop-hdds/docs/content/start/ozone-scm.png | Bin 0 -> 322764 bytes
.../static/swagger-resources/recon-api.yaml | 10 +-
.../main/java/org/apache/hadoop/hdds/fs/DU.java | 30 +-
.../org/apache/hadoop/hdds/fs/DUOptimized.java | 66 +
.../fs/{DUFactory.java => DUOptimizedFactory.java} | 52 +-
.../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 14 +-
.../hadoop/hdds/fs/SpaceUsageCheckParams.java | 9 +
...inerLocationProtocolClientSideTranslatorPB.java | 31 +-
.../hadoop/hdds/server/http/HttpServer2.java | 16 +-
.../org/apache/hadoop/hdds/utils/Archiver.java | 34 +-
.../hadoop/hdds/utils/DBCheckpointServlet.java | 50 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 8 +-
.../hadoop/hdds/utils/db/RDBBatchOperation.java | 24 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 7 +-
.../hdds/utils/db/RDBStoreAbstractIterator.java | 19 +-
.../hdds/utils/db/RDBStoreByteArrayIterator.java | 16 +-
.../hdds/utils/db/RDBStoreCodecBufferIterator.java | 18 +-
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 92 +-
.../apache/hadoop/hdds/utils/db/RawKeyValue.java | 85 -
.../org/apache/hadoop/hdds/utils/db/Table.java | 237 ++-
.../apache/hadoop/hdds/utils/db/TableIterator.java | 9 +-
.../apache/hadoop/hdds/utils/db/TypedTable.java | 210 +--
.../org/apache/hadoop/hdds/fs/TestDUOptimized.java | 71 +
...tDUFactory.java => TestDUOptimizedFactory.java} | 26 +-
.../hadoop/hdds/utils/MapBackedTableIterator.java | 10 +-
.../org/apache/hadoop/hdds/utils/TestArchiver.java | 36 +-
.../hadoop/hdds/utils/TestRDBSnapshotProvider.java | 26 +-
.../hadoop/hdds/utils/db/InMemoryTestTable.java | 19 +-
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 56 +-
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 158 +-
.../utils/db/TestRDBStoreByteArrayIterator.java | 33 +-
.../utils/db/TestRDBStoreCodecBufferIterator.java | 17 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 674 ++++----
.../hdds/utils/db/TestTypedRDBTableStore.java | 326 ++--
.../hadoop/hdds/utils/db/TestTypedTable.java | 166 +-
.../src/main/proto/ScmAdminProtocol.proto | 3 +-
hadoop-hdds/server-scm/pom.xml | 4 +
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 16 +-
.../scm/block/DeletedBlockLogStateManager.java | 5 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 31 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 9 +-
.../SCMDeletedBlockTransactionStatusManager.java | 4 +-
.../scm/block/ScmBlockDeletingServiceMetrics.java | 113 +-
.../hdds/scm/container/ContainerManager.java | 4 +-
.../hdds/scm/container/ContainerManagerImpl.java | 43 +-
.../hdds/scm/container/ContainerStateManager.java | 5 -
.../scm/container/ContainerStateManagerImpl.java | 17 +-
.../hdds/scm/container/balancer/MoveManager.java | 2 +-
.../algorithms/SCMContainerPlacementRackAware.java | 8 +
.../placement/metrics/SCMPerformanceMetrics.java | 26 +-
.../hdds/scm/ha/SCMDBCheckpointProvider.java | 5 +-
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 21 +-
.../hdds/scm/pipeline/PipelineStateManager.java | 2 +-
.../scm/pipeline/PipelineStateManagerImpl.java | 17 +-
.../hadoop/hdds/scm/pipeline/PipelineStateMap.java | 2 +-
...inerLocationProtocolServerSideTranslatorPB.java | 8 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 22 +-
.../hadoop/hdds/scm/server/SCMCertStore.java | 22 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 18 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 4 +-
.../hdds/scm/server/StorageContainerManager.java | 1 -
.../upgrade/FinalizationStateManagerImpl.java | 1 -
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 4 +-
.../scm/block/TestSCMBlockDeletingService.java | 2 +-
.../scm/container/TestContainerManagerImpl.java | 4 -
.../scm/container/TestContainerReportHandler.java | 1 -
.../scm/container/TestContainerStateManager.java | 1 -
.../TestIncrementalContainerReportHandler.java | 1 -
.../scm/container/TestUnknownContainerReport.java | 20 +-
.../TestContainerBalancerDatanodeNodeLimit.java | 2 +-
.../scm/container/balancer/TestMoveManager.java | 2 +-
.../TestSCMContainerPlacementRackAware.java | 12 +
.../hdds/scm/node/TestContainerPlacement.java | 1 -
.../scm/pipeline/TestPipelineStateManagerImpl.java | 2 +-
.../pipeline/TestWritableECContainerProvider.java | 2 +-
.../hdds/scm/cli/ContainerOperationClient.java | 13 +-
.../hdds/scm/cli/container/CreateSubcommand.java | 23 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 8 +
.../ozone/admin/nssummary/DiskUsageSubCommand.java | 2 +-
.../ozone/admin/om/lease/LeaseRecoverer.java | 14 +-
.../hdds/scm/cli/container/TestInfoSubCommand.java | 4 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 2 +
.../cli/pipeline/TestClosePipelinesSubCommand.java | 2 +-
.../cli/pipeline/TestListPipelinesSubCommand.java | 2 +-
.../ozone/shell/volume/DeleteVolumeHandler.java | 7 +-
.../client/checksum/ECFileChecksumHelper.java | 2 +-
.../checksum/ReplicatedFileChecksumHelper.java | 12 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 14 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 5 -
.../java/org/apache/hadoop/ozone/om/OmConfig.java | 20 +
.../hadoop/ozone/om/helpers/SnapshotDiffJob.java | 13 +-
.../org/apache/hadoop/ozone/om/TestOmConfig.java | 2 +
.../helpers/OldSnapshotDiffJobCodecForTesting.java | 56 +
.../om/helpers/TestOmSnapshotDiffJobCodec.java | 73 +
hadoop-ozone/dev-support/checks/_post_process.sh | 10 +-
hadoop-ozone/dev-support/checks/integration.sh | 3 +
hadoop-ozone/dev-support/checks/junit.sh | 3 +-
hadoop-ozone/dev-support/checks/rat.sh | 3 +-
.../dashboards/Ozone - DeleteKey Metrics.json | 6 +-
.../Ozone - DeleteKeyProgress Metrics.json | 1712 ++++++++++++++++++++
.../grafana/dashboards/Ozone - OM Snapshot.json | 295 +++-
.../compose/common/{hadoop.yaml => ranger.yaml} | 54 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 5 +
.../src/main/compose/ozonesecure-ha/ranger.yaml | 54 +
.../src/main/compose/ozonesecure-ha/test-ranger.sh | 61 +
hadoop-ozone/dist/src/main/compose/testlib.sh | 32 +-
.../main/smoketest/debug/ozone-debug-tests.robot | 2 +-
.../src/main/smoketest/debug/ozone-debug.robot | 2 +-
.../src/main/smoketest/recon/recon-nssummary.robot | 4 +-
.../hadoop/ozone/freon/DNRPCLoadGenerator.java | 8 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 26 +-
.../http/server/CheckUploadContentTypeFilter.java | 21 +-
.../TestReconInsightsForDeletedDirectories.java | 12 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 15 +-
.../dev-support/findbugsExcludeFile.xml | 4 +
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 616 +++++++
hadoop-ozone/integration-test/pom.xml | 5 +
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 14 +-
.../java/org/apache/hadoop/hdds/TestRemoteEx.java | 55 +
.../hadoop/hdds/scm/TestAllocateContainer.java | 23 +
.../hadoop/hdds/scm/TestContainerOperations.java | 20 +
.../hdds/scm/TestSCMDbCheckpointServlet.java | 17 +-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 5 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 8 +-
.../ozone/container/TestContainerReplication.java | 1 -
.../commandhandler/TestBlockDeletion.java | 8 +-
.../commandhandler/TestDeleteContainerHandler.java | 5 +-
.../TestRefreshVolumeUsageHandler.java | 7 +
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 19 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 3 +
.../ozone/om/TestObjectStoreWithLegacyFS.java | 3 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 3 +-
.../om/TestOzoneManagerHAWithStoppedNodes.java | 3 +-
.../om/TestOzoneManagerListVolumesSecure.java | 7 +
.../TestDirectoryDeletingServiceWithFSO.java | 16 +-
.../om/service}/TestRootedDDSWithFSO.java | 5 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 539 +++---
.../ozone/om/snapshot/TestOMDBCheckpointUtils.java | 100 ++
.../om/snapshot/TestOzoneManagerHASnapshot.java | 8 +-
.../snapshot/TestSnapshotBackgroundServices.java | 6 +-
.../TestSnapshotDirectoryCleaningService.java | 3 +-
.../reconfig/TestDatanodeReconfiguration.java | 18 +-
.../ozone/reconfig/TestOmReconfiguration.java | 2 +
.../hadoop/ozone/repair/om/TestFSORepairTool.java | 5 +-
.../hadoop/ozone/shell/TestOzoneDebugShell.java | 2 +-
.../hadoop/ozone/shell/TestReconfigShell.java | 2 +-
.../tools/contract/AbstractContractDistCpTest.java | 3 +-
hadoop-ozone/native-client/libo3fs/o3fs.c | 2 +-
hadoop-ozone/native-client/libo3fs/o3fs.h | 4 +-
.../apache/hadoop/ozone/common/BekInfoUtils.java | 4 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 2 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 74 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 36 -
.../hadoop/ozone/om/OmSnapshotInternalMetrics.java | 118 ++
.../org/apache/hadoop/ozone/om/OzoneManager.java | 20 +-
.../om/request/file/OMDirectoryCreateRequest.java | 6 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 13 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 9 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 10 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 8 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 75 -
.../request/snapshot/OMSnapshotCreateRequest.java | 3 +-
.../snapshot/OMSnapshotMoveTableKeysRequest.java | 5 +
.../request/snapshot/OMSnapshotPurgeRequest.java | 8 +-
.../snapshot/OMSnapshotSetPropertyRequest.java | 8 +-
.../om/service/AbstractKeyDeletingService.java | 586 +------
.../ozone/om/service/DirectoryDeletingService.java | 308 +++-
.../ozone/om/service/KeyDeletingService.java | 213 ++-
.../hadoop/ozone/om/service/QuotaRepairTask.java | 38 +-
.../ozone/om/service/SnapshotDeletingService.java | 74 +-
.../service/SnapshotDirectoryCleaningService.java | 484 ------
.../ozone/om/snapshot/OMDBCheckpointUtils.java | 80 +
.../ozone/om/snapshot/SnapshotDiffManager.java | 2 +-
.../OzoneDelegationTokenSecretManager.java | 19 +-
.../hadoop/ozone/security/OzoneSecretStore.java | 14 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 14 +-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 47 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 22 -
.../snapshot/TestOMSnapshotCreateRequest.java | 46 +
.../TestOMSnapshotMoveTableKeysRequest.java | 16 +
.../TestOMSnapshotPurgeRequestAndResponse.java | 19 +-
...estOMSnapshotSetPropertyRequestAndResponse.java | 24 +-
.../snapshot/TestOMSnapshotCreateResponse.java | 5 +-
.../snapshot/TestOMSnapshotDeleteResponse.java | 6 +-
.../TestOMSnapshotMoveTableKeysResponse.java | 56 +-
.../ozone/om/service/TestKeyDeletingService.java | 23 +-
.../om/service/TestOpenKeyCleanupService.java | 3 +-
.../om/snapshot/TestFSODirectoryPathResolver.java | 19 +-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 3 -
.../snapshot/TestSnapshotRequestAndResponse.java | 9 +
.../filter/TestReclaimableRenameEntryFilter.java | 5 +-
.../TestOzoneDelegationTokenSecretManager.java | 41 +
.../apache/hadoop/ozone/recon/ReconConstants.java | 2 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 2 +-
.../hadoop/ozone/recon/api/BlocksEndPoint.java | 30 +-
.../hadoop/ozone/recon/api/NSSummaryEndpoint.java | 2 +-
.../scm/ReconStorageContainerManagerFacade.java | 8 +-
.../recon/spi/ReconContainerMetadataManager.java | 6 -
.../impl/ReconContainerMetadataManagerImpl.java | 5 +-
.../ozone/recon/tasks/ContainerSizeCountTask.java | 2 +-
.../webapps/recon/ozone-recon-web/api/routes.json | 38 +-
.../src/components/navBar/navBar.tsx | 10 +-
.../src/constants/breadcrumbs.constants.tsx | 2 +-
.../src/v2/components/duMetadata/duMetadata.tsx | 2 +-
.../src/v2/components/navBar/navBar.tsx | 14 +-
.../src/v2/constants/breadcrumbs.constants.tsx | 2 +-
.../src/v2/pages/diskUsage/diskUsage.tsx | 30 +-
.../src/views/diskUsage/diskUsage.tsx | 10 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 7 +-
.../scm/AbstractReconContainerManagerTest.java | 1 -
.../ozone/recon/tasks/TestFileSizeCountTask.java | 9 +-
.../ozone/recon/tasks/TestOmTableInsightTask.java | 3 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 42 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 25 +-
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 60 +
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 4 +
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 5 +
.../ozone/s3/endpoint/BucketEndpointBuilder.java | 1 +
.../hadoop/ozone/s3/endpoint/TestBucketAcl.java | 25 +-
.../hadoop/ozone/s3/endpoint/TestBucketDelete.java | 2 -
.../hadoop/ozone/s3/endpoint/TestBucketList.java | 53 +-
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 8 +-
.../ozone/s3/endpoint/TestPermissionCheck.java | 13 +-
.../hadoop/ozone/s3/endpoint/TestS3Owner.java | 133 ++
.../ozone/s3/metrics/TestS3GatewayMetrics.java | 17 +-
.../org/apache/hadoop/ozone/debug/CheckNative.java | 63 +-
.../debug/replicas/BlockExistenceVerifier.java | 14 +-
.../ozone/debug/replicas/ChecksumVerifier.java | 14 +-
.../debug/replicas/ContainerStateVerifier.java | 18 +-
.../ozone/debug/replicas/ReplicaVerifier.java | 2 +-
.../ozone/debug/replicas/ReplicasVerify.java | 7 +-
.../debug/replicas/chunk/ChunkKeyHandler.java | 7 +-
.../apache/hadoop/ozone/debug/TestCheckNative.java | 1 +
pom.xml | 44 +-
312 files changed, 8065 insertions(+), 4766 deletions(-)
create mode 100644 hadoop-hdds/docs/content/design/dn-usedspace-calculation.md
create mode 100644 hadoop-hdds/docs/content/interface/native-cpp.md
create mode 100644 hadoop-hdds/docs/content/start/ozone-recon.png
create mode 100644 hadoop-hdds/docs/content/start/ozone-scm.png
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java
copy
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/{DUFactory.java
=> DUOptimizedFactory.java} (54%)
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RawKeyValue.java
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/fs/TestDUOptimized.java
copy
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/fs/{TestDUFactory.java
=> TestDUOptimizedFactory.java} (75%)
copy
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmKeyArgs.java
=>
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestArchiver.java
(52%)
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/OldSnapshotDiffJobCodecForTesting.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmSnapshotDiffJobCodec.java
create mode 100644
hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/Ozone -
DeleteKeyProgress Metrics.json
copy hadoop-ozone/dist/src/main/compose/common/{hadoop.yaml => ranger.yaml}
(52%)
create mode 100644
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/ranger.yaml
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/TestRemoteEx.java
rename hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/{fs/ozone
=> ozone/om/service}/TestDirectoryDeletingServiceWithFSO.java (98%)
rename hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/{fs/ozone
=> ozone/om/service}/TestRootedDDSWithFSO.java (97%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{snapshot
=> service}/TestSnapshotDeletingServiceIntegrationTest.java (58%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOMDBCheckpointUtils.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotInternalMetrics.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/SnapshotDirectoryCleaningService.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OMDBCheckpointUtils.java
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestS3Owner.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@ozone.apache.org
For additional commands, e-mail: commits-h...@ozone.apache.org
