This is an automated email from the ASF dual-hosted git repository.
sammichen pushed a change to branch HDDS-5713
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 21dcb9c1400 HDDS-13291. [DiskBalancer] Add Performance Test for
VolumeChoosingPolicy in DiskBalancer (#8661)
add a4bb94ee639 HDDS-10239. Storage Container Reconciliation. (#6121)
add 930d3f0552b HDDS-10372. SCM and Datanode communication for
reconciliation (#6506)
add 5bde01db4c1 HDDS-10923. Container Scanner should still scan unhealthy
containers. (#6809)
add d585363a7ab HDDS-10887. Implement a basic Merkle Tree Manager. (#6778)
add 8676000a6f6 Merge branch 'master' into
HDDS-10239-container-reconciliation
add bde10ab8369 HDDS-10373. Implement framework for capturing Merkle Tree
Metrics. (#6864)
add d7f302e25f0 HDDS-10926. Block deletion should update container merkle
tree. (#6875)
add ab35173e249 HDDS-10376. Add a Datanode API to supply a merkle tree for
a given container. (#6945)
add 98369a8343f Merge branch 'master' into
HDDS-10239-container-reconciliation
add 0baba47daad HDDS-11254. Reconcile commands should be handled by
datanode ReplicationSupervisor (#7076)
add 3355701853c Merge branch 'master' into
HDDS-10239-container-reconciliation
add 5cd6a072b20 Merge branch 'master' into
HDDS-10239-container-reconciliation
add c6860db56ce HDDS-10379. Datanodes should generate initial container
merkle tree during container close. (#7065)
add 9765a6a1347 HDDS-11253. Handle corrupted merkle tree files (#7083)
add 01b44374345 Merge branch 'master' into
HDDS-10239-container-reconciliation
add 04c196c9f40 HDDS-10377. Allow datanodes to do chunk level
modifications to closed containers. (#7111)
add 445eaf1f4a8 HDDS-11290. Container scanner should keep scanning after
non-fatal errors (#7127)
add 986e233599b HDDS-11471. Add new tests for container scanner detecting
multiple errors in one container (#7396)
add d17c41c0a97 HDDS-10928. Implement container comparison logic within
datanodes. (#7293)
add 7bcc9430770 Revert "Merge branch 'master' into
HDDS-10239-container-reconciliation"
add cfcabf14223 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation-merge
add 2cfd22a407c Merge branch 'master' into
HDDS-10239-container-reconciliation
add 31875f39ae0 HDDS-12079. Improve container merkle tree write handling
(#7694)
add fa8e0a422ce HDDS-11345. Add metrics specific tests for reconciliation
tasks. (#7732)
add 1aef378d70f Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation-merge
add 8fc98b6b340 Merge branch 'master' into
HDDS-10239-container-reconciliation
add be53ed7115e HDDS-12397. Persist putBlock for closed container. (#7943)
add f47b4076464 HDDS-11887. Recon - Identify container replicas difference
based on content checksums (#7942)
add b34c537940b HDDS-11763. Implement container repair logic within
datanodes. (#7474)
add 8768d048160 HDDS-12745. Container checksum should be reported during
container close and DN restart (#8204)
add 989746a0de3 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 7c591b41984 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 2b4708b70e5 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 229e5c94eaf HDDS-12849. On demand container scanner should not be
static (#8296)
add a355664093c HDDS-12980. Add unit test framework for reconciliation.
(#8402)
add 47c1eaa1eb9 Merge branch 'HDDS-10239-container-reconciliation' of
https://github.com/apache/ozone into HDDS-10239-container-reconciliation
add 9a445ed01eb HDDS-10374. Make container scanner generate merkle trees
during the scan (#7490)
add 0a53c73de21 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 9958f5bff0c HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to
CVE-2025-48734 (#8646)
add a7fc290c200 HDDS-13236. Change Table methods not to throw IOException.
(#8645)
add 6df3077fe19 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (#8503)
add d9049a2aea5 HDDS-13220. Change Recon 'Negative usedBytes' message
loglevel to DEBUG (#8648)
add 9d2b4158e7c HDDS-13234. Expired secret key can abort leader OM
startup. (#8601)
add 97262aa6d6c HDDS-13256. Updated OM Snapshot Grafana Dashboard to
reflect metric updates from HDDS-13181. (#8639)
add de683aad88f HDDS-13263. Refactor DB Checkpoint Utilities. (#8620)
add b574709dd6d HDDS-12924. datanode used space calculation optimization
(#8365)
add f713e57b190 HDDS-12482. Avoid using CommonConfigurationKeys (#8647)
add e9c0a45888b HDDS-13262. Simplify key name validation (#8619)
add f3050cffff4 HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in
SCM/Datanode. (#8638)
add a0af93e2108 HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>`
in test (#8657)
add 732985958d7 HDDS-13023. Container checksum is missing after container
import (#8459)
add 8c1b439d51e HDDS-13296. Integration check always passes due to missing
output (#8662)
add da53b5b4d64 HDDS-13299. Fix failures related to delete (#8665)
add 3c4b3bec36e Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add d207c184281 HDDS-13301. Fix TestSchemaOneBackwardsCompatibility (#8666)
add af3a0f052a6 HDDS-13307. integration (flaky) fails with all tests
passing (#8667)
add e1365a79dca HDDS-13286. Fail stream write when the volume is full.
(#8644)
add db6ed84df3c HDDS-13294. Remove the Table.close() method. (#8658)
add c043be77cf4 HDDS-13312. Bump maven-patch-plugin to 1.3 (#8673)
add b86fad8b9dc HDDS-13316. Bump commons-text to 1.13.1 (#8669)
add 774b3ee78a0 HDDS-13315. Bump log4j2 to 2.25.0 (#8670)
add 803fa3194eb HDDS-13313. Bump maven-javadoc-plugin to 3.11.2 (#8672)
add 05567e68805 HDDS-13317. Table should support empty array/String (#8676)
add e60f22745cc Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 38673465d1a HDDS-13288. Container checksum file proto changes to
account for deleted blocks. (#8649)
add a12d0a5626f HDDS-13306. Intermittent failure in
testDirectoryDeletingServiceIntervalReconfiguration (#8682)
add ff7a20d4a7b HDDS-13270. Reduce getBucket API invocations in S3 bucket
owner verification (#8653)
add b9a3775ca13 HDDS-13289. Remove usage of Jetty StringUtil (#8684)
add c34f4c1ce8a HDDS-13322. Remove module auto-detection from
flaky-test-check (#8679)
add fa02d7d3e6a Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
add 11ada9f9b6f HDDS-13318. Simplify the getRangeKVs methods in Table
(#8683)
add 1c7bcd1e77f HDDS-13325. Introduce OZONE_SERVER_OPTS for common options
for server processes (#8685)
add 48c985f8f65 HDDS-13309. Add keyIterator/valueIterator methods to
Table. (#8675)
add 23c04bd8048 HDDS-13240. Add newly added metrics into grafana
dashboard. (#8656)
add c1e84d90610 HDDS-13324. Optimize memory footprint for Recon listKeys
API (#8680)
add 278bcdda947 HDDS-13314. Remove unused maven-pdf-plugin (#8686)
add fd97f743e27 HDDS-13295. Remove jackson1 exclusions for hadoop-common
(#8687)
add 84da9cee920 HDDS-13319. Simplify KeyPrefixFilter (#8692)
add 96390ac1427 HDDS-12984. Use InodeID to identify the SST files inside
the tarball. (#8477)
add 29fc8def65b HDDS-12070. Bump Ratis to 3.2.0 (#8689)
add d0d09aacf01 HDDS-13251. Support dynamic Byteman scripts via bmsubmit
in ozonesecure-ha (#8654)
add 665574c663e HDDS-13251. Update Byteman usage README license (#8700)
add 99c163ad370 HDDS-12151. Fail write when volume is full considering min
free space (#8642)
add 7a237fe7fbb HDDS-12468. Check for space availability for all dns while
container creation in pipeline (#8663)
add bb3d2872e9f HDDS-12890. Add acceptance robot tests for ozone debug
replicas verify (#8613)
add da4d1ddbf19 HDDS-13280. Smoketest Ozone with Ranger authorizer (#8635)
add 97a89f1147d HDDS-13347. Bump maven-core to 3.9.10 (#8711)
add 65088798ba4 HDDS-13348. Bump maven-deploy-plugin to 3.1.4 (#8712)
add 4fa36540293 HDDS-13349. Bump junit to 5.13.2 (#8714)
add 0f1b93e0856 HDDS-13350. Bump commons-codec to 1.18.0 (#8715)
add 2aa71ac27a0 HDDS-13351. Bump commons-configuration2 to 2.12.0 (#8713)
add 351a6594a10 HDDS-12953. Directly create DataNodeSafeModeRule in
TestDataNodeSafeModeRule (#8716)
add 18845e0954e HDDS-13058. Fix dependency warnings in
ozone-filesystem-hadoop* (#8694)
add fa3b3684b1c HDDS-12957. Remove unused SafeModeStatus#forceExitSafeMode
field (#8717)
add f5019b2d2d2 HDDS-12945. Let Robot test check config whether security
is enabled (#8632)
add b1c27bad3f1 HDDS-13359. [Docs] Accessing Ozone using Cyberduck. (#8718)
add c48fc269409 HDDS-13355. Remove unused class OzoneManagerLockUtil
(#8720)
add 9a6428569e4 HDDS-13221. Remove unused flushFuture from
OMClientResponse (#8723)
add d79ea9c47a5 HDDS-13281. Disable Ratis metadata write to Raft Log on OM
& SCM. (#8637)
add f2ddbf6a474 HDDS-13149. Add details on Performance Optimization to
Transparent Data Encryption doc (#8622)
add ba7b3cfcd9b HDDS-13330. DBStore and related classes should not throw
IOException (#8691)
add 5e72dd4e1ef HDDS-13208. [Docs] Add volume management section under
Architecture/Datanodes. (#8585)
add b9d4e6585bb HDDS-13371. [Docs] Protect in-transit traffic (#8729)
add 627bebe7746 HDDS-13368. [Docs] Add user doc for Ozone Trash. (#8727)
add d80589388ec HDDS-13269. [Docs] Multi-raft user doc. (#8641)
add 1ca2f2a2aa2 HDDS-13358. Refactor SafeModeStatus to an enum (#8721)
add d9daba0ee80 HDDS-12727. Document high level architecture of S3 Gateway
(#8184)
add e582a3908f5 HDDS-13366. Add unique key to ACL table rows. (#8725)
add 9e5f8371b81 HDDS-13260. Add smoke test for bucket owner condition
(#8702)
add d6d92538703 HDDS-13362. Abstract OM response class should not be
annotated with CleanupTableInfo (#8732)
add 380f31d4257 HDDS-13341. Rename ScanResult.isHealthy() to hasErrors()
(#8746)
add 30db6fc2116 HDDS-13390. Bump license-maven-plugin to 2.6.0 (#8750)
add 72ffc27a9dc HDDS-13340. Fix minor items identified during
reconciliation branch merge process. (#8705)
add 550503c47ae HDDS-13392. Bump awssdk to 2.31.77 (#8748)
add 65f0c09a464 HDDS-13381. Docs: Add user documentation for Volumes,
Buckets, and Keys (#8739)
add 07f08104239 HDDS-13389. Bump junit to 5.13.3 (#8749)
add afcdebdc8f9 HDDS-13391. Bump sqlite-jdbc to 3.50.2.0 (#8751)
add 0b7bb92fd37 HDDS-13394. Bump zstd-jni to 1.5.7-4 (#8747)
add f51bb1bed2a HDDS-13087. [snapshot] Expand user doc (#8496)
add 39def5d50c9 HDDS-13268. [Docs] Advanced topics: troubleshooting OM
snapshot installation. (#8719)
add 3a641357601 HDDS-13198. Avoid warning due to wrong
min.free.space.percent default value (#8745)
add f7270513a5f HDDS-12929. Datanode Should Immediately Trigger Container
Close when Volume Full (#8460)
add 5af3467a423 HDDS-13375. Add Ozone logo to the Github README.md (#8741)
add 9b713d0b659 HDDS-12395. Add Mismatched Replica tab to the Containers
Page (#8407)
add 14748e831d8 HDDS-13401. Mark TestContainerCommandReconciliation as
flaky
add 1315df64617 HDDS-13337. Reduce config required to enable Byteman
listener (#8701)
add b610fd0a42e HDDS-13327. Improve log and error handling while starting
Recon (#8688)
add 5255c8cbc78 HDDS-13122. Add acceptance test for `ozone repair om
compact` (#8703)
add bca868865e2 HDDS-13098. Create PicoCLI mixin to standardize datanode
selection options (#8674)
add bf9cdd0df95 HDDS-13162. Improve Container Replica Debugger Tool
documentation (#8597)
add 05c7e62f45a HDDS-13404. Improve persistent test data dir
creation/cleanup (#8762)
add 7f2931707c3 Merge remote-tracking branch 'apacheupstream' into
HDDS-5713
No new revisions were added by this update.
Summary of changes:
.github/workflows/ci.yml | 2 +-
.github/workflows/intermittent-test-check.yml | 63 +-
README.md | 13 +
...s-no-flush.btm => skip-notify-group-remove.btm} | 15 +-
.../{ratis-no-flush.btm => skip-put-block.btm} | 15 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 11 +-
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 4 +
.../hdds/scm/client/TestHddsClientUtils.java | 1 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 6 +
.../hadoop/hdds/client/ReplicationConfig.java | 1 +
.../apache/hadoop/hdds/scm/client/ScmClient.java | 8 +
.../hdds/scm/container/ContainerReplicaInfo.java | 28 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 3 +-
.../apache/hadoop/hdds/scm/net/NetConstants.java | 2 +-
.../protocol/StorageContainerLocationProtocol.java | 8 +
.../ContainerCommandResponseBuilders.java | 11 +
.../hdds/scm/storage/ContainerProtocolCalls.java | 39 ++
.../apache/hadoop/hdds/utils/db/Proto3Codec.java | 22 +-
.../ozone/container/common/helpers/BlockData.java | 6 +-
.../common/src/main/resources/ozone-default.xml | 10 -
.../apache/hadoop/hdds/utils/db/CodecTestUtil.java | 29 +
.../ozone/HddsDatanodeClientProtocolServer.java | 4 +-
.../org/apache/hadoop/ozone/audit/DNAction.java | 3 +-
.../checksum/ContainerChecksumTreeManager.java | 452 ++++++++++++++
.../container/checksum/ContainerDiffReport.java | 113 ++++
.../checksum/ContainerMerkleTreeMetrics.java | 155 +++++
.../checksum/ContainerMerkleTreeWriter.java | 245 ++++++++
.../checksum/DNContainerOperationClient.java | 134 ++++
.../container/checksum/ReconcileContainerTask.java | 96 +++
.../ozone/container/checksum}/package-info.java | 4 +-
.../container/common/helpers/ContainerMetrics.java | 20 +-
.../container/common/helpers/ContainerUtils.java | 52 +-
.../helpers}/TokenHelper.java | 14 +-
.../common/impl/BlockDeletingService.java | 22 +-
.../ozone/container/common/impl/ContainerData.java | 39 +-
.../container/common/impl/ContainerDataYaml.java | 4 +-
.../ozone/container/common/impl/ContainerSet.java | 42 ++
.../container/common/impl/HddsDispatcher.java | 52 +-
.../container/common/interfaces/Container.java | 91 +--
.../container/common/interfaces/DBHandle.java | 4 +-
.../ozone/container/common/interfaces/Handler.java | 31 +-
.../{package-info.java => ScanResult.java} | 15 +-
.../common/statemachine/DatanodeConfiguration.java | 78 ++-
.../common/statemachine/DatanodeStateMachine.java | 7 +
....java => ReconcileContainerCommandHandler.java} | 62 +-
.../states/endpoint/HeartbeatEndpointTask.java | 6 +
.../container/common/utils/ContainerLogger.java | 41 +-
.../container/common/utils/DatanodeStoreCache.java | 14 +-
.../hadoop/ozone/container/common/utils/RawDB.java | 3 +-
.../container/common/utils/ReferenceCountedDB.java | 12 +-
.../ozone/container/common/volume/HddsVolume.java | 41 +-
.../container/common/volume/MutableVolumeSet.java | 11 +
.../container/common/volume/StorageVolume.java | 28 +-
.../ozone/container/common/volume/VolumeUsage.java | 1 -
.../ECReconstructionCoordinator.java | 3 +-
.../container/keyvalue/KeyValueContainer.java | 37 +-
.../container/keyvalue/KeyValueContainerCheck.java | 534 ++++++++--------
.../container/keyvalue/KeyValueContainerData.java | 6 +-
.../KeyValueContainerMetadataInspector.java | 16 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 660 +++++++++++++++++++-
.../container/keyvalue/TarContainerPacker.java | 15 +
.../keyvalue/helpers/KeyValueContainerUtil.java | 56 +-
.../container/keyvalue/impl/BlockManagerImpl.java | 91 ++-
.../keyvalue/impl/FilePerBlockStrategy.java | 26 +-
.../keyvalue/impl/KeyValueStreamDataChannel.java | 1 +
.../keyvalue/impl/StreamDataChannelBase.java | 5 +
.../keyvalue/interfaces/BlockManager.java | 24 +
.../statemachine/background/BlockDeletingTask.java | 65 +-
.../container/metadata/AbstractDatanodeStore.java | 51 +-
.../ozone/container/metadata/AbstractRDBStore.java | 26 +-
.../ozone/container/metadata/DBStoreManager.java | 16 +-
.../metadata/DatanodeStoreSchemaThreeImpl.java | 13 +-
.../ozone/container/metadata/DatanodeTable.java | 57 +-
.../metadata/SchemaOneChunkInfoListCodec.java | 10 +-
.../metadata/SchemaOneDeletedBlocksTable.java | 55 +-
.../WitnessedContainerMetadataStoreImpl.java | 7 +-
.../ozoneimpl/BackgroundContainerDataScanner.java | 60 +-
.../BackgroundContainerMetadataScanner.java | 39 +-
.../container/ozoneimpl/ContainerController.java | 48 +-
.../container/ozoneimpl/ContainerScanError.java | 71 +++
.../container/ozoneimpl/ContainerScanHelper.java | 164 +++++
.../ozoneimpl/ContainerScannerConfiguration.java | 4 +
.../ozone/container/ozoneimpl/DataScanResult.java | 69 +++
.../container/ozoneimpl/MetadataScanResult.java | 90 +++
.../ozoneimpl/OnDemandContainerDataScanner.java | 150 +----
.../ozone/container/ozoneimpl/OzoneContainer.java | 39 +-
.../container/replication/ContainerImporter.java | 2 +-
.../replication/ReplicationSupervisor.java | 4 +-
.../commands/ReconcileContainerCommand.java | 105 ++++
.../checksum/ContainerMerkleTreeTestUtils.java | 349 +++++++++++
.../checksum/TestContainerChecksumTreeManager.java | 575 ++++++++++++++++++
.../checksum/TestContainerMerkleTreeWriter.java | 317 ++++++++++
.../checksum/TestReconcileContainerTask.java | 124 ++++
.../common/BlockDeletingServiceTestImpl.java | 3 +-
.../ozone/container/common/ContainerTestUtils.java | 129 +++-
.../container/common/TestBlockDeletingService.java | 126 ++--
.../common/TestKeyValueContainerData.java | 4 +
.../TestSchemaOneBackwardsCompatibility.java | 16 +-
.../TestSchemaTwoBackwardsCompatibility.java | 4 +-
.../common/impl/TestContainerDataYaml.java | 34 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 3 +-
.../common/impl/TestContainerPersistence.java | 3 +-
.../container/common/impl/TestContainerSet.java | 57 ++
.../container/common/impl/TestHddsDispatcher.java | 30 +-
.../container/common/interfaces/TestHandler.java | 4 +-
.../statemachine/TestDatanodeConfiguration.java | 9 +
.../common/statemachine/TestStateContext.java | 4 +
.../TestCloseContainerCommandHandler.java | 12 +-
.../TestReconcileContainerCommandHandler.java | 188 ++++++
.../states/endpoint/TestHeartbeatEndpointTask.java | 39 ++
.../container/common/volume/TestHddsVolume.java | 24 +
.../diskbalancer/TestDiskBalancerService.java | 7 +-
.../keyvalue/TestContainerCorruptions.java | 233 +++++++
...stContainerReconciliationWithMockDatanodes.java | 623 +++++++++++++++++++
.../keyvalue/TestKeyValueBlockIterator.java | 8 +-
.../container/keyvalue/TestKeyValueContainer.java | 2 +
.../keyvalue/TestKeyValueContainerCheck.java | 321 +++++++---
.../container/keyvalue/TestKeyValueHandler.java | 252 +++++++-
.../TestKeyValueHandlerWithUnhealthyContainer.java | 57 +-
.../container/keyvalue/TestTarContainerPacker.java | 2 +-
.../keyvalue/impl/TestBlockManagerImpl.java | 75 ++-
.../keyvalue/impl/TestFilePerBlockStrategy.java | 204 +++++++
.../impl/TestKeyValueStreamDataChannel.java | 30 +
.../TestBackgroundContainerDataScanner.java | 73 ++-
.../TestBackgroundContainerMetadataScanner.java | 55 +-
.../ozoneimpl/TestContainerScannersAbstract.java | 33 +-
.../container/ozoneimpl/TestDataScanResult.java | 83 +++
.../ozoneimpl/TestMetadataScanResult.java | 58 ++
.../TestOnDemandContainerDataScanner.java | 157 +++--
.../container/ozoneimpl/TestOzoneContainer.java | 8 +-
.../replication/TestContainerImporter.java | 4 +-
.../replication/TestGrpcReplicationService.java | 8 +-
.../replication/TestReplicationSupervisor.java | 71 ++-
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 3 +-
hadoop-hdds/docs/content/concept/Datanodes.md | 34 ++
hadoop-hdds/docs/content/concept/OzoneS3Gateway.md | 112 ++++
.../docs/content/concept/VolumesBucketsKeys.md | 183 ++++++
.../content/design/container-reconciliation.md | 344 +++++++++++
.../content/design/dn-usedspace-calculation.md | 91 +++
.../docs/content/design/full-volume-handling.md | 163 +++++
hadoop-hdds/docs/content/feature/OM-HA.md | 1 +
hadoop-hdds/docs/content/feature/Snapshot.md | 219 +++++--
hadoop-hdds/docs/content/feature/Trash.md | 74 +++
.../docs/content/feature/multi-raft-support.md | 114 ++++
.../docs/content/interface/CyberDuckOzoneS3.md | 122 ++++
hadoop-hdds/docs/content/interface/S3.md | 1 +
hadoop-hdds/docs/content/security/SecuringTDE.md | 42 ++
.../content/security/protect-in-transit-traffic.md | 60 ++
.../docs/content/tools/ContainerLogParser.md | 26 +-
.../docs/content/tools/ContainerLogParser.zh.md | 239 ++++++++
.../_index.zh.md => troubleshooting/_index.md} | 11 +-
.../troubleshooting/om-ha-snapshot-installation.md | 39 ++
.../main/java/org/apache/hadoop/hdds/fs/DU.java | 30 +-
.../org/apache/hadoop/hdds/fs/DUOptimized.java | 66 ++
.../fs/{DUFactory.java => DUOptimizedFactory.java} | 52 +-
.../hadoop/hdds/fs/SpaceUsageCheckFactory.java | 14 +-
.../hadoop/hdds/fs/SpaceUsageCheckParams.java | 9 +
...inerLocationProtocolClientSideTranslatorPB.java | 10 +
.../hadoop/hdds/server/http/HttpServer2.java | 16 +-
.../org/apache/hadoop/hdds/utils/Archiver.java | 43 ++
.../hadoop/hdds/utils/DBCheckpointServlet.java | 58 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 8 +-
.../hadoop/hdds/utils/MetadataKeyFilters.java | 151 ++---
.../hdds/utils/db/DBColumnFamilyDefinition.java | 6 +-
.../hadoop/hdds/utils/db/DBConfigFromFile.java | 8 +-
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 86 +--
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 30 +-
.../hadoop/hdds/utils/db/RDBBatchOperation.java | 24 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 99 +--
.../hdds/utils/db/RDBStoreAbstractIterator.java | 15 +-
.../hdds/utils/db/RDBStoreByteArrayIterator.java | 17 +-
.../hdds/utils/db/RDBStoreCodecBufferIterator.java | 15 +-
.../org/apache/hadoop/hdds/utils/db/RDBTable.java | 121 ++--
.../org/apache/hadoop/hdds/utils/db/Table.java | 157 ++---
.../apache/hadoop/hdds/utils/db/TableIterator.java | 55 +-
.../apache/hadoop/hdds/utils/db/TypedTable.java | 169 +++---
.../org/apache/hadoop/hdds/fs/TestDUOptimized.java | 71 +++
...tDUFactory.java => TestDUOptimizedFactory.java} | 26 +-
.../hadoop/hdds/utils/MapBackedTableIterator.java | 7 +-
.../org/apache/hadoop/hdds/utils/TestArchiver.java | 89 +++
.../hadoop/hdds/utils/TestRDBSnapshotProvider.java | 23 +-
.../hadoop/hdds/utils/db/InMemoryTestTable.java | 18 +-
.../hadoop/hdds/utils/db/TestDBStoreBuilder.java | 56 +-
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 160 ++---
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 669 ++++++++++----------
.../hdds/utils/db/TestTypedRDBTableStore.java | 323 +++++-----
.../hadoop/hdds/utils/db/TestTypedTable.java | 251 ++++++--
.../src/main/proto/ScmAdminProtocol.proto | 22 +-
.../src/main/proto/DatanodeClientProtocol.proto | 43 +-
.../interface-client/src/main/proto/hdds.proto | 2 +
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 18 +-
.../src/main/proto/ScmServerProtocol.proto | 1 +
.../hdds/utils/db/RocksDatabaseException.java | 10 +-
hadoop-hdds/server-scm/pom.xml | 4 +
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 10 +-
.../scm/block/DeletedBlockLogStateManager.java | 5 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 22 +-
.../container/AbstractContainerReportHandler.java | 1 +
.../hdds/scm/container/ContainerManager.java | 9 +-
.../hdds/scm/container/ContainerManagerImpl.java | 27 +-
.../hdds/scm/container/ContainerReplica.java | 13 +
.../hdds/scm/container/ContainerStateManager.java | 5 -
.../scm/container/ContainerStateManagerImpl.java | 17 +-
.../ReconcileContainerEventHandler.java | 92 +++
.../ReconciliationEligibilityHandler.java | 139 +++++
.../container/reconciliation}/package-info.java | 5 +-
.../apache/hadoop/hdds/scm/events/SCMEvents.java | 7 +
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 6 +
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 18 +-
.../hdds/scm/ha/SCMDBCheckpointProvider.java | 5 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 4 +-
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 25 +-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 1 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 3 +-
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 9 +
.../hdds/scm/pipeline/PipelineManagerImpl.java | 16 +
.../hdds/scm/pipeline/PipelineStateManager.java | 2 +-
.../scm/pipeline/PipelineStateManagerImpl.java | 16 +-
.../scm/pipeline/WritableECContainerProvider.java | 6 +
...inerLocationProtocolServerSideTranslatorPB.java | 13 +
.../hdds/scm/safemode/SCMSafeModeManager.java | 91 +--
.../hdds/scm/server/SCMBlockProtocolServer.java | 4 +-
.../hadoop/hdds/scm/server/SCMCertStore.java | 4 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 48 +-
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 13 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 7 -
.../hdds/scm/server/StorageContainerManager.java | 14 +-
.../upgrade/FinalizationStateManagerImpl.java | 1 -
.../org/apache/hadoop/ozone/audit/SCMAction.java | 3 +-
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 14 +
.../hadoop/hdds/scm/block/TestBlockManager.java | 5 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 4 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 6 +
.../hdds/scm/container/SimpleMockNodeManager.java | 2 +
.../scm/container/TestContainerManagerImpl.java | 67 +-
.../scm/container/TestContainerReportHandler.java | 152 ++++-
.../scm/container/TestContainerStateManager.java | 1 -
.../TestIncrementalContainerReportHandler.java | 139 ++++-
.../scm/container/TestUnknownContainerReport.java | 18 -
.../TestReconcileContainerEventHandler.java | 312 ++++++++++
.../hdds/scm/ha/TestBackgroundSCMService.java | 5 +-
.../apache/hadoop/hdds/scm/ha/TestSCMContext.java | 7 +-
.../hadoop/hdds/scm/ha/TestSCMServiceManager.java | 20 +-
.../hdds/scm/node/TestContainerPlacement.java | 1 -
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 6 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 6 +-
.../hdds/scm/pipeline/MockPipelineManager.java | 5 +
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 90 ++-
.../scm/safemode/TestDataNodeSafeModeRule.java | 23 +-
.../hdds/scm/cli/ContainerOperationClient.java | 5 +
.../hdds/scm/cli/container/ContainerCommands.java | 1 +
...oseSubcommand.java => ReconcileSubcommand.java} | 19 +-
.../cli/datanode/DecommissionStatusSubCommand.java | 33 +-
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 30 +-
.../hdds/scm/cli/datanode/NodeSelectionMixin.java | 66 ++
.../hdds/scm/cli/datanode/UsageInfoSubcommand.java | 25 +-
.../datanode/TestDecommissionStatusSubCommand.java | 35 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 2 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 7 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 5 -
.../java/org/apache/hadoop/ozone/om/OmConfig.java | 20 +
.../hadoop/ozone/om/helpers/SnapshotDiffJob.java | 13 +-
.../hadoop/ozone/om/lock/OzoneManagerLockUtil.java | 96 ---
.../org/apache/hadoop/ozone/om/TestOmConfig.java | 2 +
.../helpers/OldSnapshotDiffJobCodecForTesting.java | 56 ++
.../om/helpers/TestOmSnapshotDiffJobCodec.java | 73 +++
hadoop-ozone/dev-support/checks/_post_process.sh | 10 +-
hadoop-ozone/dev-support/checks/integration.sh | 3 +
hadoop-ozone/dev-support/checks/junit.sh | 3 +-
hadoop-ozone/dev-support/checks/rat.sh | 3 +-
hadoop-ozone/dist/pom.xml | 2 +-
.../dashboards/Ozone - DeleteKey Metrics.json | 406 ++++++++++++-
.../Ozone - DeleteKeyProgress Metrics.json | 2 +-
.../grafana/dashboards/Ozone - OM Snapshot.json | 295 ++++++++-
.../dist/src/main/compose/common/hadoop-test.sh | 2 +-
.../dist/src/main/compose/common/init-kdc.sh | 5 +
.../dist/src/main/compose/common/ranger.yaml | 4 +
.../dist/src/main/compose/common/replicas-test.sh | 64 +-
.../src/main/compose/compatibility/docker-config | 1 +
.../dist/src/main/compose/ozone-om-prepare/test.sh | 14 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 1 -
.../src/main/compose/ozonesecure-ha/docker-config | 3 +
.../src/main/compose/ozonesecure-ha/ranger.yaml | 13 +-
.../compose/ozonesecure-ha/test-debug-tools.sh | 17 +-
.../src/main/compose/ozonesecure-ha/test-ranger.sh | 4 +
.../{test-leadership.sh => test-repair-tools.sh} | 25 +-
.../dist/src/main/compose/ozonesecure-ha/test.sh | 3 +
hadoop-ozone/dist/src/main/compose/restart/test.sh | 3 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 18 +-
.../src/main/compose/upgrade/compose/ha/load.sh | 3 +-
.../main/compose/upgrade/compose/non-ha/load.sh | 3 +-
.../dist/src/main/compose/upgrade/testlib.sh | 15 -
.../dist/src/main/dockerlibexec/entrypoint.sh | 4 +
.../dist/src/main/smoketest/__init__.robot | 2 +-
.../dist/src/main/smoketest/admincli/cert.robot | 1 +
.../src/main/smoketest/admincli/container.robot | 52 +-
.../src/main/smoketest/admincli/datanode.robot | 12 +-
.../src/main/smoketest/admincli/pipeline.robot | 2 +-
.../src/main/smoketest/admincli/reconfigure.robot | 3 +-
.../smoketest/admincli/replicationmanager.robot | 2 +-
.../src/main/smoketest/balancer/testBalancer.robot | 3 +-
.../dist/src/main/smoketest/basic/basic.robot | 1 +
.../dist/src/main/smoketest/basic/links.robot | 1 +
.../src/main/smoketest/basic/ozone-shell-lib.robot | 1 +
.../src/main/smoketest/basic/ozone-shell.robot | 3 +-
.../src/main/smoketest/basic/single_node.robot | 1 +
.../smoketest/certrotation/cert-rotation.robot | 2 +-
.../dist/src/main/smoketest/commonlib.robot | 11 +-
.../dist/src/main/smoketest/compatibility/dn.robot | 7 +-
.../compatibility/{recon.robot => lib.resource} | 19 +-
.../dist/src/main/smoketest/compatibility/om.robot | 13 +-
.../src/main/smoketest/compatibility/recon.robot | 11 +-
.../src/main/smoketest/compatibility/scm.robot | 7 +-
.../src/main/smoketest/debug/auditparser.robot | 1 +
...-datanode.robot => block-existence-check.robot} | 17 +-
...datanode.robot => corrupt-block-checksum.robot} | 26 +-
.../debug/ozone-debug-corrupt-block.robot | 46 --
.../smoketest/debug/ozone-debug-keywords.robot | 63 ++
.../src/main/smoketest/debug/ozone-debug-ldb.robot | 2 +-
.../debug/ozone-debug-stale-datanode.robot | 46 --
.../smoketest/debug/ozone-debug-tests-ec3-2.robot | 49 +-
.../smoketest/debug/ozone-debug-tests-ec6-3.robot | 100 ---
.../main/smoketest/debug/ozone-debug-tests.robot | 8 +-
.../src/main/smoketest/debug/ozone-debug.robot | 98 ---
...atanode.robot => stale-datanode-checksum.robot} | 25 +-
.../dist/src/main/smoketest/ec/lib.resource | 1 +
.../dist/src/main/smoketest/ec/rewrite.robot | 2 +-
.../src/main/smoketest/ec/upgrade-ec-check.robot | 2 +-
.../dist/src/main/smoketest/fcq/s3_om_fcq.robot | 1 +
.../src/main/smoketest/freon/generate-chunk.robot | 2 +
.../src/main/smoketest/freon/validate-chunk.robot | 3 +
.../main/smoketest/grpc/grpc-om-s3-metrics.robot | 2 +-
.../main/smoketest/hsync/upgrade-hsync-check.robot | 2 +-
.../main/smoketest/httpfs/operations_tests.robot | 4 +-
.../dist/src/main/smoketest/lib/BytemanLibrary.py | 77 +++
.../omha/data-creation-before-om-bootstrap.robot | 1 +
.../omha/data-validation-after-om-bootstrap.robot | 1 +
.../main/smoketest/omha/om-cancel-prepare.robot | 3 +-
.../src/main/smoketest/omha/om-fetch-key.robot | 2 +-
.../dist/src/main/smoketest/omha/om-prepare.robot | 3 +-
.../dist/src/main/smoketest/omha/om-prepared.robot | 1 +
.../dist/src/main/smoketest/omha/om-roles.robot | 3 +-
.../main/smoketest/ozone-fi/BytemanKeywords.robot | 119 ++++
.../dist/src/main/smoketest/ozone-fi/README.md | 211 +++++++
.../smoketest/ozone-fi/byteman_faults_sample.robot | 60 ++
.../dist/src/main/smoketest/ozone-lib/freon.robot | 1 -
.../dist/src/main/smoketest/recon/recon-api.robot | 1 +
.../src/main/smoketest/recon/recon-nssummary.robot | 1 +
.../main/smoketest/recon/recon-taskstatus.robot | 1 +
.../src/main/smoketest/repair/om-compact.robot | 60 ++
.../src/main/smoketest/s3/MultipartUpload.robot | 32 +
.../dist/src/main/smoketest/s3/bucketcreate.robot | 8 +
.../dist/src/main/smoketest/s3/bucketdelete.robot | 7 +
.../dist/src/main/smoketest/s3/buckethead.robot | 6 +
.../dist/src/main/smoketest/s3/commonawslib.robot | 30 +
.../dist/src/main/smoketest/s3/objectcopy.robot | 9 +
.../dist/src/main/smoketest/s3/objectdelete.robot | 18 +
.../dist/src/main/smoketest/s3/objecthead.robot | 10 +
.../dist/src/main/smoketest/s3/objectlist.robot | 6 +
.../dist/src/main/smoketest/s3/objectputget.robot | 15 +-
.../dist/src/main/smoketest/s3/objecttagging.robot | 15 +
.../dist/src/main/smoketest/s3/s3_getsecret.robot | 2 +-
.../src/main/smoketest/s3/secretgenerate.robot | 4 +-
.../dist/src/main/smoketest/s3/secretrevoke.robot | 4 +-
.../main/smoketest/scmha/scm-decommission.robot | 4 +-
.../smoketest/security/ozone-secure-token.robot | 1 +
.../main/smoketest/snapshot/snapshot-acls.robot | 3 +-
.../snapshot/upgrade-snapshot-check.robot | 2 +-
.../smoketest/upgrade/check-finalization.robot | 1 +
.../dist/src/main/smoketest/upgrade/finalize.robot | 1 +
.../dist/src/main/smoketest/upgrade/generate.robot | 2 +-
.../dist/src/main/smoketest/upgrade/prepare.robot | 1 +
.../dist/src/main/smoketest/upgrade/validate.robot | 2 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 1 +
.../dist/src/shell/ozone/ozone-functions.sh | 12 +
.../ozone/freon/ClosedContainerReplicator.java | 7 +-
.../TestReconInsightsForDeletedDirectories.java | 12 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 15 +-
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 14 +-
.../apache/hadoop/hdds/scm/TestCloseContainer.java | 134 +++-
.../hdds/scm/TestSCMDbCheckpointServlet.java | 19 +-
...estStorageContainerManagerHAWithAllRunning.java | 2 +-
.../hadoop/hdds/upgrade/TestScmHAFinalization.java | 87 +--
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 5 +-
.../ozone/client/rpc/TestECKeyOutputStream.java | 4 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 8 +-
.../apache/hadoop/ozone/container/TestHelper.java | 14 +
.../TestRefreshVolumeUsageHandler.java | 7 +
.../container/metrics/TestContainerMetrics.java | 3 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 75 +++
.../container/server/TestContainerServer.java | 4 +-
.../server/TestSecureContainerServer.java | 3 +-
.../TestContainerCommandReconciliation.java | 673 +++++++++++++++++++++
...tBackgroundContainerDataScannerIntegration.java | 42 +-
...kgroundContainerMetadataScannerIntegration.java | 47 +-
.../TestContainerScannerIntegrationAbstract.java | 207 +------
...estOnDemandContainerDataScannerIntegration.java | 50 +-
.../hadoop/ozone/om/TestOMDbCheckpointServlet.java | 21 +-
.../TestOMDbCheckpointServletInodeBasedXfer.java | 368 +++++++++++
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 3 +
.../ozone/om/TestObjectStoreWithLegacyFS.java | 3 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 3 +-
.../om/TestOzoneManagerHAWithStoppedNodes.java | 3 +-
.../om/TestOzoneManagerListVolumesSecure.java | 7 +
.../TestDirectoryDeletingServiceWithFSO.java | 11 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 6 +-
.../ozone/om/snapshot/TestOMDBCheckpointUtils.java | 100 +++
.../om/snapshot/TestOzoneManagerHASnapshot.java | 4 +-
.../snapshot/TestSnapshotBackgroundServices.java | 6 +-
.../TestSnapshotDirectoryCleaningService.java | 3 +-
.../hadoop/ozone/repair/om/TestFSORepairTool.java | 5 +-
.../hadoop/ozone/shell/TestOzoneTenantShell.java | 2 +-
.../hadoop/ozone/shell/TestReconfigShell.java | 40 +-
.../tools/contract/AbstractContractDistCpTest.java | 3 +-
hadoop-ozone/ozone-manager/pom.xml | 4 -
.../apache/hadoop/ozone/common/BekInfoUtils.java | 4 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 74 +--
.../om/OMDBCheckpointServletInodeBasedXfer.java | 436 +++++++++++++
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 25 +-
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 19 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 3 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 5 +
.../om/request/file/OMDirectoryCreateRequest.java | 6 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 13 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 9 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 10 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 8 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 75 ---
.../ozone/om/request/upgrade/OMPrepareRequest.java | 27 +-
.../hadoop/ozone/om/response/OMClientResponse.java | 11 -
.../response/key/AbstractOMKeyDeleteResponse.java | 4 -
.../AbstractS3MultipartAbortResponse.java | 9 -
.../S3MultipartUploadAbortResponseWithFSO.java | 3 +-
.../hadoop/ozone/om/service/QuotaRepairTask.java | 10 +-
.../ozone/om/snapshot/OMDBCheckpointUtils.java | 80 +++
.../hadoop/ozone/om/snapshot/OmSnapshotUtils.java | 21 +
.../OzoneDelegationTokenSecretManager.java | 19 +-
.../hadoop/ozone/security/OzoneSecretStore.java | 14 +-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 47 +-
.../ozone/om/request/key/TestOMKeyRequest.java | 22 -
.../TestOMSnapshotPurgeRequestAndResponse.java | 3 +-
...estOMSnapshotSetPropertyRequestAndResponse.java | 8 +-
.../ozone/om/response/TestCleanupTableInfo.java | 20 +-
.../snapshot/TestOMSnapshotCreateResponse.java | 5 +-
.../snapshot/TestOMSnapshotDeleteResponse.java | 6 +-
.../ozone/om/service/TestKeyDeletingService.java | 7 +-
.../om/service/TestOpenKeyCleanupService.java | 3 +-
.../om/snapshot/TestFSODirectoryPathResolver.java | 4 +-
.../ozone/om/snapshot/TestSnapshotCache.java | 3 +-
.../filter/TestReclaimableRenameEntryFilter.java | 5 +-
.../TestOzoneDelegationTokenSecretManager.java | 41 ++
hadoop-ozone/ozonefs-hadoop2/pom.xml | 51 +-
hadoop-ozone/ozonefs-hadoop3-client/pom.xml | 25 +
hadoop-ozone/ozonefs-hadoop3/pom.xml | 46 +-
.../recon/schema/ContainerSchemaDefinition.java | 3 +-
hadoop-ozone/recon/pom.xml | 4 -
.../org/apache/hadoop/ozone/recon/ReconServer.java | 46 +-
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 26 +-
.../hadoop/ozone/recon/api/BlocksEndPoint.java | 30 +-
.../api/types/UnhealthyContainersResponse.java | 13 +
.../ozone/recon/fsck/ContainerHealthStatus.java | 15 +-
.../ozone/recon/fsck/ContainerHealthTask.java | 34 +-
.../ozone/recon/fsck/ReconSafeModeMgrTask.java | 6 +-
.../ozone/recon/persistence/ContainerHistory.java | 13 +-
.../recon/recovery/ReconOmMetadataManagerImpl.java | 4 +-
.../ozone/recon/scm/ContainerReplicaHistory.java | 16 +-
.../ozone/recon/scm/ReconContainerManager.java | 18 +-
.../scm/ReconStorageContainerManagerFacade.java | 14 +-
.../recon/spi/ReconContainerMetadataManager.java | 6 -
.../spi/impl/OzoneManagerServiceProviderImpl.java | 3 +
.../impl/ReconContainerMetadataManagerImpl.java | 5 +-
.../ozone/recon/tasks/ContainerSizeCountTask.java | 2 +-
.../ozone/recon/upgrade/ReconLayoutFeature.java | 3 +-
...> UnhealthyContainerReplicaMismatchAction.java} | 34 +-
.../webapps/recon/ozone-recon-web/api/db.json | 40 ++
.../src/__tests__/datanodes/Datanodes.test.tsx | 19 +-
.../__tests__/datanodes/DatanodesTable.test.tsx | 15 +-
.../mocks/pipelineMocks/pipelinesServer.ts | 4 +-
.../src/__tests__/pipelines/Pipelines.test.tsx | 19 +-
.../__tests__/pipelines/PipelinesTable.test.tsx | 8 +-
.../src/__tests__/utils/pipelines.utils.tsx | 4 +-
.../src/components/aclDrawer/aclDrawer.tsx | 5 +-
.../src/components/navBar/navBar.tsx | 8 +-
.../src/v2/components/aclDrawer/aclDrawer.tsx | 2 +-
.../src/v2/components/duMetadata/duMetadata.tsx | 12 +-
.../src/v2/components/navBar/navBar.tsx | 12 +-
.../src/v2/components/tables/containersTable.tsx | 56 +-
.../src/v2/components/tables/pipelinesTable.tsx | 17 +-
.../src/v2/pages/containers/containers.less | 13 +-
.../src/v2/pages/containers/containers.tsx | 27 +-
.../src/v2/pages/diskUsage/diskUsage.tsx | 22 +-
.../src/v2/types/container.types.ts | 2 +
.../src/views/diskUsage/diskUsage.tsx | 29 +-
.../ozone-recon-web/src/views/insights/om/om.tsx | 20 +-
.../views/missingContainers/missingContainers.tsx | 12 +-
.../ozone/recon/api/TestContainerEndpoint.java | 74 ++-
.../recon/fsck/TestContainerHealthStatus.java | 51 ++
.../ozone/recon/fsck/TestContainerHealthTask.java | 42 +-
.../TestContainerHealthTaskRecordGenerator.java | 51 ++
.../scm/AbstractReconContainerManagerTest.java | 1 -
.../ozone/recon/scm/TestReconContainerManager.java | 6 +-
.../ozone/recon/scm/TestReconPipelineManager.java | 6 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 13 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 71 ++-
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 57 +-
.../hadoop/ozone/s3/endpoint/TestS3Owner.java | 36 ++
.../ozone/s3/metrics/TestS3GatewayMetrics.java | 1 +
.../datanode/container/ContainerCommands.java | 5 +-
.../apache/hadoop/ozone/debug/om/PrefixParser.java | 22 +-
pom.xml | 75 +--
510 files changed, 17778 insertions(+), 5092 deletions(-)
copy dev-support/byteman/{ratis-no-flush.btm => skip-notify-group-remove.btm}
(72%)
copy dev-support/byteman/{ratis-no-flush.btm => skip-put-block.btm} (75%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerDiffReport.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeMetrics.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
copy hadoop-hdds/{client/src/main/java/org/apache/hadoop/hdds/scm/client =>
container-service/src/main/java/org/apache/hadoop/ozone/container/checksum}/package-info.java
(87%)
rename
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/{ec/reconstruction
=> common/helpers}/TokenHelper.java (88%)
copy
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/{package-info.java
=> ScanResult.java} (76%)
copy
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/{ReconstructECContainersCommandHandler.java
=> ReconcileContainerCommandHandler.java} (56%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScanError.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScanHelper.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/DataScanResult.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/MetadataScanResult.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/protocol/commands/ReconcileContainerCommand.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeTestUtils.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerMerkleTreeWriter.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestReconcileContainerTask.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestReconcileContainerCommandHandler.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestContainerCorruptions.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestContainerReconciliationWithMockDatanodes.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestDataScanResult.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestMetadataScanResult.java
create mode 100644 hadoop-hdds/docs/content/concept/OzoneS3Gateway.md
create mode 100644 hadoop-hdds/docs/content/concept/VolumesBucketsKeys.md
create mode 100644 hadoop-hdds/docs/content/design/container-reconciliation.md
create mode 100644 hadoop-hdds/docs/content/design/dn-usedspace-calculation.md
create mode 100644 hadoop-hdds/docs/content/design/full-volume-handling.md
create mode 100644 hadoop-hdds/docs/content/feature/Trash.md
create mode 100644 hadoop-hdds/docs/content/feature/multi-raft-support.md
create mode 100644 hadoop-hdds/docs/content/interface/CyberDuckOzoneS3.md
create mode 100644
hadoop-hdds/docs/content/security/protect-in-transit-traffic.md
create mode 100644 hadoop-hdds/docs/content/tools/ContainerLogParser.zh.md
copy hadoop-hdds/docs/content/{feature/_index.zh.md =>
troubleshooting/_index.md} (80%)
create mode 100644
hadoop-hdds/docs/content/troubleshooting/om-ha-snapshot-installation.md
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/DUOptimized.java
copy
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/fs/{DUFactory.java
=> DUOptimizedFactory.java} (54%)
create mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/fs/TestDUOptimized.java
copy
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/fs/{TestDUFactory.java
=> TestDUOptimizedFactory.java} (75%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/reconciliation/ReconcileContainerEventHandler.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/reconciliation/ReconciliationEligibilityHandler.java
copy
{hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/execution
=>
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/reconciliation}/package-info.java
(86%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/reconciliation/TestReconcileContainerEventHandler.java
copy
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/container/{CloseSubcommand.java
=> ReconcileSubcommand.java} (66%)
create mode 100644
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/NodeSelectionMixin.java
delete mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OzoneManagerLockUtil.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/OldSnapshotDiffJobCodecForTesting.java
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmSnapshotDiffJobCodec.java
copy hadoop-ozone/dist/src/main/compose/ozonesecure-ha/{test-leadership.sh =>
test-repair-tools.sh} (75%)
copy hadoop-ozone/dist/src/main/smoketest/compatibility/{recon.robot =>
lib.resource} (64%)
copy
hadoop-ozone/dist/src/main/smoketest/debug/{ozone-debug-dead-datanode.robot =>
block-existence-check.robot} (69%)
copy
hadoop-ozone/dist/src/main/smoketest/debug/{ozone-debug-dead-datanode.robot =>
corrupt-block-checksum.robot} (60%)
delete mode 100644
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-corrupt-block.robot
create mode 100644
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-keywords.robot
delete mode 100644
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-stale-datanode.robot
delete mode 100644
hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug-tests-ec6-3.robot
delete mode 100644 hadoop-ozone/dist/src/main/smoketest/debug/ozone-debug.robot
rename
hadoop-ozone/dist/src/main/smoketest/debug/{ozone-debug-dead-datanode.robot =>
stale-datanode-checksum.robot} (61%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/lib/BytemanLibrary.py
create mode 100644
hadoop-ozone/dist/src/main/smoketest/ozone-fi/BytemanKeywords.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/ozone-fi/README.md
create mode 100644
hadoop-ozone/dist/src/main/smoketest/ozone-fi/byteman_faults_sample.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/repair/om-compact.robot
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/checksum/TestContainerCommandReconciliation.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServletInodeBasedXfer.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOMDBCheckpointUtils.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/OMDBCheckpointUtils.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/upgrade/{InitialConstraintUpgradeAction.java
=> UnhealthyContainerReplicaMismatchAction.java} (77%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@ozone.apache.org
For additional commands, e-mail: commits-h...@ozone.apache.org
