This is an automated email from the ASF dual-hosted git repository.
aswinshakil pushed a change to branch HDDS-10239-container-reconciliation
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 9a445ed01eb HDDS-10374. Make container scanner generate merkle trees
during the scan (#7490)
add 8136119256a HDDS-13048. Create new module for Recon integration tests
(#8464)
add 21b49d32e1f HDDS-12965. Fix warnings about "used undeclared"
dependencies (#8468)
add 619c05d1ac9 HDDS-13008. Exclude same SST files when calculating full
snapdiff (#8423)
add 35e1126bffa HDDS-13042. [snapshot] Add future proofing test cases for
unsupported file system API (#8458)
add e1c779abe71 HDDS-13051. Use DatanodeID in server-scm. (#8465)
add 0cc6fcc4b0e HDDS-13043. Replace != with assertNotEquals in
TestSCMContainerPlacementRackAware (#8470)
add d1a2f48c8a7 HDDS-13057. Increment block delete processed transaction
counts regardless of log level (#8466)
add f4102380dfe HDDS-13061. Add test for key ACL operations without
permission (#8472)
add 211c76cd9ff HDDS-13060. Change NodeManager.addDatanodeCommand(..) to
use DatanodeID (#8471)
add 1825cdf6057 HDDS-13049. Deprecate VolumeName & BucketName in
OmKeyPurgeRequest and prevent Key version purge on Block Deletion Failure
(#8463)
add 7035846377f HDDS-13040. Add user doc highlighting the difference
between Ozone ACL and S3 ACL. (#8457)
add bafe6d9e446 HDDS-13064. [snapshot] Add test coverage for
SnapshotUtils.isBlockLocationInfoSame() (#8476)
add 5d1b43d44fa HDDS-13076. Refactor OzoneManagerLock class to rename
Resource class to LeveledResource (#8482)
add 6bf121e664f HDDS-13032. Support proper S3OwnerId representation (#8478)
add eeb98c7921e HDDS-13030. Snapshot Purge should unset deep cleaning flag
for next 2 snapshots in the chain (#8451)
add eb5dabd4d3e HDDS-12926. Remove *.tmp.* exclusion in DU (#8486)
add 435fe7e241f HDDS-12870. Fix listObjects corner cases (#8307)
add 75902681be0 HDDS-13079. Improve logging in DN for delete operation.
(#8489)
add e9e8b303f9d HDDS-12935. Support unsigned chunked upload with
STREAMING-UNSIGNED-PAYLOAD-TRAILER (#8366)
add a8ed19bb238 HDDS-13031. Implement a Flat Lock resource in
OzoneManagerLock (#8446)
add 5c7726a5ce6 HDDS-11539. OzoneClientCache `@PreDestroy` is never called
(#8493)
add 0171befb3d5 HDDS-13073. Set pipeline ID in checksums verifier to avoid
cached pipeline with different node (#8480)
add 3bb8858598d HDDS-12378. Change default hdds.scm.safemode.min.datanode
to 3 (#8331)
add 6ec8f85e59c HDDS-13080. Improve delete metrics to show number of
timeout DN command from SCM (#8497)
add a88ff710dde HDDS-10979. Support STANDARD_IA S3 storage class to accept
EC replication config (#8399)
add cb9867bcd31 HDDS-13065. Refactor SnapshotCache to return
AutoCloseSupplier instead of ReferenceCounted (#8473)
add 9cc41940836 HDDS-13089. [snapshot] Add an integration test to verify
snapshotted data can be read by S3 SDK client (#8495)
add 7cdc865259a HDDS-13100. ozone admin datanode list --json should output
a newline at the end (#8499)
add 87dfa5ac2db HDDS-13014. Improve PrometheusMetricsSink#normalizeName
performance (#8438)
add 54ed1158885 HDDS-9210. Update snapshot chain restore test to
incorporate snapshot delete. (#8484)
add 64bb29de1f3 HDDS-12777. Use module-specific name for generated config
files (#8475)
add 988b4479244 HDDS-5287. Document S3 ACL classes (#8501)
add 322ca93b4a6 HDDS-13025. Refactor KeyDeletingService to use
ReclaimableKeyFilter (#8450)
add 1db1cca78cf HDDS-13116. Bump jline to 3.30.3 (#8504)
add 360d13907a6 HDDS-13017. Fix warnings due to non-test scoped test
dependencies (#8479)
add 10d5555c49e HDDS-13115. Bump awssdk to 2.31.50 (#8505)
add fff80fc2d51 HDDS-13118. Remove duplicate mockito-core dependency from
hdds-test-utils (#8508)
add 4c1c6cf6302 HDDS-12714. Create acceptance test framework for debug and
repair tools (#8510)
add fd3d70c2a1a HDDS-13026. KeyDeletingService should also delete
RenameEntries (#8447)
add b8b226c3c85 HDDS-12928. datanode min free space configuration (#8388)
add 7027ab7e91a HDDS-13124. Respect config
hdds.datanode.use.datanode.hostname when reading from datanode (#8518)
add 10c470d800d HDDS-12817. Add EC block index in the ozone debug replicas
chunk-info (#8515)
add 58c87a862d4 HDDS-12177. Set runtime scope where missing (#8513)
add 08c0ab84ba5 HDDS-13075. Fix default value in description of container
placement policy configs (#8511)
add 07f48686c54 HDDS-13099. ozone admin datanode list ignores --json flag
when --id filter is used (#8500)
add 53673c564d6 HDDS-11244. OmPurgeDirectoriesRequest should clean up File
and Directory tables of AOS for deleted snapshot directories (#8509)
add 568b2286b83 HDDS-13067. Container Balancer delete commands should not
be sent with an expiration time in the past (#8491)
add 006e691245d HDDS-12503. Compact snapshot DB before evicting a snapshot
out of cache (#8141)
add 3bfb7affaf8 HDDS-13136. KeyDeleting Service should not run for already
deep cleaned snapshots (#8525)
add cc1e4d13ad9 HDDS-13156. Bump awssdk to 2.31.54 (#8536)
add 168f6901fa5 HDDS-13155. Bump jline to 3.30.4 (#8535)
add 8e552b468c1 HDDS-13157. Bump exec-maven-plugin to 3.5.1 (#8534)
add 0e8c8d4d441 HDDS-13158. Bump junit to 5.13.0 (#8537)
add 8e8c432bb7e HDDS-13071. Documentation for Container Replica Debugger
Tool (#8485)
add 6d592206a2f HDDS-13104. Move auditparser acceptance test under debug
(#8527)
add 3c25e7d6344 HDDS-12595. Add verifier for container replica states
(#8422)
add 7a7b9a83c04 HDDS-13139. Introduce bucket layout flag in freon rk
command (#8539)
add b3ff67c4d54 HDDS-13068. Validate Container Balancer move timeout and
replication timeout configs (#8490)
add 19b9b9c9366 HDDS-13166. Set pipeline ID in BlockExistenceVerifier to
avoid cached pipeline with different node (#8549)
add 46a93d0150e HDDS-12817. Addendum rename ecIndex to replicaIndex in
chunkinfo output (#8552)
add 902e7152218 HDDS-13159. Refactor KeyManagerImpl for getting deleted
subdirectories and deleted subFiles (#8538)
add 315ef204d83 HDDS-13175. Leftover reference to OM-specific trash
implementation (#8563)
add f19789dc885 HDDS-13170. Reclaimable filter should always reclaim
entries when buckets and volumes have already been deleted (#8551)
add 2360af4c5bc HDDS-13169. Intermittent failure in
testSnapshotOperationsNotBlockedDuringCompaction (#8553)
add 81fc4c4f859 HDDS-12550. Use DatanodeID instead of UUID in NodeManager
CommandQueue. (#8560)
add 2a761f79bc4 HDDS-11967. [Docs]DistCP Integration in Kerberized
environment. (#8531)
add c13a3fed9ec HDDS-13154 Link more Grafana dashboard json files to the
Observability user doc (#8533)
add f388317a6c8 HDDS-12554. Support callback on completed reconfiguration
(#8391)
add e0f08b2471c HDDS-13179. rename-generated-config fails on re-compile
without clean (#8569)
add b3d75ab801f HDDS-12976. Clean up unused dependencies (#8521)
add 81295a53cb9 HDDS-13013. [Snapshot] Add metrics and tests for snapshot
operations. (#8436)
add e1964f2eae0 HDDS-13201. Bump jersey2 to 2.47 (#8578)
add 07ee5dd531f HDDS-13204. Bump awssdk to 2.31.59 (#8582)
add ba5177ec134 HDDS-13202. Bump build-helper-maven-plugin to 3.6.1 (#8579)
add 750b629b8b1 HDDS-13203. Bump Bouncy Castle to 1.81 (#8580)
add 9e6955ee4f5 HDDS-13205. Bump common-custom-user-data-maven-extension
to 2.0.3 (#8581)
add 31d13de52a1 HDDS-13165. [Docs] Python client developer guide. (#8556)
add e936e4deb10 HDDS-12134. Implement Snapshot Cache lock for OM Bootstrap
(#8474)
add 5081ba23b41 HDDS-13034. Refactor DirectoryDeletingService to use
ReclaimableDirFilter and ReclaimableKeyFilter (#8546)
add 5e273a48da9 HDDS-12977. Fail build on dependency problems (#8574)
new 0a53c73de21 Merge branch 'master' of https://github.com/apache/ozone
into HDDS-10239-container-reconciliation
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 3 +
.github/workflows/intermittent-test-check.yml | 15 +-
.mvn/extensions.xml | 2 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 18 +-
hadoop-hdds/common/pom.xml | 69 +--
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 2 +-
.../hadoop/hdds/conf/OzoneConfiguration.java | 60 +--
.../hadoop/hdds/utils/BackgroundService.java | 43 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 2 +-
.../common/src/main/resources/ozone-default.xml | 34 +-
.../conf/TestGeneratedConfigurationOverwrite.java | 4 +-
.../hadoop/hdds/protocol/MockDatanodeDetails.java | 31 +-
hadoop-hdds/config/pom.xml | 3 +
.../hadoop/hdds/conf/ConfigFileGenerator.java | 14 +-
hadoop-hdds/container-service/pom.xml | 26 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 2 +
.../common/statemachine/DatanodeConfiguration.java | 49 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 32 +-
.../protocol/commands/CommandForDatanode.java | 6 +-
.../statemachine/TestDatanodeConfiguration.java | 25 +-
.../common/volume/TestReservedVolumeSpace.java | 9 +-
.../replication/TestReplicationSupervisor.java | 4 +-
.../content/concept/StorageContainerManager.md | 4 +-
.../content/concept/StorageContainerManager.zh.md | 2 +-
.../content/design/dn-min-space-configuration.md | 108 ++++
hadoop-hdds/docs/content/feature/Observability.md | 2 +-
hadoop-hdds/docs/content/integration/DistCp.md | 75 +++
hadoop-hdds/docs/content/interface/Ofs.md | 2 +-
hadoop-hdds/docs/content/interface/Ofs.zh.md | 2 +-
hadoop-hdds/docs/content/interface/Python.md | 239 +++++++++
hadoop-hdds/docs/content/recipe/Boto3Tutorial.md | 123 +++++
hadoop-hdds/docs/content/recipe/PyArrowTutorial.md | 141 ++++++
.../content/recipe/PythonRequestsOzoneHttpFS.md | 170 +++++++
hadoop-hdds/docs/content/security/SecurityAcls.md | 13 +
.../docs/content/tools/ContainerLogParser.md | 251 ++++++++++
hadoop-hdds/erasurecode/pom.xml | 5 +
hadoop-hdds/framework/pom.xml | 53 +-
.../hadoop/hdds/conf/ReconfigurableBase.java | 202 ++++++++
.../hdds/conf/ReconfigurationChangeCallback.java | 14 +-
.../hadoop/hdds/conf/ReconfigurationHandler.java | 54 +-
.../java/org/apache/hadoop/hdds/fs/DUFactory.java | 3 +-
.../hdds/utils/PrometheusMetricsSinkUtil.java | 54 +-
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 18 +
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 16 +
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 17 +
hadoop-hdds/interface-client/pom.xml | 10 +-
hadoop-hdds/interface-server/pom.xml | 6 +
hadoop-hdds/managed-rocksdb/pom.xml | 8 +
hadoop-hdds/rocks-native/pom.xml | 9 +-
hadoop-hdds/rocksdb-checkpoint-differ/pom.xml | 25 +-
.../org/apache/ozone/rocksdb/util/RdbUtil.java | 19 +
hadoop-hdds/server-scm/pom.xml | 19 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 34 +-
.../hdds/scm/block/SCMBlockDeletingService.java | 2 +-
.../SCMDeletedBlockTransactionStatusManager.java | 111 ++---
.../scm/block/ScmBlockDeletingServiceMetrics.java | 27 +-
.../balancer/AbstractFindTargetGreedy.java | 8 +-
.../scm/container/balancer/ContainerBalancer.java | 19 +
.../scm/container/balancer/FindSourceGreedy.java | 8 +-
.../hdds/scm/container/balancer/MoveManager.java | 88 ++--
.../ContainerPlacementPolicyFactory.java | 3 +-
.../container/replication/ReplicationManager.java | 2 +-
.../apache/hadoop/hdds/scm/node/CommandQueue.java | 31 +-
.../hadoop/hdds/scm/node/DatanodeUsageInfo.java | 5 +
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 2 +-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 16 +-
.../hadoop/hdds/scm/node/SCMNodeManager.java | 25 +-
.../hdds/scm/pipeline/PipelineManagerImpl.java | 27 +-
.../hdds/scm/safemode/DataNodeSafeModeRule.java | 7 +-
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 14 +-
.../hdds/scm/server/StorageContainerManager.java | 2 +
.../org/apache/hadoop/hdds/scm/HddsTestUtils.java | 2 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 40 +-
.../scm/block/TestSCMBlockDeletingService.java | 13 +-
.../TestSCMDeleteBlocksCommandStatusManager.java | 26 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 23 +-
.../hdds/scm/container/SimpleMockNodeManager.java | 7 +-
.../container/TestCloseContainerEventHandler.java | 6 +-
.../container/balancer/TestContainerBalancer.java | 19 +
.../scm/container/balancer/TestMoveManager.java | 51 ++
.../TestSCMContainerPlacementRackAware.java | 50 +-
.../TestReconcileContainerEventHandler.java | 18 +-
.../replication/TestReplicationManager.java | 26 +-
.../TestReplicationManagerScenarios.java | 11 +-
.../hadoop/hdds/scm/node/TestCommandQueue.java | 56 +--
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 4 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 45 +-
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 23 +-
.../scm/pipeline/TestPipelinePlacementFactory.java | 21 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 1 +
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 1 +
.../server/TestSCMDatanodeHeartbeatDispatcher.java | 4 +-
hadoop-hdds/test-utils/pom.xml | 6 +-
hadoop-ozone/cli-admin/pom.xml | 1 +
.../hdds/scm/cli/datanode/ListInfoSubcommand.java | 11 +-
.../scm/cli/datanode/TestListInfoSubcommand.java | 83 ++++
hadoop-ozone/cli-shell/pom.xml | 15 +-
.../hadoop/ozone/shell/ReplicationOptions.java | 2 +-
.../ozone/shell/keys/ChecksumKeyHandler.java | 2 +-
hadoop-ozone/client/pom.xml | 33 +-
.../hadoop/ozone/client}/OzoneClientUtils.java | 5 +-
.../hadoop/ozone/client}/TestOzoneClientUtils.java | 4 +-
hadoop-ozone/common/pom.xml | 18 +-
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 20 +
.../hadoop/ozone/om/S3SecretLockedManager.java | 2 +-
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 4 +-
.../hadoop/ozone/om/lock/IOzoneManagerLock.java | 108 +++-
.../hadoop/ozone/om/lock/OmReadOnlyLock.java | 11 +-
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 408 ++++++++-------
.../hadoop/ozone/om/lock/OzoneManagerLockUtil.java | 12 +-
.../request/validation/RegisterValidator.java | 2 +-
.../ozone/om/helpers/TestOmSnapshotInfo.java | 8 +-
.../hadoop/ozone/om/lock/TestKeyPathLock.java | 20 +-
.../hadoop/ozone/om/lock/TestOzoneManagerLock.java | 189 +++++--
hadoop-ozone/csi/pom.xml | 82 ++--
hadoop-ozone/datanode/pom.xml | 10 +-
hadoop-ozone/dev-support/checks/junit.sh | 4 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dist/pom.xml | 20 +
.../dist/src/main/compose/common/ec-test.sh | 2 +
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 -
.../src/main/compose/ozonescripts/docker-config | 1 +
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../main/compose/ozonesecure-ha/debug-tools.yaml | 162 ++++++
.../test.sh => ozonesecure-ha/test-debug-tools.sh} | 44 +-
.../src/main/k8s/definitions/ozone/config.yaml | 1 +
.../examples/getting-started/config-configmap.yaml | 1 +
.../k8s/examples/minikube/config-configmap.yaml | 1 +
.../k8s/examples/ozone-dev/config-configmap.yaml | 1 +
.../k8s/examples/ozone-ha/config-configmap.yaml | 1 +
.../main/k8s/examples/ozone/config-configmap.yaml | 1 +
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 4 -
hadoop-ozone/dist/src/main/license/jar-report.txt | 5 -
.../{auditparser => debug}/auditparser.robot | 6 +-
.../smoketest/debug/ozone-debug-tests-ec3-2.robot | 9 +-
.../smoketest/debug/ozone-debug-tests-ec6-3.robot | 5 +-
.../main/smoketest/debug/ozone-debug-tests.robot | 8 +-
.../src/main/smoketest/debug/ozone-debug.robot | 5 +-
.../src/main/smoketest/ec/awss3ecstorage.robot | 90 ++++
.../main/smoketest/httpfs/operations_tests.robot | 1 +
.../dist/src/main/smoketest/s3/awss3.robot | 8 +
.../dist/src/main/smoketest/s3/bucketlist.robot | 3 +-
.../dist/src/main/smoketest/s3/objectlist.robot | 66 +++
.../src/main/smoketest/s3/s3_compatbility_check.sh | 1 +
.../fault-injection-test/mini-chaos-tests/pom.xml | 55 ++-
hadoop-ozone/freon/pom.xml | 11 +-
.../hadoop/ozone/freon/RandomKeyGenerator.java | 30 +-
hadoop-ozone/httpfsgateway/pom.xml | 63 ++-
.../fs/http/server/HttpFSServerWebServer.java | 5 +-
hadoop-ozone/insight/pom.xml | 42 +-
.../ozone/insight/TestConfigurationSubCommand.java | 53 +-
.../dev-support/findbugsExcludeFile.xml | 0
.../pom.xml | 110 ++++-
.../hadoop/ozone/StandardOutputTestBase.java | 0
.../apache/hadoop/ozone/recon/ReconService.java | 1 -
.../hadoop/ozone/recon}/TestNSSummaryAdmin.java | 3 +-
.../ozone/recon/TestReconAndAdminContainerCLI.java | 0
.../hadoop/ozone/recon/TestReconAsPassiveScm.java | 0
.../ozone/recon/TestReconContainerEndpoint.java | 0
.../hadoop/ozone/recon/TestReconEndpointUtil.java | 0
.../TestReconInsightsForDeletedDirectories.java | 0
.../hadoop/ozone/recon/TestReconScmSnapshot.java | 0
.../apache/hadoop/ozone/recon/TestReconTasks.java | 0
.../ozone/recon/TestReconWithOzoneManager.java | 0
.../ozone/recon/TestReconWithOzoneManagerFSO.java | 0
.../ozone/recon/TestReconWithOzoneManagerHA.java | 0
.../src/test/resources/ozone-site.xml | 20 +-
hadoop-ozone/integration-test-s3/pom.xml | 30 ++
.../apache/hadoop/ozone/s3/S3GatewayService.java | 2 -
.../ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | 102 +++-
.../ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java | 138 +++++-
.../src/test/resources/ozone-site.xml | 5 +-
hadoop-ozone/integration-test/pom.xml | 337 +++++++++++--
.../fs/ozone/AbstractOzoneFileSystemTest.java | 3 +-
.../ozone/AbstractRootedOzoneFileSystemTest.java | 3 +-
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 28 +-
.../hdds/scm/TestStorageContainerManager.java | 13 +-
.../ozone/TestContainerBalancerOperations.java | 4 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 1 +
.../rpc/TestContainerStateMachineFailures.java | 2 +-
.../TestCloseContainerByPipeline.java | 8 +-
.../commandhandler/TestCloseContainerHandler.java | 2 +-
.../commandhandler/TestDeleteContainerHandler.java | 26 +-
.../TestRefreshVolumeUsageHandler.java | 2 +
.../ozoneimpl/TestOzoneContainerWithTLS.java | 3 +-
.../hadoop/ozone/freon/TestOMSnapshotDAG.java | 8 +-
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 23 +
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 2 +-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 56 +++
.../ozone/om/TestOmContainerLocationCache.java | 12 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 89 ++++
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 101 ++++
.../om/snapshot/TestOzoneManagerHASnapshot.java | 61 ++-
.../snapshot/TestSnapshotBackgroundServices.java | 5 +-
...TestSnapshotDeletingServiceIntegrationTest.java | 104 +++-
.../TestSnapshotDirectoryCleaningService.java | 31 +-
.../ozone/reconfig/TestOmReconfiguration.java | 2 +
.../hadoop/ozone/shell/TestOzoneDebugShell.java | 3 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 10 +-
.../hadoop/ozone/shell/TestReconfigShell.java | 55 ++-
.../src/test/resources/ozone-site.xml | 12 +
hadoop-ozone/interface-client/pom.xml | 24 +-
.../src/main/proto/OmClientProtocol.proto | 5 +-
hadoop-ozone/interface-storage/pom.xml | 38 +-
hadoop-ozone/mini-cluster/pom.xml | 16 +-
hadoop-ozone/ozone-manager/pom.xml | 26 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 2 +-
.../hadoop/ozone/om/DeletingServiceMetrics.java | 6 +
.../org/apache/hadoop/ozone/om/KeyManager.java | 77 ++-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 289 +++++------
.../org/apache/hadoop/ozone/om/ListIterator.java | 2 +-
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 2 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 30 ++
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 156 ------
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 23 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 69 ++-
.../hadoop/ozone/om/PendingKeysDeletion.java | 18 +-
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 2 +-
.../hadoop/ozone/om/SstFilteringService.java | 12 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 7 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 4 +-
.../ozone/om/lock/OBSKeyPathLockStrategy.java | 4 +-
.../ozone/om/lock/RegularBucketLockStrategy.java | 2 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 6 +-
.../om/request/bucket/OMBucketCreateRequest.java | 4 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 4 +-
.../om/request/bucket/OMBucketSetOwnerRequest.java | 2 +-
.../request/bucket/OMBucketSetPropertyRequest.java | 2 +-
.../om/request/bucket/acl/OMBucketAclRequest.java | 2 +-
.../om/request/file/OMDirectoryCreateRequest.java | 2 +-
.../file/OMDirectoryCreateRequestWithFSO.java | 2 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 2 +-
.../request/file/OMFileCreateRequestWithFSO.java | 2 +-
.../ozone/om/request/file/OMFileRequest.java | 16 +
.../om/request/file/OMRecoverLeaseRequest.java | 2 +-
.../om/request/key/OMAllocateBlockRequest.java | 2 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 2 +-
.../key/OMDirectoriesPurgeRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 2 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 2 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 2 +-
.../om/request/key/OMKeyDeleteRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyPurgeRequest.java | 6 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 2 +-
.../om/request/key/OMKeyRenameRequestWithFSO.java | 2 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 2 +-
.../ozone/om/request/key/OMKeySetTimesRequest.java | 2 +-
.../request/key/OMKeySetTimesRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 2 +-
.../ozone/om/request/key/OMKeysRenameRequest.java | 2 +-
.../om/request/key/OMOpenKeysDeleteRequest.java | 2 +-
.../ozone/om/request/key/acl/OMKeyAclRequest.java | 2 +-
.../om/request/key/acl/OMKeyAclRequestWithFSO.java | 2 +-
.../request/key/acl/prefix/OMPrefixAclRequest.java | 2 +-
.../S3ExpiredMultipartUploadsAbortRequest.java | 2 +-
.../S3InitiateMultipartUploadRequest.java | 2 +-
.../S3InitiateMultipartUploadRequestWithFSO.java | 2 +-
.../multipart/S3MultipartUploadAbortRequest.java | 2 +-
.../S3MultipartUploadCommitPartRequest.java | 2 +-
.../S3MultipartUploadCompleteRequest.java | 2 +-
.../s3/tagging/S3DeleteObjectTaggingRequest.java | 2 +-
.../S3DeleteObjectTaggingRequestWithFSO.java | 2 +-
.../s3/tagging/S3PutObjectTaggingRequest.java | 2 +-
.../tagging/S3PutObjectTaggingRequestWithFSO.java | 2 +-
.../s3/tenant/OMTenantAssignAdminRequest.java | 2 +-
.../tenant/OMTenantAssignUserAccessIdRequest.java | 2 +-
.../request/s3/tenant/OMTenantCreateRequest.java | 4 +-
.../request/s3/tenant/OMTenantDeleteRequest.java | 2 +-
.../s3/tenant/OMTenantRevokeAdminRequest.java | 2 +-
.../tenant/OMTenantRevokeUserAccessIdRequest.java | 2 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 4 +-
.../request/snapshot/OMSnapshotDeleteRequest.java | 4 +-
.../request/snapshot/OMSnapshotPurgeRequest.java | 7 +-
.../request/snapshot/OMSnapshotRenameRequest.java | 11 +-
.../om/request/volume/OMQuotaRepairRequest.java | 4 +-
.../om/request/volume/OMVolumeCreateRequest.java | 4 +-
.../om/request/volume/OMVolumeDeleteRequest.java | 4 +-
.../om/request/volume/OMVolumeSetOwnerRequest.java | 2 +-
.../om/request/volume/OMVolumeSetQuotaRequest.java | 2 +-
.../om/request/volume/acl/OMVolumeAclRequest.java | 2 +-
.../key/OMDirectoriesPurgeResponseWithFSO.java | 39 +-
.../ozone/om/response/key/OMKeyPurgeResponse.java | 17 +-
.../OMSnapshotMoveDeletedKeysResponse.java | 8 +-
.../snapshot/OMSnapshotMoveTableKeysResponse.java | 6 +-
.../response/snapshot/OMSnapshotPurgeResponse.java | 8 +-
.../om/service/AbstractKeyDeletingService.java | 192 ++++----
.../ozone/om/service/DirectoryDeletingService.java | 518 ++++++++++++-------
.../ozone/om/service/KeyDeletingService.java | 546 +++++++--------------
.../ozone/om/service/SnapshotDeletingService.java | 8 +-
.../service/SnapshotDirectoryCleaningService.java | 10 +-
.../ozone/om/snapshot/MultiSnapshotLocks.java | 6 +-
.../hadoop/ozone/om/snapshot/ReferenceCounted.java | 12 +-
.../hadoop/ozone/om/snapshot/SnapshotCache.java | 134 ++++-
.../ozone/om/snapshot/SnapshotDiffManager.java | 43 +-
.../hadoop/ozone/om/snapshot/SnapshotUtils.java | 7 +-
.../om/snapshot/filter/ReclaimableDirFilter.java | 4 +-
.../om/snapshot/filter/ReclaimableFilter.java | 36 +-
.../om/snapshot/filter/ReclaimableKeyFilter.java | 6 +-
.../filter/ReclaimableRenameEntryFilter.java | 4 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 32 +-
.../ozone/om/request/OMRequestTestUtils.java | 37 +-
.../TestOMDirectoriesPurgeRequestAndResponse.java | 127 ++++-
.../key/TestOMKeyPurgeRequestAndResponse.java | 63 ++-
.../ozone/om/request/key/TestOMKeyRequest.java | 26 +-
.../s3/multipart/TestS3MultipartRequest.java | 6 +-
.../TestOMSnapshotPurgeRequestAndResponse.java | 30 +-
.../TestOMSnapshotMoveTableKeysResponse.java | 8 +-
.../ozone/om/service/TestKeyDeletingService.java | 228 ++++++++-
.../ozone/om/snapshot/TestMultiSnapshotLocks.java | 2 +-
.../ozone/om/snapshot/TestSnapshotCache.java | 190 ++++++-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 37 +-
.../ozone/om/snapshot/TestSnapshotUtils.java | 172 +++++++
.../ozone/om/snapshot/TestSstFilteringService.java | 3 +-
.../filter/AbstractReclaimableFilterTest.java | 39 +-
.../snapshot/filter/TestReclaimableDirFilter.java | 8 +-
.../om/snapshot/filter/TestReclaimableFilter.java | 34 +-
.../snapshot/filter/TestReclaimableKeyFilter.java | 10 +-
.../filter/TestReclaimableRenameEntryFilter.java | 8 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 1 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 1 +
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 1 +
.../fs/ozone/BasicRootedOzoneFileSystem.java | 1 +
hadoop-ozone/ozonefs-hadoop2/pom.xml | 1 +
hadoop-ozone/ozonefs-hadoop3-client/pom.xml | 3 -
hadoop-ozone/ozonefs-hadoop3/pom.xml | 1 +
hadoop-ozone/ozonefs-shaded/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 4 -
hadoop-ozone/pom.xml | 1 +
hadoop-ozone/recon-codegen/pom.xml | 49 +-
hadoop-ozone/recon/pom.xml | 95 ++--
.../hadoop/ozone/recon/ReconControllerModule.java | 1 -
.../hadoop/ozone/recon}/ReconSqlDbConfig.java | 2 +-
.../ozone/recon/scm/TestReconNodeManager.java | 4 +-
hadoop-ozone/s3-secret-store/pom.xml | 4 +
hadoop-ozone/s3gateway/pom.xml | 111 ++---
.../apache/hadoop/ozone/s3/OzoneClientCache.java | 80 +--
.../hadoop/ozone/s3/OzoneClientProducer.java | 37 +-
.../hadoop/ozone/s3/S3GatewayConfigKeys.java | 4 +
.../hadoop/ozone/s3/SignedChunksInputStream.java | 86 +++-
...tStream.java => UnsignedChunksInputStream.java} | 115 +++--
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 46 +-
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 3 +-
.../s3/endpoint/ListMultipartUploadsResult.java | 4 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 121 +++--
.../hadoop/ozone/s3/endpoint/RootEndpoint.java | 2 +-
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 4 +-
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 13 +-
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 14 +-
.../hadoop/ozone/s3/exception/S3ErrorTable.java | 6 +
.../ozone/s3/signature/StringToSignProducer.java | 55 ++-
.../org/apache/hadoop/ozone/s3/util/S3Consts.java | 8 +-
.../apache/hadoop/ozone/s3/util/S3StorageType.java | 57 +--
.../org/apache/hadoop/ozone/s3/util/S3Utils.java | 175 ++++---
.../ozone/s3secret/S3SecretEndpointBase.java | 22 +-
.../ozone/s3secret/S3SecretManagementEndpoint.java | 7 +
...ientProducer.java => TestOzoneClientCache.java} | 36 +-
.../ozone/s3/TestSignedChunksInputStream.java | 231 ++++++---
.../ozone/s3/TestUnsignedChunkInputStream.java | 223 +++++++++
.../ozone/s3/endpoint/BucketEndpointBuilder.java} | 19 +-
.../hadoop/ozone/s3/endpoint/EndpointBuilder.java | 2 +-
.../hadoop/ozone/s3/endpoint/TestBucketList.java | 58 +++
.../hadoop/ozone/s3/endpoint/TestListParts.java | 3 +
.../s3/endpoint/TestMultipartUploadComplete.java | 3 +
.../s3/endpoint/TestMultipartUploadWithCopy.java | 5 +
.../hadoop/ozone/s3/endpoint/TestObjectGet.java | 2 +
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 13 +-
.../ozone/s3/endpoint/TestObjectTaggingDelete.java | 3 +
.../ozone/s3/endpoint/TestObjectTaggingGet.java | 3 +
.../ozone/s3/endpoint/TestObjectTaggingPut.java | 4 +-
.../hadoop/ozone/s3/endpoint/TestPartUpload.java | 7 +
.../s3/endpoint/TestPartUploadWithStream.java | 3 +
.../ozone/s3/endpoint/TestPermissionCheck.java | 3 +
.../hadoop/ozone/s3/endpoint/TestRootList.java | 2 +-
.../ozone/s3/endpoint/TestUploadWithStream.java | 2 +
.../ozone/s3/metrics/TestS3GatewayMetrics.java | 3 +
.../apache/hadoop/ozone/s3/util/TestS3Utils.java | 184 +++----
.../hadoop/ozone/s3secret/TestSecretGenerate.java | 2 +-
.../hadoop/ozone/s3secret/TestSecretRevoke.java | 4 +-
hadoop-ozone/tools/pom.xml | 99 ++--
.../debug/replicas/BlockExistenceVerifier.java | 1 +
.../ozone/debug/replicas/ChecksumVerifier.java | 1 +
.../debug/replicas/ContainerStateVerifier.java | 194 ++++++++
.../ozone/debug/replicas/ReplicasVerify.java | 18 +
.../debug/replicas/chunk/ChunkKeyHandler.java | 17 +-
pom.xml | 51 +-
386 files changed, 8596 insertions(+), 3571 deletions(-)
create mode 100644
hadoop-hdds/docs/content/design/dn-min-space-configuration.md
create mode 100644 hadoop-hdds/docs/content/interface/Python.md
create mode 100644 hadoop-hdds/docs/content/recipe/Boto3Tutorial.md
create mode 100644 hadoop-hdds/docs/content/recipe/PyArrowTutorial.md
create mode 100644 hadoop-hdds/docs/content/recipe/PythonRequestsOzoneHttpFS.md
create mode 100644 hadoop-hdds/docs/content/tools/ContainerLogParser.md
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/ReconfigurableBase.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/contract/TestRootedOzoneContract.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/conf/ReconfigurationChangeCallback.java
(73%)
rename hadoop-ozone/{ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone
=> client/src/main/java/org/apache/hadoop/ozone/client}/OzoneClientUtils.java
(98%)
rename hadoop-ozone/{ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone
=>
client/src/test/java/org/apache/hadoop/ozone/client}/TestOzoneClientUtils.java
(98%)
create mode 100644
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/debug-tools.yaml
copy hadoop-ozone/dist/src/main/compose/{ozone-om-prepare/test.sh =>
ozonesecure-ha/test-debug-tools.sh} (63%)
mode change 100755 => 100644
rename hadoop-ozone/dist/src/main/smoketest/{auditparser =>
debug}/auditparser.robot (94%)
create mode 100644 hadoop-ozone/dist/src/main/smoketest/ec/awss3ecstorage.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/s3/objectlist.robot
copy hadoop-ozone/{cli-shell =>
integration-test-recon}/dev-support/findbugsExcludeFile.xml (100%)
copy hadoop-ozone/{mini-cluster => integration-test-recon}/pom.xml (58%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/StandardOutputTestBase.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/ReconService.java
(98%)
rename
hadoop-ozone/{integration-test/src/test/java/org/apache/hadoop/ozone/shell =>
integration-test-recon/src/test/java/org/apache/hadoop/ozone/recon}/TestNSSummaryAdmin.java
(98%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconAsPassiveScm.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconContainerEndpoint.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconEndpointUtil.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconInsightsForDeletedDirectories.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconScmSnapshot.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconTasks.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconWithOzoneManager.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconWithOzoneManagerFSO.java
(100%)
rename hadoop-ozone/{integration-test =>
integration-test-recon}/src/test/java/org/apache/hadoop/ozone/recon/TestReconWithOzoneManagerHA.java
(100%)
copy hadoop-ozone/{integration-test-s3 =>
integration-test-recon}/src/test/resources/ozone-site.xml (97%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotUtils.java
rename hadoop-ozone/{recon-codegen/src/main/java/org/apache/ozone/recon/schema
=> recon/src/main/java/org/apache/hadoop/ozone/recon}/ReconSqlDbConfig.java
(99%)
copy
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/{SignedChunksInputStream.java
=> UnsignedChunksInputStream.java} (51%)
rename
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/{TestOzoneClientProducer.java
=> TestOzoneClientCache.java} (70%)
create mode 100644
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestUnsignedChunkInputStream.java
copy
hadoop-ozone/s3gateway/src/{main/java/org/apache/hadoop/ozone/s3/endpoint/PutBucketAclRequestUnmarshaller.java
=> test/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpointBuilder.java}
(71%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/replicas/ContainerStateVerifier.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
