This is an automated email from the ASF dual-hosted git repository.
pifta pushed a change to branch HDDS-5447-httpfs
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 121b54b HDDS-5520 Add lifecycle management to HttpFS server (#2552)
add b447ffc HDDS-5401. Add more metrics to ReplicationManager to help
monitor replication progress. (#2382)
add 622275f HDDS-5513. Race condition upon dn restart at prefinalization.
(#2471)
add 99fa944 HDDS-5453. Add metrics in SCM block deleting service. (#2438)
add d557817 HDDS-5524. Use try-with-resources to properly close resources
in DirectoryServerSource (#2481)
add 8739fd2 HDDS-5512. Intermittent failure in
TestSCMInstallSnapshotWithHA (#2473)
add d0f0f80 HDDS-5501. Support to upload/read keys from encrypted buckets
through S3G (#2467)
add 01c6165 HDDS-5511. ContainerStateMap should handle ecReplication
config map (#2470)
add 7df1925 HDDS-5362. [FSO] Support bucket layouts in OM (#2357)
add 0b5e965 HDDS-5302. CleanUp V1 Code of DeletedBlockLogImpl (#2465)
add e2d1727 HDDS-5472. Old versions of location in OmKeyLocationInfoGroup
causes OOM of OM (#2448)
add fd7e850 HDDS-5460: ReplicationConfig#getDefault is hardcoded with
RatisReplicationConfig (#2430)
add ec031d47 HDDS-5515. SKip update firstUpgradeActionLayoutVersion when
on-disk layout version is up-to-date. (#2472)
add cead31e HDDS-5530. Support links to encrypted buckets. (#2487)
add b3a2a3e HDDS-5441. Disallow same set of DNs to be part of multiple
pipelines. (#2416)
add 30287c7 HDDS-5539. Fix actual value in assertion in TestRDBStore
(#2494)
add 00a5afe HDDS-5354. Allow to restrict available ReplicationConfig
(#2345)
add 4503249 HDDS-5532. Missing integration test cleanup (#2496)
add 181d44a HDDS-5527. Move tests back to root partition (#2486)
add 61396ba HDDS-4668. Intermittent failure in TestOMRatisSnapshots
(#2474)
add 94b8c4b HDDS-5516. Duplicate metrics registered while running
checkScmHA upon scm startup. (#2475)
add e23fb3c HDDS-5473. Bump jetty version to 9.4.43.v20210629 (#2449)
add bc1dfb8 HDDS-5538. Uncaught NPE in GrpcReplicationService. (#2503)
add 063e7fd HDDS-5390. reconPipelineReportHandler should not retry when
pipeline not found (#2371)
add 2020d30 HDDS-5503. On finalize upgrade actions not running. (#2476)
add d51bd4b HDDS-5021. dev-support Dockerfile is badly outdated (#2480)
add f9ec7e9 HDDS-5360. DN failed to process all delete block commands in
one heartbeat interval (#2420)
add 2edb375 HDDS-5546.OM Service ID change causes OM startup failure.
(#2512)
add f922b0d HDDS-5555. remove pipeline manager v1 code (#2511)
add fa7dc30 HDDS-5549. Add 1.1 to supported versions in security policy
(#2519)
add 7eed08f HDDS-5531. For Link Buckets avoid showing metadata. (#2502)
add 1c96a5b HDDS-5558. vUnit invocation unit() may produce NPE (#2513)
add 4625061 HDDS-5449 Recon namespace summary 'du' information should
return replicated size of a key (#2489)
add dc9bb4e HDDS-5283. getStorageSize cast to int can cause issue (#2303)
add a956b49 HDDS-4929. Select target datanodes and containers to move for
Container Balancer (#2441)
add ee929dc HDDS-5000. Run CI checks selectively (#2479)
add 43554e9 HDDS-5608. Fix wrong command in ugrade doc (#2524)
add 44be763 HDDS-5358. Incorrect cache entry invalidation causes
intermittent failure in testGetS3SecretAndRevokeS3Secret (#2518)
add 1045ffc HDDS-5498. Containers are always quasi closed during
finalization. (#2515)
add cef8bf5 HDDS-5508. Add documentation regarding access encrypted
buckets from S3G (#2504)
add 66ee3b4 HDDS-5618. feature/_index.zh.md retranslation (#2535)
add 05c7a98 HDDS-5253. Support container move HA (#2488)
add c9c715c HDDS-5603. Stop ContainerBalancer when SCM is stopped. (#2523)
add 2b70185 HDDS-5533. Update Documentation for Trash (#2505)
add 375f3c4 HDDS-5483. Validate block file length during put block (#2463)
add 8b28e46 HDDS-5458. Extend s3 compatibility script, and fix the
related errors (#2482)
add 8088cf8 HDDS-5601. Publish upgrade documentation on Ozone website.
(#2527)
add bbf9acf HDDS-5054. Merge SCM HA configs to ScmConfigKeys. (#2506)
add d963991 HDDS-5624. Disable test
TestPipelineClose.testPipelineCloseWithLogFailure (#2546)
add 75532de HDDS-5319. Ignore flaky tests in
TestSCMUpdateServiceGrpcServer
add c654ed8 HDDS-5606. Intermittent failure in
TestBlockDeletion#testContainerStatisticsAfterDelete (#2528)
add 800d9a9 HDDS-5175. ozone fs trash service does not handle bucket -rm
-R (#2458)
add bf9fc53 HDDS-5622. Some improvements in SCMCommonPlacementPolicy
(#2543)
add f4ef0e2 HDDS-5621. Support returning nodes of a level in
NetworkTopology (#2541)
add b175d0b HDDS-5437. Move dead DN out of topology (#2435)
add 3accc98 HDDS-5616. Document how to run ozone sample cluster with
FSO-enabled (#2539)
add a595d06 Revert "HDDS-5483. Validate block file length during put
block (#2463)"
add 04816a6 HDDS-5620. Fix and extend hadoop.http.idle_timeout.ms to 60k
(#2540)
add 18aad55 HDDS-5377. Add a Namespace Summary tab to Recon Web UI (#2530)
add db574d1 HDDS-5632. Intermittent failure in
TestOzoneManagerBootstrap#testBootstrapTwoNewOMs (#2550)
add 52ec27c HDDS-5610. Remove or clarify doc regarding OM federation
(#2529)
add 541a0dc HDDS-5640. Add scm ratis metrics (#2553)
add d90bc30 HDDS-5495. Ozone version mismatch in Kubernetes test lib
(#2544)
add 33fcd77 HDDS-5556. GrpcReplication Client might fail in SCM HA
Cluster (#2556)
add 1d8bfe0 HDDS-5540. Acceptance test: able to generate result in
xunit/junit format as well (#2514)
add d3f223c HDDS-5658. Handle InterruptedException in
DeleteBlocksCommandHandler and ContainerBalancer (#2560)
add b1eaf8f HDDS-5393. Return latest version of key location for client
on createKey/createFile (#2432)
add 6ce3036 HDDS-5554. Option to disable checksum verification. (#2522)
add f92ee75 HDDS-5619. Ozone data corruption issue on follower node.
(#2538)
add 03b0b5b HDDS-5391 Fix issues in SCMHAConfiguration (#2567)
add 318456d HDDS-5637. Add documentation for Recon Namespace REST APIs
(#2566)
add 52e9bb5 HDDS-5655. SCM terminates when allocatecontainer happens on
closed pipeline. (#2569)
add fdbf8c3 HDDS-5668. Fix TestOzoneConfigurationFields. (#2576)
add af6c018 HDDS-5637. Addendum: Add documentation for Recon Namespace
REST APIs (#2579)
add 5b4b21a HDDS-5521. Use short principal name as owner of a volume.
(#2573)
add 25561f8 HDDS-5623. Add data validation checks on datanode. (#2542)
add e12a6b9 HDDS-5611. Fixed NullPointerException in
ContainerStateMachine during Pipeline Close. (#2548)
add 30859fe HDDS-5687. support Optional<T> as parameters of commandLine
(#2588)
add 7dcaad1 HDDS-5681. Speed up TestOzoneManagerHAWithACL (#2587)
add b593fa8 HDDS-5688. Rpc should not retry if exception is
ContainerNotFoundException. (#2589)
add db75046 HDDS-5662. Fix blank traceId for FsShell command (#2568)
add 2958ab3 HDDS-5535. SCM-HA documents configuration error (#2492)
add 33b3f9e HDDS-5669. Fix NSSummaryEndpoint's NPE bug during OM
initialization (#2592)
add aab19a6 HDDS-5368. Add CLI command: ozone admin namespace summary
(#2565)
add 9baca7c HDDS-5692. Combine some tests in TestOzoneManagerHAWithData
to reuse mini-Clusters (#2593)
add 758b8ca HDDS-5689. chooseRandom in NetworkTopology should exclude the
affinity node too (#2590)
add 194de5b HDDS-5680. Fix bug for incomplete SCM ratis log (#2586)
add cdac8bc HDDS-5644. Speed up decommission tests using a background
Mini Cluster provider (#2554)
add fe42640 HDDS-5667. documentation page layout (#2604)
add 730cb66 HDDS-5706. Fix ReplicationManager zero metrics for inflight
actions. (#2605)
add ea1ff1e HDDS-5690. Speed up TestContainerReplication by removing
testSkipDemmissionAndMaintenanceNode (#2591)
add f71474a HDDS-5700. Improve LOG message of decommission progress.
(#2598)
add aa8cfcd HDDS-5709. do not call removeTransactionsFromDB if nothing to
remove (#2608)
new 94d1673 Merge branch 'master' into HDDS-5447-httpfs
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/post-commit.yml | 130 ++--
SECURITY.md | 1 +
dev-support/ci/lib/_all_libs.sh | 34 +
dev-support/ci/lib/_initialization.sh | 140 ++++
dev-support/ci/lib/_script_init.sh | 42 ++
dev-support/ci/lib/_start_end.sh | 112 +++
dev-support/ci/lib/_traps.sh | 38 +
dev-support/ci/lib/_verbosity.sh | 57 ++
dev-support/ci/selective_ci_checks.bats | 210 ++++++
dev-support/ci/selective_ci_checks.sh | 522 +++++++++++++
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 5 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 10 +
.../hadoop/hdds/client/ReplicationConfig.java | 10 +-
.../hdds/client/ReplicationConfigValidator.java | 63 ++
.../apache/hadoop/hdds/recon/ReconConfigKeys.java | 10 +
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 85 +++
.../apache/hadoop/hdds/scm/client/ScmClient.java | 4 +-
.../hadoop/hdds/scm/ha/SCMHAConfiguration.java | 320 --------
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 8 +-
.../org/apache/hadoop/hdds/scm/net/InnerNode.java | 7 +
.../apache/hadoop/hdds/scm/net/InnerNodeImpl.java | 27 +
.../hadoop/hdds/scm/net/NetworkTopology.java | 7 +
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 41 +-
.../protocol/StorageContainerLocationProtocol.java | 11 +-
.../ozone/upgrade/BasicUpgradeFinalizer.java | 18 +-
.../common/src/main/resources/ozone-default.xml | 165 +++++
.../hadoop/hdds/client/TestReplicationConfig.java | 24 +
.../client/TestReplicationConfigValidator.java | 73 ++
.../hdds/scm/net/TestNetworkTopologyImpl.java | 59 +-
.../apache/hadoop/hdds/conf/TimeDurationUtil.java | 3 +
.../container/common/helpers/ContainerUtils.java | 34 +-
.../common/statemachine/DatanodeConfiguration.java | 69 ++
.../common/statemachine/DatanodeStateMachine.java | 11 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 276 +++++--
.../server/ratis/ContainerStateMachine.java | 19 +-
.../transport/server/ratis/XceiverServerRatis.java | 18 +-
.../container/keyvalue/KeyValueContainerCheck.java | 2 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 28 +-
.../container/keyvalue/helpers/ChunkUtils.java | 18 +-
.../keyvalue/helpers/KeyValueContainerUtil.java | 2 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 9 +-
.../container/ozoneimpl/ContainerController.java | 24 +-
.../replication/GrpcReplicationClient.java | 4 +-
.../OnDemandContainerReplicationSource.java | 10 +-
.../container/replication/ReplicationServer.java | 7 +-
.../container/stream/DirectoryServerSource.java | 12 +-
.../upgrade/DataNodeUpgradeFinalizer.java | 3 +-
.../common/impl/TestContainerDataYaml.java | 35 +-
.../TestCloseContainerCommandHandler.java | 33 +-
hadoop-hdds/docs/content/concept/OzoneManager.md | 2 +-
.../docs/content/concept/OzoneManager.zh.md | 2 +-
.../docs/content/feature/Nonrolling-Upgrade.md | 107 +++
hadoop-hdds/docs/content/feature/OM-HA.zh.md | 4 +-
hadoop-hdds/docs/content/feature/Quota.zh.md | 4 +-
hadoop-hdds/docs/content/feature/Recon.zh.md | 2 +-
hadoop-hdds/docs/content/feature/SCM-HA.md | 4 +-
hadoop-hdds/docs/content/feature/SCM-HA.zh.md | 4 +-
hadoop-hdds/docs/content/feature/_index.zh.md | 14 +-
.../feature/how-to-do-a-nonrolling-upgrade.md | 54 --
hadoop-hdds/docs/content/interface/Ofs.md | 28 +-
hadoop-hdds/docs/content/interface/ReconApi.md | 217 +++++-
hadoop-hdds/docs/content/security/SecuringTDE.md | 54 ++
.../docs/themes/ozonedoc/layouts/index.html | 2 +
.../container/common/helpers/MoveDataNodePair.java | 71 ++
.../hadoop/hdds/scm/metadata/SCMMetadataStore.java | 6 +
...inerLocationProtocolClientSideTranslatorPB.java | 30 +-
.../org/apache/hadoop/hdds/server/JsonUtils.java | 2 +
.../hadoop/hdds/server/http/HttpServer2.java | 17 +-
...equestLogAppender.java => TestHttpServer2.java} | 40 +-
.../apache/hadoop/hdds/utils/db/TestRDBStore.java | 2 +-
.../src/main/proto/ScmAdminProtocol.proto | 4 +-
.../src/main/proto/DatanodeClientProtocol.proto | 1 +
.../interface-client/src/main/proto/hdds.proto | 5 +
.../src/main/proto/SCMRatisProtocol.proto | 1 +
.../hadoop/hdds/scm/SCMCommonPlacementPolicy.java | 47 +-
.../java/org/apache/hadoop/hdds/scm/ScmUtils.java | 10 +
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 27 +-
.../hadoop/hdds/scm/block/DeletedBlockLogImpl.java | 280 ++++---
.../hdds/scm/block/DeletedBlockLogImplV2.java | 432 -----------
.../hdds/scm/block/SCMBlockDeletingService.java | 16 +-
.../scm/block/ScmBlockDeletingServiceMetrics.java | 186 +++++
.../scm/command/CommandStatusReportHandler.java | 6 +-
.../hdds/scm/container/ContainerManagerImpl.java | 67 +-
.../hdds/scm/container/ContainerReplicaCount.java | 1 +
.../hdds/scm/container/ContainerStateManager.java | 10 +-
.../hdds/scm/container/ReplicationManager.java | 578 +++++++++++----
.../scm/container/balancer/ContainerBalancer.java | 635 ++++++++++++----
.../balancer/ContainerBalancerConfiguration.java | 206 +++++-
.../balancer/ContainerBalancerMetrics.java | 13 +-
.../ContainerBalancerSelectionCriteria.java | 169 +++++
.../container/balancer/ContainerMoveSelection.java | 49 +-
.../scm/container/balancer/FindTargetGreedy.java | 135 ++++
.../scm/container/balancer/FindTargetStrategy.java | 71 ++
.../algorithms/SCMContainerPlacementRackAware.java | 26 +-
.../replication/ReplicationManagerMetrics.java | 189 +++++
.../scm/container/states/ContainerQueryKey.java | 33 +-
.../scm/container/states/ContainerStateMap.java | 41 +-
.../hadoop/hdds/scm/ha/InterSCMGrpcClient.java | 7 +-
.../org/apache/hadoop/hdds/scm/ha/RatisUtil.java | 130 ++--
.../hdds/scm/ha/SCMHADBTransactionBufferImpl.java | 6 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 2 +
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 26 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 6 +-
...actionCodec.java => MoveDataNodePairCodec.java} | 36 +-
.../hadoop/hdds/scm/metadata/SCMDBDefinition.java | 13 +-
.../hdds/scm/metadata/SCMMetadataStoreImpl.java | 13 +
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 18 +
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 13 +
.../hdds/scm/node/HealthyReadOnlyNodeHandler.java | 12 +
.../apache/hadoop/hdds/scm/node/NodeManager.java | 9 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 27 +
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 20 +
...ManagerV2Impl.java => PipelineManagerImpl.java} | 73 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 8 +-
.../hdds/scm/pipeline/PipelineReportHandler.java | 2 +
.../hdds/scm/pipeline/SCMPipelineManager.java | 801 --------------------
.../pipeline/WritableRatisContainerProvider.java | 99 ++-
...inerLocationProtocolServerSideTranslatorPB.java | 40 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 56 +-
.../hdds/scm/server/StorageContainerManager.java | 34 +-
.../scm/server/upgrade/SCMUpgradeFinalizer.java | 5 +-
.../ScmHAUnfinalizedStateValidationAction.java | 13 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 6 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 19 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 67 +-
.../container/TestCloseContainerEventHandler.java | 6 +-
.../hdds/scm/container/TestReplicationManager.java | 474 ++++++++++--
.../scm/container/TestSCMContainerManager.java | 6 +-
.../container/balancer/TestContainerBalancer.java | 466 ++++++++++--
.../TestSCMContainerPlacementRackAware.java | 27 +
.../TestSCMContainerPlacementRandom.java | 68 ++
.../hadoop/hdds/scm/ha/TestSCMHAConfiguration.java | 25 -
.../hdds/scm/node/TestContainerPlacement.java | 4 +-
.../hdds/scm/node/TestDatanodeAdminMonitor.java | 1 +
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 36 +-
.../hdds/scm/pipeline/MockPipelineManager.java | 20 +
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 40 +-
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 821 ---------------------
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 6 +
.../safemode/TestHealthyPipelineSafeModeRule.java | 14 +-
.../TestOneReplicaPipelineSafeModeRule.java | 6 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 20 +-
.../server/TestSCMUpdateServiceGrpcServer.java | 3 +
.../ozone/container/common/TestEndPoint.java | 3 +-
.../org/apache/hadoop/hdds/cli/OzoneAdmin.java | 7 +
.../hdds/scm/cli/ContainerBalancerCommands.java | 19 +-
.../scm/cli/ContainerBalancerStartSubcommand.java | 16 +-
.../hdds/scm/cli/ContainerOperationClient.java | 26 +-
.../org/apache/hadoop/ozone/client/BucketArgs.java | 25 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 45 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 41 +-
.../hadoop/ozone/om/helpers/BucketLayout.java | 62 ++
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 56 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 37 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 5 +-
.../hadoop/ozone/om/helpers/OzoneFSUtils.java | 25 +-
.../om/helpers/TestOmKeyLocationInfoGroup.java | 19 +
hadoop-ozone/dev-support/docker/Dockerfile | 38 +-
hadoop-ozone/dist/src/main/compose/ozone/README.md | 6 +
.../dist/src/main/compose/ozone/docker-config | 2 +
hadoop-ozone/dist/src/main/compose/ozone/run.sh | 4 +
.../src/main/compose/ozonesecure-ha/docker-config | 3 +
.../src/main/compose/ozonesecure/docker-config | 5 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 3 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 7 +-
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh | 4 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 46 +-
.../dist/src/main/license/update-jar-report.sh | 2 +-
.../dist/src/main/smoketest/basic/links.robot | 10 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 2 +-
.../dist/src/main/smoketest/s3/boto3.robot | 5 +-
.../dist/src/main/smoketest/s3/boto_client.py | 10 +-
.../dist/src/main/smoketest/s3/bucketcreate.robot | 3 +-
.../dist/src/main/smoketest/s3/bucketdelete.robot | 3 +-
.../dist/src/main/smoketest/s3/buckethead.robot | 3 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 10 +-
.../src/main/smoketest/s3/s3_compatbility_check.sh | 39 +
hadoop-ozone/integration-test/pom.xml | 5 +
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 12 +-
.../fs/ozone/TestOzoneFileSystemWithFSO.java | 3 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 39 +-
.../fs/ozone/TestRootedOzoneFileSystemWithFSO.java | 21 +
.../ozone/contract/ITestOzoneContractDistCp.java | 8 +
.../rooted/ITestRootedOzoneContractDistCp.java | 8 +
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 10 +-
.../apache/hadoop/hdds/scm/TestSCMSnapshot.java | 6 +-
.../TestContainerStateManagerIntegration.java | 40 +-
.../hdds/scm/pipeline/TestMultiRaftSetup.java | 173 +++++
.../hdds/scm/pipeline/TestPipelineClose.java | 2 +
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 24 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 7 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 2 +-
.../hadoop/ozone/MiniOzoneClusterProvider.java | 283 +++++++
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 21 +-
.../ozone/TestContainerBalancerOperations.java | 11 +-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 40 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 11 +-
.../hadoop/ozone/TestStorageContainerManager.java | 10 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 130 +++-
.../ozone/container/TestContainerReplication.java | 48 --
.../commandhandler/TestBlockDeletion.java | 125 +++-
.../server/TestSecureContainerServer.java | 6 +
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 3 -
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 13 +-
.../hadoop/ozone/om/TestOMStartupWithLayout.java | 197 -----
.../hadoop/ozone/om/TestObjectStoreWithFSO.java | 76 +-
.../hadoop/ozone/om/TestOmBlockVersioning.java | 21 +-
.../hadoop/ozone/om/TestOzoneManagerHAWithACL.java | 24 +-
.../ozone/om/TestOzoneManagerHAWithData.java | 32 +-
.../ozone/recon/TestReconWithOzoneManagerFSO.java | 5 +-
.../hadoop/ozone/scm/TestCloseContainer.java | 8 +
.../hadoop/ozone/scm/TestFailoverWithSCMHA.java | 99 ++-
.../ozone/scm/TestSCMInstallSnapshotWithHA.java | 19 +-
.../scm/node/TestDecommissionAndMaintenance.java | 42 +-
...TestScmAdminHA.java => TestNSSummaryAdmin.java} | 58 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 9 +-
.../src/main/proto/OmClientProtocol.proto | 7 +
.../org/apache/hadoop/ozone/om/OzoneManager.java | 31 +
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 4 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 12 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 6 +-
.../om/request/bucket/OMBucketCreateRequest.java | 102 +--
.../ozone/om/request/file/OMFileCreateRequest.java | 3 +-
.../request/file/OMFileCreateRequestWithFSO.java | 3 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 3 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 3 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 24 +-
.../request/s3/security/S3RevokeSecretRequest.java | 2 +-
.../ozone/om/upgrade/OMUpgradeFinalizer.java | 3 +-
.../om/request/key/TestOMKeyCreateRequest.java | 70 +-
.../request/key/TestOMKeyCreateRequestWithFSO.java | 13 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 4 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 2 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 4 +-
.../org/apache/hadoop/fs/ozone/OzoneFsShell.java | 4 +-
.../apache/hadoop/ozone/recon/ReconHttpServer.java | 6 +-
.../hadoop/ozone/recon/ReconServerConfigKeys.java | 4 -
.../hadoop/ozone/recon/api/NSSummaryEndpoint.java | 379 ++++++++--
.../hadoop/ozone/recon/api/types/DUResponse.java | 70 ++
.../recon/api/types/NamespaceSummaryResponse.java | 19 +-
.../ozone/recon/api/types/ResponseStatus.java | 1 +
.../ozone/recon/scm/ReconPipelineManager.java | 16 +-
.../recon/scm/ReconPipelineReportHandler.java | 22 +-
.../webapps/recon/ozone-recon-web/api/db.json | 324 +++++++-
.../webapps/recon/ozone-recon-web/api/routes.json | 22 +-
.../src/components/navBar/navBar.tsx | 5 +
.../src/components/rightDrawer/rightDrawer.tsx | 72 ++
.../src/constants/breadcrumbs.constants.tsx | 3 +-
.../webapps/recon/ozone-recon-web/src/routes.tsx | 5 +
.../insights.less => diskUsage/diskUsage.less} | 31 +-
.../src/views/diskUsage/diskUsage.tsx | 444 +++++++++++
.../src/views/overview/overview.tsx | 3 +-
.../ozone/recon/OMMetadataManagerTestUtils.java | 23 +
.../ozone/recon/api/TestNSSummaryEndpoint.java | 168 ++++-
.../java/org/apache/hadoop/ozone/s3/Gateway.java | 29 +
.../hadoop/ozone/s3/S3GatewayConfigKeys.java | 7 +
.../ozone/admin/nssummary/DiskUsageSubCommand.java | 214 ++++++
.../admin/nssummary/FileSizeDistSubCommand.java | 129 ++++
.../ozone/admin/nssummary/NSSummaryAdmin.java | 137 ++++
.../ozone/admin/nssummary/NSSummaryCLIUtils.java | 174 +++++
.../admin/nssummary/QuotaUsageSubCommand.java | 117 +++
.../ozone/admin/nssummary/SummarySubCommand.java | 115 +++
.../admin/{om => nssummary}/package-info.java | 6 +-
.../ozone/genesis/BenchMarkContainerStateMap.java | 6 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 20 +-
.../ozone/shell/bucket/InfoBucketHandler.java | 52 +-
.../hadoop/ozone/shell/keys/CopyKeyHandler.java | 4 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 4 +-
.../ozone/shell/volume/CreateVolumeHandler.java | 4 +-
pom.xml | 8 +-
270 files changed, 10949 insertions(+), 4594 deletions(-)
create mode 100755 dev-support/ci/lib/_all_libs.sh
create mode 100644 dev-support/ci/lib/_initialization.sh
create mode 100755 dev-support/ci/lib/_script_init.sh
create mode 100644 dev-support/ci/lib/_start_end.sh
create mode 100644 dev-support/ci/lib/_traps.sh
create mode 100644 dev-support/ci/lib/_verbosity.sh
create mode 100644 dev-support/ci/selective_ci_checks.bats
create mode 100755 dev-support/ci/selective_ci_checks.sh
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/ReplicationConfigValidator.java
delete mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHAConfiguration.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/TestReplicationConfigValidator.java
create mode 100644 hadoop-hdds/docs/content/feature/Nonrolling-Upgrade.md
delete mode 100644
hadoop-hdds/docs/content/feature/how-to-do-a-nonrolling-upgrade.md
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/container/common/helpers/MoveDataNodePair.java
copy
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/server/http/{TestHttpRequestLogAppender.java
=> TestHttpServer2.java} (50%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/DeletedBlockLogImplV2.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/block/ScmBlockDeletingServiceMetrics.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerSelectionCriteria.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/ContainerMetadata.java
=>
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerMoveSelection.java
(54%)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetGreedy.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindTargetStrategy.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ReplicationManagerMetrics.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/metadata/{DeletedBlocksTransactionCodec.java
=> MoveDataNodePairCodec.java} (55%)
rename
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/{PipelineManagerV2Impl.java
=> PipelineManagerImpl.java} (93%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SCMPipelineManager.java
delete mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMPipelineManager.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/BucketLayout.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestMultiRaftSetup.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterProvider.java
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMStartupWithLayout.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/{TestScmAdminHA.java
=> TestNSSummaryAdmin.java} (56%)
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/components/rightDrawer/rightDrawer.tsx
copy
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/{insights/insights.less
=> diskUsage/diskUsage.less} (75%)
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/views/diskUsage/diskUsage.tsx
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/DiskUsageSubCommand.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/FileSizeDistSubCommand.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/NSSummaryAdmin.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/NSSummaryCLIUtils.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/QuotaUsageSubCommand.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/nssummary/SummarySubCommand.java
copy hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/{om =>
nssummary}/package-info.java (88%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
