This is an automated email from the ASF dual-hosted git repository.
adoroszlai pushed a change to branch
dependabot-npm_and_yarn-hadoop-ozone-recon-src-main-resources-webapps-recon-ozone-recon-web-es5-ext-0.10.64
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 301fc0ac0a [auto] Generated pnpm-lock from actions for
d55e0355b3ee8ac5eb6312b7cb686b12ef6b30cc
add b89d0e5239 Revert "[auto] Generated pnpm-lock from actions for
d55e0355b3ee8ac5eb6312b7cb686b12ef6b30cc"
add 9fb61ffa3b HDDS-10413. Recon - UnsupportedOperationException while
merging Incremental Container Reports. (#6260)
add 1e98ebb449 HDDS-10432. Hadoop FS client write(byte[], int, int) is
very slow in streaming (#6287)
add e0bf7b4f7b HDDS-10370. Recon - Handle the pre-existing missing empty
containers in clusters. (#6255)
add 8c4ab8e4cd HDDS-8683. Container balancer thread interrupt may not work
(#6179)
add f4406546ac HDDS-10416. Move HA-specific settings to
MiniOzoneHAClusterImpl.Builder (#6275)
add 4da5a64224 HDDS-10425. Increase OM transaction index for non-Ratis
based on existing Ratis transactionInfoTable (#6281)
add 543c9e79dd HDDS-9235. ReplicationManager metrics not collected after
restart. (#6280)
add 1830fe2055 HDDS-10367. Fix possible NPE in listKeysLight, listStatus,
listStatusLight (#6221)
add aa68aec220 HDDS-10324. Metadata are not updated when keys are
overwritten. (#6273)
add 4095ef1f48 HDDS-10278. Simplify tests using assertDoesNotThrow (#6291)
add 8fcd039cef HDDS-10437. Rename method to
getContainersPendingReplication (#6293)
add 01f8d62fbe HDDS-10331. Rename Java constants of ex-DFS config keys
(#6290)
add 1eeaa0baa2 HDDS-10144. Zero-Copy in replication (#6049)
add c3271b8bf4 HDDS-10144. (addendum) Zero-Copy in replication
add 083e9140a1 HDDS-10433. Add Prometheus scrape target for Datanodes in
Compose v2 (#6288)
add 804366a0f0 HDDS-6713. Avoid the need to cast to MiniOzoneHAClusterImpl
(#6295)
add 384103a2dd HDDS-10282. Fix pagination on the OM DB Insights page in
Recon (#6190)
add 0a5fc695b4 HDDS-7810. Support namespace summaries (du, dist & counts)
for OBJECT_STORE buckets. (#4245)
add 11fddc4191 HDDS-10041. Do not start the daemon inside the
OzoneManagerDoubleBuffer constructor. (#6300)
add 3a872b4a10 HDDS-10447. Extract helper methods from Ozone native ACL
unit tests (#6306)
add 2710129ce4 HDDS-10453. Bump httpclient to 4.5.14 (#6311)
add 61dbb0870a HDDS-10455. Bump protobuf-maven-plugin to 0.6.1 (#6313)
add 11c5eb86a4 HDDS-10456. Bump slf4j to 2.0.12 (#6312)
add b513cdcb54 HDDS-10439. Remove setConf from MiniOzoneCluster public
interface (#6320)
add a65991fa33 HDDS-10444. Reduce string concatenation in
ContainerImporter#importContainer (#6307)
add 650e77753b HDDS-10459. Bump snappy-java to 1.1.10.5 (#6324)
add a248ed1ea4 HDDS-10329. [Snapshot] Add unit-test for recreating
snapshots with deleted snapshot names. (#6298)
add ff1e4143ad HDDS-10430. Race condition around Pipeline#nodesInOrder
(#6316)
add f7a421b27b HDDS-10446. Refactor Node2ObjectsMap, Node2PipelineMap,
Node2ContainerMap (#6303)
add 80592135ed HDDS-10458. Mention `ozone admin datanode status
decommission` in docs (#6322)
add b69674c7f8 HDDS-10412. Prefix ACL check needs to resolve the bucket
link (#6268)
add f16b1affeb HDDS-10424. Improve error message for prefix without
trailing slash (#6328)
add 9a6ece27e7 HDDS-10457. Remove dependency commons-pool2 (#6317)
add 4243721dda HDDS-10450. Add GitHub actions labeler for the
reconciliation feature branch. (#6310)
add 87d8d6155c HDDS-10467. Reduce metrics visibility (#6332)
add 2f05353940 HDDS-10384. RPC client reusing thread resources. (#6326)
add 418528adea HDDS-10460. Refine audit logging for bucket property update
operations (#6329)
add 309e45996c HDDS-10472. Audit log should include EC replication config
(#6338)
add 140c5deffb HDDS-9343. Shift sortDatanodes logic to OM (#5391)
add a145dd5bfd HDDS-9343. (addendum) Shift sortDatanodes logic to OM
(#5391)
add 7c8160fe2c HDDS-10482. OMRequestTestUtils.createOmKeyInfo should set
key modification time (#6343)
add 9dab4d2173 HDDS-10487. Intermittent crash in TestSnapshotDiffManager
(#6346)
add d94aadc4b8 HDDS-10480. Avoid proto2 ByteString.toByteArray() calls.
(#6342)
add 48bc30f514 HDDS-10470. Populate Maven dependency cache in separate
workflow (#6340)
add 1322add415 HDDS-10493. Fix parameter number warning in DatanodeDetails
and ContainerInfo (#6349)
add 6222aed33e HDDS-10495. Removed JMockit dependency (#6352)
add f8a1e3cc3a HDDS-10090. Clean up TestStorageContainerManager (#6347)
add bf91862796 HDDS-10473. Extract constants in HddsDispatcher (#6337)
add c439b7341a HDDS-10499. Bump org.kohsuke.metainf-services to 1.11
(#6354)
add 4ee5fd917b HDDS-10500. Bump Jackson to 2.16.1 (#6356)
add cb255044e2 HDDS-10501. Bump maven-deploy-plugin to 3.1.1 (#6353)
add 5c57bcb37d HDDS-10188. Add unit tests around Distcp across buckets
(#6132)
add 131eec0587 HDDS-10485. Enable container scanner by default. (#6345)
add 07639859ec HDDS-10504. Remove unused VolumeInfo#configuredCapacity
(#6363)
add d68ea97e02 HDDS-10503. Bump jgrapht to 1.4.0 (#6364)
add 6c95fc90ee HDDS-10502. Improve generic type of IOUtils.close (#6361)
add a8dd1c2cc2 HDDS-10475. Refine audit logging for bucket creation (#6366)
add 91af26ab39 HDDS-10414. Some acceptance tests fail with Docker Compose
V2 (#6269)
add 6cd0b6f751 HDDS-10454. Make OzoneAcl immutable (#6319)
add e42b46d69c HDDS-10497. [hsync] Refresh block token immediately if
block token expires. (#6371)
add 3ce3a284bf HDDS-10512. Reorder initialization of ScmTopologyClient in
OM (#6374)
add 9222193084 HDDS-10492. [Doc] Update zh translation to Recon
Architecture (#6348)
add 359bbad1f1 HDDS-10050. Clean up TestStorageContainerManagerHelper
(#6373)
add 71e4ff373a HDDS-10498. Improved configuration for license compliance
(#6358)
add 2ffdb3ccb8 HDDS-10516. Add metrics for chunk read from internal ratis
(#6375)
add 3d193fc06e HDDS-10521. ETag field should not be returned during
GetObject if the key does not contain ETag field (#6377)
add f4ba2beaaa HDDS-10440. Set new cluster ID and config instance in
MiniOzoneCluster#build(). (#6325)
add b82a2a2842 HDDS-10496. Fetch dependencies for cache using actual build
(#6359)
add a99c12bd04 HDDS-10523. Extract module-specific gitignore (#6379)
add e2a4609f3c HDDS-10526. Bump follow-redirects to 1.15.6 (#6382)
add 0bf2a29cbb HDDS-10120. BindException in some integration tests with
Java 17 (#6384)
add 01deb4f679 HDDS-10534. Removed objenesis dependency (#6391)
add 8efcaffaa4 HDDS-10533. Bump junit5 to 5.10.2 (#6387)
add 238bff09e9 HDDS-10537. Bump docker-maven-plugin to 0.44.0 (#6390)
add d1e37d6458 HDDS-10535. Remove unused
RangerRestMultiTenantAccessController. (#6392)
add 9291f89428 HDDS-10543. Bump LMAX Disruptor to 3.4.4 (#6388)
add 89b700d5bf HDDS-10505. Move space reservation logic to VolumeUsage
(#6370)
add a122acca97 HDDS-10549. Fix typo in Freon error message (#6397)
add 9afaeacf71 HDDS-10546. OM startup failure as leader is not getting
ready (#6395)
add d4606e1795 HDDS-10182. Add P99 quantiles and Min/Max Metrics for
Xceiver Client Performance Metrics (#6070)
add 264cbc6192 HDDS-10476. Add metrics about bucket utilization. (#6344)
add e9073166da HDDS-9802. Tool to fix corrupted snapshot chain (#6386)
add 8582214315 HDDS-10515. Reduce OzoneAcl constructor count (#6396)
add ca8f1f76b2 HDDS-10552. Downgrade Surefire to 3.0.0-M4 (#6406)
add e6ed46a262 HDDS-9085. Recon Directories Pending for Deletion Inside
OMDB Insight (#5521)
add 8dabb2bb2f HDDS-10554. Bump Zookeeper to 3.8.4 (#6407)
add 8a522f0c3f HDDS-10311. Speed up TestOmMetrics (#6409)
add f399f1ba5c HDDS-10539. Replace GSON with Jackson in multitenancy code.
(#6399)
add d45acfb2bb HDDS-10544. Move LMAX Disruptor to runtime scope (#6394)
add 72fef614dc HDDS-10436. datanode status decommission command should
have json output option (#6369)
add d70d247a49 HDDS-10551. Improve HTTPFS Documentation for Iterative
Liststatus Functionality (#6412)
add c4dc6a072f HDDS-5865. Make read retry interval and attempts in
BlockInputStream configurable (#6408)
add e3a7224a81 HDDS-10477. Make Rocksdb tools native lib compatible with
all chipset with the same arch (#6341)
add 97038ef097 HDDS-10118. hdds-rocks-native fails to build with Java11+
(#6415)
add 4364ef7cff HDDS-10566. Bump commons-configuration2 to 2.10.1 (#6417)
add 74f04a7328 HDDS-10556. Checkstyle summary excludes errors with xml
(#6410)
add 44493ef760 HDDS-10575. Bump jersey2.version to 2.42 (#6423)
add ab56117d8e HDDS-10576. Bump maven-assembly-plugin to 3.7.1 (#6422)
add 426259664d HDDS-10577. Bump maven-shade-plugin to 3.5.2 (#6425)
add 042f4c5c9b HDDS-10580. Bump maven-eclipse-plugin to 2.10 (#6424)
add 2bc5f531e2 HDDS-10541. Replace GSON with Jackson in debug commands
(#6401)
add 1ceec5c2bc HDDS-10540. Replace GSON with Jackson in nssummary. (#6400)
add f9e5178cdd HDDS-10143. Intermittent failure in
TestOzoneRpcClientWithRatis.testParallelDeleteBucketAndCreateKey (#6335)
add 4ca8edb64a HDDS-10562. Fix infinite loop in ozone fs -ls /volume
(#6416)
add 82c02d3eec HDDS-10564. Make Outputstream writeExecutor daemon threads.
(#6418)
add 873ae852ee HDDS-10579. Remove org.ow2.asm dependency (#6427)
add b267a57831 HDDS-10553. Add test case for creating file with EC
replication config (#6405)
add 48e547ac29 HDDS-10448. Orientation fixes for List Keys Metrics
Dashboard (#6308)
add e39166f8a7 HDDS-10581. NPE in SummarySubCommand and
DiskUsageSubCommand (#6429)
add 77384eaeca HDDS-8247. Run acceptance tests with JDK 17 (#6428)
add 07a68c02d4 HDDS-10583. Thread name prefix in ReplicationSupervisor is
null (#6430)
add 8eca5b77c0 HDDS-10547. Fix shared buffer for datanode checksum
calculation (#6402)
add 9bdac501a7 HDDS-10584. Exclude proto3 classes from coverage (#6431)
add 7cb1737e32 HDDS-10586. Avoid loading network topology layer schema
file for every read (#6434)
add d03f68f9dd HDDS-10510. Improve output of containerbalancer stop (#6433)
add 80bafd0f59 HDDS-10578. Clean up Hadoop dependencies (#6432)
add c6c611fa60 HDDS-10587. Reset ETag's thread-local MessageDigest
instance on exception (#6435)
add 6822d53e93 HDDS-10588. Bump hadoop-shaded-guava to 1.2.0 (#6440)
add 7feafe91f2 HDDS-10518. Create base Builder in WithMetadata and
WithObjectID (#6378)
add e68183e169 HDDS-10574. Improve TestObjectPut (#6426)
add 7559e1f6b9 HDDS-10524. [Snapshot] Invalidate the cache entry from
snapshotInfoTable cache in OMSnapshotPurgeRequest (#6443)
add 3651d6c719 HDDS-10594. Update file encryption info when overwriting
key (#6445)
add 140cf0f162 HDDS-9764. Add Robot test for JSON output where missing
(#6448)
add 38e3add916 HDDS-10600. Bump nimbus-jose-jwt to 9.37.2 (#6454)
add 9f6ac2fc37 HDDS-9242. Merge unit and integration checks into a
combined one (#6199)
add 85c9c97fc5 HDDS-10570. S3A: `fs -touch` creates directory instead of
empty file in FSO bucket (#6452)
add cb5d51983d HDDS-9534. Support namespace summaries (du, dist & counts)
for LEGACY buckets with file system disabled (#5517)
add 29edabb374 HDDS-10618. Bump gson to 2.10.1 (#6459)
add 33c05b42da HDDS-10620. Bump build-helper-maven-plugin to 3.5.0 (#6462)
add 8bd1a92539 HDDS-10622. Bump commons-io to 2.16.0 (#6463)
add 8a257840a4 HDDS-10623. Bump spotbugs-maven-plugin to 3.1.12.2 (#6460)
add 7c20f388a5 HDDS-10624. Bump reflections to 0.10.2 (#6461)
add 0599aeec5b HDDS-10621. Remove TimeoutFuture in favour of the original
one in Guava (#6464)
add 077fff4c4f HDDS-10358. Allow Container Balancer tuning options to be
set in CLI (#6241)
add 83d75861b0 HDDS-10613. Fix typo in Snapshot feature doc (#6465)
add a73fb3731b HDDS-10507. Use equals() instead of == for nodes in
NetworkTopology (#6368)
add f94c9f9a52 HDDS-10597. Use MutableGauge for threshold based
SafeModeMetrics (#6446)
add 92bc617801 HDDS-10462. Fail Datanode Decommission Early (#6367)
add ccaaf57183 HDDS-10316. Speed up TestReconTasks (#6223)
add 129cdc18ed HDDS-10612. Add Robot test to verify Container Balancer for
RATIS containers (#6457)
add f07743096f HDDS-10636. Ozone Recon - Filter EMPTY MISSING Containers
in UnHealthy State Containers API. (#6471)
add a38570a799 HDDS-9930. Remove open keys as well when keys are deleted
from KeyTable (#6079) (#6472)
add 7da5ecb585 HDDS-10590. [Snapshot] Synchronized snapshot purge, set
snapshot property and SstFilteringService (#6456)
add 3467db1b1c HDDS-9200. [Snapshot] Added logs and metrics for snapshot
purge and set property APIs (#6453)
add e34de9fbdb HDDS-5567. interface/Cli.md translation (#6474)
add 6b92a37af1 HDDS-10644. Intermittent failure in testBalancer.robot
(#6481)
add 301664e46a HDDS-10206. Expose jmx metrics for snapshot cache size on
the ozone manager. (#6138)
add 87dd27347c HDDS-10661. Bump zstd-jni to 1.5.6-2 (#6483)
add 31c2cfb0ab Revert "HDDS-10316. Speed up TestReconTasks (#6223)"
add df6c386a53 HDDS-10659. Remove cglib dependency (#6488)
add 5759427b02 HDDS-10660. Bump jacoco to 0.8.12 (#6487)
add 91be6b39a2 HDDS-10662. Bump maven-gpg-plugin to 3.2.2 (#6484)
add 6ed1e58dd5 HDDS-10230. Preventing V3 Schema from Creating Container DB
in the Wrong Location (#6113)
add b8faba6b87 HDDS-10643. Avoid terminating SCM by statemachine during
normal SCM stop. (#6478)
add e31e68ab2a HDDS-10560. Link rocksdb lib to Ozone rocksdb tools lib
relative path instead of absolute path (#6414)
add 7a11aaf5d8 Merge remote-tracking branch 'origin/master' into
dependabot-npm_and_yarn-hadoop-ozone-recon-src-main-resources-webapps-recon-ozone-recon-web-es5-ext-0.10.64
No new revisions were added by this update.
Summary of changes:
.../checks/integration.sh => .github/labeler.yml | 10 +-
.github/workflows/ci.yml | 20 +-
.github/workflows/intermittent-test-check.yml | 6 +-
.../unit.sh => .github/workflows/label-pr.yml | 21 +-
.github/workflows/populate-cache.yml | 74 +
.github/workflows/repeat-acceptance.yml | 12 +-
.gitignore | 35 +-
LICENSE.txt | 2 -
dev-support/ci/selective_ci_checks.bats | 53 +-
dev-support/ci/selective_ci_checks.sh | 38 +-
dev-support/rat/rat-exclusions.txt | 73 +
.../hadoop/hdds/scm/ContainerClientMetrics.java | 15 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 33 +
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 4 +-
.../hadoop/hdds/scm/XceiverClientMetrics.java | 44 +-
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 4 +-
.../hdds/scm/storage/AbstractCommitWatcher.java | 2 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 23 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 5 +-
.../hadoop/hdds/scm/storage/CommitWatcher.java | 23 +-
.../hdds/scm/storage/RatisBlockOutputStream.java | 9 +-
.../ozone/client/io/BlockInputStreamFactory.java | 6 +-
.../client/io/BlockInputStreamFactoryImpl.java | 13 +-
.../ozone/client/io/ByteBufferOutputStream.java | 5 +
.../hadoop/ozone/client/io/ECBlockInputStream.java | 16 +-
.../ozone/client/io/ECBlockInputStreamFactory.java | 6 +-
.../client/io/ECBlockInputStreamFactoryImpl.java | 13 +-
.../ozone/client/io/ECBlockInputStreamProxy.java | 14 +-
.../io/ECBlockReconstructedStripeInputStream.java | 10 +-
.../hdds/scm/storage/DummyBlockInputStream.java | 9 +-
.../storage/DummyBlockInputStreamWithRetry.java | 9 +-
.../hdds/scm/storage/TestBlockInputStream.java | 24 +-
.../hadoop/ozone/client/io/ECStreamTestUtil.java | 6 +-
.../client/io/TestBlockInputStreamFactoryImpl.java | 14 +-
.../ozone/client/io/TestECBlockInputStream.java | 110 +-
.../client/io/TestECBlockInputStreamProxy.java | 13 +-
.../io/TestECBlockReconstructedInputStream.java | 10 +-
.../TestECBlockReconstructedStripeInputStream.java | 12 +-
hadoop-hdds/common/pom.xml | 9 -
.../hadoop/hdds/client/ECReplicationConfig.java | 8 +
.../hadoop/hdds/conf/OzoneConfiguration.java | 60 +-
.../apache/hadoop/hdds/fs/SpaceUsageSource.java | 4 +-
.../hadoop/hdds/protocol/DatanodeDetails.java | 99 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 4 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 74 +-
.../apache/hadoop/hdds/scm/client/ScmClient.java | 12 +-
.../hadoop/hdds/scm/container/ContainerInfo.java | 42 +-
.../container/common/helpers/AllocatedBlock.java | 14 +-
.../org/apache/hadoop/hdds/scm/net/InnerNode.java | 14 +
.../apache/hadoop/hdds/scm/net/InnerNodeImpl.java | 159 +-
.../hadoop/hdds/scm/net/NetworkTopologyImpl.java | 31 +-
.../java/org/apache/hadoop/hdds/scm/net/Node.java | 20 +
.../org/apache/hadoop/hdds/scm/net/NodeImpl.java | 15 +
.../hadoop/hdds/scm/net/NodeSchemaManager.java | 8 +
.../apache/hadoop/hdds/scm/pipeline/Pipeline.java | 120 +-
.../protocol/StorageContainerLocationProtocol.java | 11 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 9 +
.../org/apache/hadoop/hdds/server/JsonUtils.java | 18 +
.../java/org/apache/hadoop/hdds/utils/IOUtils.java | 4 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 196 +-
.../hadoop/ozone/grpc/metrics/GrpcMetrics.java | 4 +-
.../hadoop/util/PerformanceMetricsInitializer.java | 2 +-
.../common/src/main/resources/ozone-default.xml | 17 +-
.../hadoop/hdds/fs/MockSpaceUsageSource.java | 22 +
.../dev-support/findbugsExcludeFile.xml | 12 -
.../container/common/helpers/ContainerMetrics.java | 19 +
.../container/common/impl/HddsDispatcher.java | 77 +-
.../common/transport/server/XceiverServerGrpc.java | 8 +-
.../common/transport/server/ratis/CSMMetrics.java | 15 +-
.../server/ratis/ContainerStateMachine.java | 12 +-
.../transport/server/ratis/XceiverServerRatis.java | 108 +-
.../container/common/volume/AbstractFuture.java | 1295 ---------
.../ozone/container/common/volume/HddsVolume.java | 21 +-
.../container/common/volume/MutableVolumeSet.java | 12 +-
.../container/common/volume/StorageVolume.java | 5 -
.../common/volume/StorageVolumeChecker.java | 10 +-
.../common/volume/ThrottledAsyncChecker.java | 3 +-
.../container/common/volume/TimeoutFuture.java | 161 --
.../ozone/container/common/volume/VolumeInfo.java | 93 +-
.../container/common/volume/VolumeInfoMetrics.java | 6 +-
.../ozone/container/common/volume/VolumeUsage.java | 125 +-
.../ECReconstructionCoordinator.java | 10 +-
.../container/keyvalue/KeyValueContainerCheck.java | 6 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 10 +-
.../helpers/KeyValueContainerLocationUtil.java | 10 +-
.../keyvalue/impl/ChunkManagerFactory.java | 4 +-
.../ozoneimpl/ContainerScannerConfiguration.java | 4 +-
.../container/replication/ContainerImporter.java | 13 +-
.../replication/GrpcReplicationService.java | 93 +-
.../container/replication/MeasuredReplicator.java | 22 +-
.../container/replication/ReplicationServer.java | 36 +-
.../replication/ReplicationSupervisor.java | 7 +-
.../replication/SendContainerRequestHandler.java | 14 +-
.../ozone/container/common/ContainerTestUtils.java | 2 +-
.../ozone/container/common/SCMTestUtils.java | 4 +-
.../container/common/TestDatanodeStateMachine.java | 12 +-
.../container/common/impl/TestHddsDispatcher.java | 163 +-
.../statemachine/TestDatanodeConfiguration.java | 2 +-
.../container/common/utils/TestHddsVolumeUtil.java | 35 +
.../common/volume/TestPeriodicVolumeChecker.java | 2 +-
.../common/volume/TestReservedVolumeSpace.java | 28 +-
.../container/common/volume/TestVolumeSet.java | 2 +-
.../common/volume/TestVolumeSetDiskChecks.java | 11 +-
.../container/ozoneimpl/TestContainerReader.java | 2 +-
.../TestContainerScannerConfiguration.java | 4 +-
.../container/ozoneimpl/TestOzoneContainer.java | 2 +-
.../replication/TestGrpcReplicationService.java | 181 +-
.../TestGrpcReplicationServiceWithZeroCopy.java | 20 +-
.../replication/TestReplicationSupervisor.java | 16 +-
.../TestSendContainerRequestHandler.java | 2 +-
.../upgrade/TestDatanodeUpgradeToSchemaV3.java | 4 +-
.../upgrade/TestDatanodeUpgradeToScmHA.java | 3 +
hadoop-hdds/docs/.gitignore | 2 +
hadoop-hdds/docs/content/feature/Decommission.md | 8 +
.../docs/content/feature/Decommission.zh.md | 8 +
hadoop-hdds/docs/content/feature/ErasureCoding.md | 18 +
hadoop-hdds/docs/content/feature/Snapshot.md | 2 +-
hadoop-hdds/docs/content/interface/Cli.zh.md | 311 ++
hadoop-hdds/docs/content/interface/HttpFS.md | 2 +-
hadoop-hdds/docs/content/interface/ReconApi.zh.md | 600 +++-
hadoop-hdds/docs/content/interface/_index.zh.md | 2 +-
hadoop-hdds/docs/pom.xml | 24 -
hadoop-hdds/framework/pom.xml | 2 +
.../hadoop/hdds/scm/client/ScmTopologyClient.java | 137 +
.../hadoop/hdds/scm/client/package-info.java | 21 +-
.../scm/protocol/ScmBlockLocationProtocol.java | 8 +
...lockLocationProtocolClientSideTranslatorPB.java | 43 +
...inerLocationProtocolClientSideTranslatorPB.java | 63 +-
.../hdds/server/events/EventWatcherMetrics.java | 14 +-
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 6 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 8 +-
.../hdds/utils/db/TestTypedRDBTableStore.java | 8 +-
hadoop-hdds/hadoop-dependency-client/pom.xml | 28 +
hadoop-hdds/hadoop-dependency-server/pom.xml | 28 +
hadoop-hdds/hadoop-dependency-test/pom.xml | 5 +
.../src/main/proto/ScmAdminProtocol.proto | 7 +
.../interface-client/src/main/proto/hdds.proto | 24 +
.../src/main/proto/ScmServerProtocol.proto | 11 +-
.../utils/db/managed/JniLibNamePropertyWriter.java | 5 +-
.../utils/db/managed/ManagedRocksObjectUtils.java | 8 +
hadoop-hdds/pom.xml | 39 -
hadoop-hdds/rocks-native/pom.xml | 7 +-
hadoop-hdds/rocks-native/src/CMakeLists.txt | 24 +-
.../hadoop/hdds/utils/NativeLibraryLoader.java | 35 +-
.../utils/db/managed/ManagedRawSSTFileReader.java | 4 +-
.../hadoop/hdds/utils/TestNativeLibraryLoader.java | 3 +-
.../org/apache/ozone/graph/TestPrintableGraph.java | 84 +
.../server-scm/dev-support/findbugsExcludeFile.xml | 5 +
.../scm/container/balancer/ContainerBalancer.java | 12 +-
.../balancer/ContainerBalancerConfiguration.java | 4 +
.../container/replication/ReplicationManager.java | 1 +
.../replication/ReplicationManagerMetrics.java | 13 +-
.../hadoop/hdds/scm/ha/SCMRatisResponse.java | 20 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 2 +-
.../hadoop/hdds/scm/node/DatanodeAdminMonitor.java | 2 +-
.../hdds/scm/node/DatanodeAdminMonitorImpl.java | 3 +-
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 1 -
.../hdds/scm/node/NodeDecommissionManager.java | 82 +-
.../apache/hadoop/hdds/scm/node/NodeStatus.java | 1 -
.../hdds/scm/node/states/Node2ContainerMap.java | 92 -
.../hdds/scm/node/states/Node2PipelineMap.java | 28 +-
...lockLocationProtocolServerSideTranslatorPB.java | 15 +
...inerLocationProtocolServerSideTranslatorPB.java | 38 +-
.../hadoop/hdds/scm/safemode/SafeModeMetrics.java | 25 +-
.../hdds/scm/security/RootCARotationMetrics.java | 2 +-
.../hdds/scm/server/SCMBlockProtocolServer.java | 15 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 176 +-
.../hdds/scm/server/SCMContainerMetrics.java | 1 -
.../scm/server/SCMDatanodeHeartbeatDispatcher.java | 14 +-
.../hdds/scm/server/StorageContainerManager.java | 6 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 1 -
.../hdds/scm/container/Node2ContainerMap.java} | 101 +-
.../TestNode2ContainerMap.java | 6 +-
.../TestContainerReplicaPendingOps.java | 8 +
.../replication/TestECUnderReplicationHandler.java | 8 +
.../replication/TestReplicationManager.java | 8 +
.../hdds/scm/node/TestContainerPlacement.java | 2 +-
.../hdds/scm/node/TestDatanodeAdminMonitor.java | 8 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 342 ++-
.../ozone/container/common/TestEndPoint.java | 22 +-
.../scm/cli/ContainerBalancerStartSubcommand.java | 70 +-
.../scm/cli/ContainerBalancerStopSubcommand.java | 3 +-
.../hdds/scm/cli/ContainerOperationClient.java | 21 +-
.../cli/datanode/DecommissionStatusSubCommand.java | 94 +-
.../scm/cli/datanode/DecommissionSubCommand.java | 7 +-
.../datanode/TestContainerBalancerSubCommand.java | 9 +-
.../datanode/TestDecommissionStatusSubCommand.java | 2 +-
.../cli/datanode/TestDecommissionSubCommand.java | 7 +-
.../cli/datanode/TestMaintenanceSubCommand.java | 3 +-
.../cli/datanode/TestRecommissionSubCommand.java | 3 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 8 +-
.../apache/hadoop/ozone/client/OzoneVolume.java | 8 +-
.../client/checksum/ECBlockChecksumComputer.java | 50 +-
.../checksum/ReplicatedBlockChecksumComputer.java | 23 +-
.../hadoop/ozone/client/io/ECKeyOutputStream.java | 11 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 34 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 30 +-
.../ozone/client/io/TestKeyInputStreamEC.java | 12 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 41 +
.../java/org/apache/hadoop/ozone/OzoneAcl.java | 211 +-
.../hadoop/ozone/om/helpers/OmBucketArgs.java | 30 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 76 +-
.../hadoop/ozone/om/helpers/OmDirectoryInfo.java | 38 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 48 +-
.../ozone/om/helpers/OmMultipartKeyInfo.java | 11 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 69 +-
.../hadoop/ozone/om/helpers/OzoneAclUtil.java | 120 +-
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 29 +-
.../hadoop/ozone/om/helpers/WithMetadata.java | 45 +-
.../hadoop/ozone/om/helpers/WithObjectID.java | 53 +
.../ozone/om/helpers/WithParentObjectId.java | 31 +
...OzoneManagerProtocolClientSideTranslatorPB.java | 45 +-
.../hadoop/ozone/security/acl/OzoneObjInfo.java | 10 +
.../org/apache/hadoop/ozone/TestOzoneAcls.java | 154 +-
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 17 +-
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 2 +-
.../hadoop/ozone/om/helpers/TestOmVolumeArgs.java | 8 +-
.../hadoop/ozone/om/helpers/TestOzoneAclUtil.java | 38 +-
.../dev-support/checks/_mvn_unit_report.sh | 2 +-
hadoop-ozone/dev-support/checks/checkstyle.sh | 2 +-
hadoop-ozone/dev-support/checks/coverage.sh | 8 +-
hadoop-ozone/dev-support/checks/integration.sh | 2 +-
hadoop-ozone/dev-support/checks/junit.sh | 6 +-
hadoop-ozone/dev-support/checks/rat.sh | 8 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 -
hadoop-ozone/dist/.gitignore | 1 +
hadoop-ozone/dist/pom.xml | 2 +-
.../dashboards/Ozone - ListKey Metrics.json | 112 +-
.../.ssh/environment => ozone-balancer/.env} | 6 +-
.../compose/ozone-balancer/docker-compose.yaml | 179 ++
.../src/main/compose/ozone-balancer/docker-config | 59 +
.../.ssh/environment => ozone-balancer/test.sh} | 19 +-
.../dist/src/main/compose/ozone-ha/test.sh | 8 +-
.../dist/src/main/compose/ozone-om-ha/Dockerfile | 3 +-
.../dist/src/main/compose/ozone/prometheus.yml | 24 +-
.../dist/src/main/compose/ozonescripts/Dockerfile | 4 +-
.../src/main/compose/ozonesecure-ha/docker-config | 1 -
.../src/main/compose/ozonesecure-mr/docker-config | 1 -
.../src/main/compose/ozonesecure/docker-config | 1 -
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 3 -
hadoop-ozone/dist/src/main/license/jar-report.txt | 4 +-
.../src/main/smoketest/admincli/container.robot | 15 +
.../src/main/smoketest/admincli/datanode.robot | 14 +
.../src/main/smoketest/balancer/testBalancer.robot | 144 +
.../dist/src/main/smoketest/basic/links.robot | 15 +
.../dist/src/main/smoketest/recon/recon-api.robot | 12 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 24 +-
.../dist/src/main/smoketest/s3/objecthead.robot | 17 +-
.../dist/src/main/smoketest/s3/objectputget.robot | 24 +-
.../smoketest/security/ozone-secure-tenant.robot | 2 +-
.../dist/src/main/smoketest/topology/cli.robot | 21 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 6 +
.../fault-injection-test/mini-chaos-tests/pom.xml | 5 -
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 11 +-
hadoop-ozone/httpfsgateway/pom.xml | 6 +-
hadoop-ozone/integration-test/pom.xml | 5 -
.../fs/contract/AbstractContractAppendTest.java | 2 +
.../fs/contract/AbstractContractConcatTest.java | 2 +
.../AbstractContractCopyFromLocalTest.java | 2 +
.../AbstractContractGetFileStatusTest.java | 2 +
.../AbstractContractMultipartUploaderTest.java | 4 +
.../fs/contract/AbstractContractOpenTest.java | 2 +
.../AbstractContractRootDirectoryTest.java | 2 +
.../fs/contract/AbstractContractSeekTest.java | 4 +
.../fs/contract/AbstractContractSetTimesTest.java | 2 +
.../fs/contract/AbstractContractUnbufferTest.java | 2 +
.../fs/ozone/AbstractOzoneFileSystemTest.java | 14 +
.../ozone/AbstractRootedOzoneFileSystemTest.java | 125 +-
.../hadoop/fs/ozone/OzoneFileSystemTests.java | 16 +
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 25 +-
.../java/org/apache/hadoop/fs/ozone/TestHSync.java | 41 +
.../fs/ozone/TestOzoneFileSystemWithStreaming.java | 4 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 13 +-
.../hadoop/fs/ozone/TestOzoneFsSnapshot.java | 83 +-
.../hadoop/fs/ozone/TestRootedDDSWithFSO.java | 25 +-
.../org/apache/hadoop/fs/ozone/TestSafeMode.java | 2 +-
.../hadoop/hdds/scm/TestFailoverWithSCMHA.java | 2 +-
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 4 +-
.../hdds/scm/TestSCMInstallSnapshotWithHA.java | 2 +-
.../hadoop/hdds/scm/TestSecretKeySnapshot.java | 9 +-
.../apache/hadoop/hdds/scm/TestSecretKeysApi.java | 5 +-
.../hdds/scm/TestStorageContainerManager.java | 383 ++-
.../hdds/scm/TestStorageContainerManagerHA.java | 2 +-
.../scm/TestStorageContainerManagerHelper.java | 198 --
.../hadoop/hdds/scm/TestXceiverClientGrpc.java | 2 +-
.../scm/node/TestDecommissionAndMaintenance.java | 6 +-
.../hadoop/hdds/scm/storage/TestCommitWatcher.java | 4 +-
.../hadoop/hdds/upgrade/TestHDDSUpgrade.java | 9 +-
.../hadoop/hdds/upgrade/TestScmHAFinalization.java | 9 +-
.../db/managed/TestRocksObjectLeakDetector.java | 3 -
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 63 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 5 +-
.../hadoop/ozone/MiniOzoneClusterProvider.java | 34 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 45 +-
.../org/apache/hadoop/ozone/RatisTestHelper.java | 6 +-
.../org/apache/hadoop/ozone/TestBlockTokens.java | 12 +-
.../apache/hadoop/ozone/TestBlockTokensCLI.java | 5 +-
.../ozone/TestContainerBalancerOperations.java | 77 +-
.../java/org/apache/hadoop/ozone/TestDataUtil.java | 29 +
.../apache/hadoop/ozone/TestDelegationToken.java | 4 +
.../ozone/TestGetClusterTreeInformation.java | 87 +
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 18 +-
.../hadoop/ozone/TestMultipartObjectGet.java | 5 +-
.../apache/hadoop/ozone/TestOMSortDatanodes.java | 187 ++
.../hadoop/ozone/TestSecureOzoneCluster.java | 67 +-
.../hadoop/ozone/UniformDatanodesFactory.java | 24 +-
.../client/rpc/TestContainerStateMachine.java | 2 +-
.../rpc/TestContainerStateMachineFailures.java | 10 +-
.../rpc/TestContainerStateMachineFlushDelay.java | 2 +-
.../rpc/TestContainerStateMachineStream.java | 2 +-
.../client/rpc/TestFailureHandlingByClient.java | 2 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 2 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 2 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 24 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 68 +-
.../rpc/TestOzoneRpcClientForAclAuditLog.java | 4 +-
.../client/rpc/TestOzoneRpcClientWithRatis.java | 7 +-
.../transport/server/ratis/TestCSMMetrics.java | 4 +-
.../container/metrics/TestContainerMetrics.java | 8 +-
.../metrics/TestDatanodeQueueMetrics.java | 8 +-
.../container/ozoneimpl/TestOzoneContainer.java | 10 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 5 +
.../ozoneimpl/TestSecureOzoneContainer.java | 9 +-
.../container/server/TestContainerServer.java | 19 +-
.../server/TestSecureContainerServer.java | 25 +-
.../ozone/dn/ratis/TestDnRatisLogParser.java | 2 +-
.../hadoop/ozone/om/TestAddRemoveOzoneManager.java | 3 +-
.../apache/hadoop/ozone/om/TestBucketOwner.java | 6 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 37 +-
.../hadoop/ozone/om/TestListKeysWithFSO.java | 27 +
.../org/apache/hadoop/ozone/om/TestListStatus.java | 99 +-
.../hadoop/ozone/om/TestOMBucketLayoutUpgrade.java | 8 +-
.../hadoop/ozone/om/TestOMEpochForNonRatis.java | 46 +
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 2 +-
.../hadoop/ozone/om/TestOMUpgradeFinalization.java | 8 +-
.../ozone/om/TestOmContainerLocationCache.java | 11 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 556 ++--
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 20 +-
.../ozone/om/TestOzoneManagerHAWithAllRunning.java | 139 +-
.../om/TestOzoneManagerListVolumesSecure.java | 3 +
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 25 +-
.../ozone/om/multitenant/RangerUserRequest.java | 28 +-
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 62 +-
.../om/{ => snapshot}/TestOmSnapshotDisabled.java | 9 +-
.../TestOmSnapshotDisabledRestart.java | 8 +-
.../{ => snapshot}/TestOmSnapshotFileSystem.java | 7 +-
.../TestOmSnapshotFileSystemFso.java | 2 +-
.../TestOmSnapshotFileSystemLegacy.java | 2 +-
.../{ => snapshot}/TestOzoneManagerHASnapshot.java | 7 +-
.../om/snapshot/TestOzoneManagerSnapshotAcl.java | 7 +-
.../snapshot/TestOzoneManagerSnapshotProvider.java | 2 +-
.../om/snapshot/TestOzoneSnapshotRestore.java | 7 +-
.../TestSnapshotBackgroundServices.java | 9 +-
.../TestSnapshotDeletingService.java | 26 +-
.../TestSnapshotDirectoryCleaningService.java | 28 +-
.../ozone/parser/TestOzoneHARatisLogParser.java | 2 +-
.../ozone/recon/TestReconAndAdminContainerCLI.java | 4 +-
.../TestReconInsightsForDeletedDirectories.java | 28 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 25 +-
.../ozone/recon/TestReconWithOzoneManager.java | 41 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 10 +-
.../ozone/shell/TestDeletedBlocksTxnShell.java | 2 +-
.../shell/TestOzoneContainerUpgradeShell.java | 6 -
.../hadoop/ozone/shell/TestOzoneDebugShell.java | 1 -
.../hadoop/ozone/shell/TestOzoneShellHA.java | 43 +-
.../hadoop/ozone/shell/TestOzoneTenantShell.java | 82 +-
.../apache/hadoop/ozone/shell/TestScmAdminHA.java | 2 +-
.../ozone/shell/TestTransferLeadershipShell.java | 2 +-
.../tools/contract/AbstractContractDistCpTest.java | 2 +
.../apache/hadoop/ozone/om/OMMetadataManager.java | 34 +-
.../hadoop/ozone/om/helpers/OmPrefixInfo.java | 62 +-
.../hadoop/ozone/om/helpers/OzoneAclStorage.java | 15 +-
.../hadoop/ozone/om/helpers/TestOmPrefixInfo.java | 16 +-
.../ozone/om/helpers/TestOmPrefixInfoCodec.java | 2 +-
hadoop-ozone/ozone-manager/pom.xml | 5 -
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 3 +-
.../hadoop/ozone/om/BucketUtilizationMetrics.java | 118 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 101 +-
.../org/apache/hadoop/ozone/om/ListIterator.java | 2 +-
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 49 +
.../hadoop/ozone/om/OMPerformanceMetrics.java | 26 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 42 +-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 2 +-
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 4 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 60 +-
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 64 +-
.../org/apache/hadoop/ozone/om/ResolvedBucket.java | 11 +
.../RangerRestMultiTenantAccessController.java | 681 -----
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 12 +-
.../OzoneManagerDoubleBufferMetrics.java | 10 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 29 +-
.../ozone/om/ratis/metrics/package-info.java | 21 -
.../hadoop/ozone/om/request/RequestAuditor.java | 16 +-
.../om/request/bucket/OMBucketCreateRequest.java | 3 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 2 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 24 +-
.../om/request/key/OMKeyDeleteRequestWithFSO.java | 23 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 24 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 36 +-
.../om/request/key/OmKeysDeleteRequestWithFSO.java | 36 +-
.../request/key/acl/prefix/OMPrefixAclRequest.java | 86 +-
.../key/acl/prefix/OMPrefixAddAclRequest.java | 29 +-
.../key/acl/prefix/OMPrefixRemoveAclRequest.java | 21 +-
.../key/acl/prefix/OMPrefixSetAclRequest.java | 21 +-
.../request/snapshot/OMSnapshotPurgeRequest.java | 222 +-
.../snapshot/OMSnapshotSetPropertyRequest.java | 44 +-
.../om/request/validation/ValidatorRegistry.java | 2 +-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 18 +-
.../response/key/OMKeyDeleteResponseWithFSO.java | 12 +-
.../om/response/key/OMKeysDeleteResponse.java | 17 +-
.../response/key/OMKeysDeleteResponseWithFSO.java | 13 +-
.../response/snapshot/OMSnapshotPurgeResponse.java | 8 +-
.../hadoop/ozone/om/snapshot/SnapshotCache.java | 11 +-
.../ozone/om/upgrade/OMLayoutVersionManager.java | 4 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 21 +-
.../protocolPB/OzoneManagerRequestHandler.java | 27 +-
.../hadoop/ozone/protocolPB/RequestHandler.java | 39 +-
.../org/apache/hadoop/ozone/om/OmTestManagers.java | 5 +
.../ozone/om/ScmBlockLocationTestingClient.java | 11 +
.../hadoop/ozone/om/TestAuthorizerLockImpl.java | 9 +-
.../ozone/om/TestBucketUtilizationMetrics.java | 126 +
.../apache/hadoop/ozone/om/TestChunkStreams.java | 9 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 80 +-
.../hadoop/ozone/om/TestOMMultiTenantManager.java | 2 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 1 +
.../om/ratis/TestOzoneManagerDoubleBuffer.java | 4 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 4 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 14 +-
.../om/ratis/TestOzoneManagerStateMachine.java | 7 +-
.../ozone/om/request/OMRequestTestUtils.java | 1 +
.../request/TestOMClientRequestWithUserInfo.java | 92 +-
.../om/request/file/TestOMFileCreateRequest.java | 6 +-
.../om/request/key/TestOMKeyCreateRequest.java | 155 +-
.../om/request/key/TestOMPrefixAclRequest.java | 10 +-
.../TestS3InitiateMultipartUploadRequest.java | 11 +-
...estS3InitiateMultipartUploadRequestWithFSO.java | 2 +-
.../TestOMSnapshotPurgeRequestAndResponse.java | 67 +-
...estOMSnapshotSetPropertyRequestAndResponse.java | 48 +-
.../om/response/key/TestOMKeyDeleteResponse.java | 2 +-
.../key/TestOMKeyDeleteResponseWithFSO.java | 2 +-
.../om/response/key/TestOMKeysDeleteResponse.java | 3 +-
.../key/TestOMKeysDeleteResponseWithFSO.java | 3 +-
.../key/acl/prefix/TestOMPrefixAclResponse.java | 27 +-
.../volume/TestOMVolumeDeleteResponse.java | 15 +-
.../ozone/om/service/TestKeyDeletingService.java | 17 +-
.../ozone/om/snapshot/TestSnapshotCache.java | 42 +-
.../ozone/om/{ => snapshot}/TestSnapshotChain.java | 5 +-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 13 +-
.../ozone/om/{ => snapshot}/TestSnapshotInfo.java | 4 +-
.../om/{ => snapshot}/TestSstFilteringService.java | 11 +-
.../ozone/security/acl/OzoneNativeAclTestUtil.java | 170 ++
.../security/acl/TestOzoneNativeAuthorizer.java | 54 +-
.../hadoop/ozone/security/acl/TestParentAcl.java | 75 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 4 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 8 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 1 +
hadoop-ozone/ozonefs-hadoop3-client/pom.xml | 2 +
hadoop-ozone/ozonefs-hadoop3/pom.xml | 1 +
hadoop-ozone/ozonefs-shaded/pom.xml | 2 +-
hadoop-ozone/pom.xml | 67 +-
hadoop-ozone/recon/.gitignore | 1 +
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 7 +-
.../ozone/recon/api/handlers/BucketHandler.java | 39 +-
.../ozone/recon/api/handlers/EntityHandler.java | 103 +-
.../ozone/recon/api/handlers/FSOBucketHandler.java | 2 +-
.../recon/api/handlers/LegacyBucketHandler.java | 2 +-
.../ozone/recon/api/handlers/OBSBucketHandler.java | 268 ++
.../ozone/recon/fsck/ContainerHealthTask.java | 9 +-
.../recon/metrics/OzoneManagerSyncMetrics.java | 30 +-
.../persistence/ContainerHealthSchemaManager.java | 11 +
.../recon/recovery/ReconOMMetadataManager.java | 8 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 5 +
.../hadoop/ozone/recon/tasks/NSSummaryTask.java | 44 +-
.../ozone/recon/tasks/NSSummaryTaskWithLegacy.java | 308 +-
.../ozone/recon/tasks/NSSummaryTaskWithOBS.java | 236 ++
.../webapps/recon/ozone-recon-web/api/db.json | 245 +-
.../webapps/recon/ozone-recon-web/api/routes.json | 34 +-
.../webapps/recon/ozone-recon-web/pnpm-lock.yaml | 3059 ++++++++++----------
.../ozone-recon-web/src/views/insights/om/om.less | 17 +-
.../ozone-recon-web/src/views/insights/om/om.tsx | 500 ++--
.../ozone/recon/OMMetadataManagerTestUtils.java | 16 +-
.../apache/hadoop/ozone/recon/TestReconUtils.java | 25 +
.../ozone/recon/api/TestContainerEndpoint.java | 17 +
.../hadoop/ozone/recon/api/TestEndpoints.java | 19 +-
.../recon/api/TestNSSummaryEndpointWithLegacy.java | 11 +-
... => TestNSSummaryEndpointWithOBSAndLegacy.java} | 1083 ++++---
.../ozone/recon/api/TestOpenContainerCount.java | 26 +-
.../hadoop/ozone/recon/common/CommonUtils.java | 12 +-
.../ozone/recon/fsck/TestContainerHealthTask.java | 71 +-
...TestReconIncrementalContainerReportHandler.java | 27 +
.../impl/TestOzoneManagerServiceProviderImpl.java | 16 +-
.../ozone/recon/tasks/TestNSSummaryTask.java | 4 +-
...a => TestNSSummaryTaskWithLegacyOBSLayout.java} | 452 +--
...maryTask.java => TestNSSummaryTaskWithOBS.java} | 428 +--
hadoop-ozone/s3gateway/pom.xml | 5 +
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 19 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 92 +-
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 66 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 2 +-
.../hadoop/ozone/client/OzoneBucketStub.java | 82 +-
.../hadoop/ozone/client/OzoneVolumeStub.java | 9 +-
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 395 +--
.../hadoop/ozone/s3/endpoint/TestPartUpload.java | 58 +
.../s3/endpoint/TestPartUploadWithStream.java | 2 +-
.../ozone/s3/endpoint/TestUploadWithStream.java | 2 +-
.../ozone/admin/nssummary/DiskUsageSubCommand.java | 36 +-
.../admin/nssummary/FileSizeDistSubCommand.java | 19 +-
.../ozone/admin/nssummary/NSSummaryCLIUtils.java | 6 -
.../admin/nssummary/QuotaUsageSubCommand.java | 16 +-
.../ozone/admin/nssummary/SummarySubCommand.java | 17 +-
.../apache/hadoop/ozone/debug/ChunkKeyHandler.java | 71 +-
.../hadoop/ozone/debug/ContainerChunkInfo.java | 22 +
.../apache/hadoop/ozone/debug/ReadReplicas.java | 55 +-
.../hadoop/ozone/freon/BaseFreonGenerator.java | 2 +-
.../hadoop/ozone/freon/DatanodeChunkValidator.java | 2 +-
.../hadoop/ozone/freon/DatanodeSimulator.java | 4 +-
.../freon/containergenerator/GeneratorOm.java | 4 +-
.../apache/hadoop/ozone/repair/OzoneRepair.java | 64 +
.../org/apache/hadoop/ozone/repair/RDBRepair.java | 58 +
.../hadoop/ozone/repair/om/SnapshotRepair.java | 200 ++
.../hadoop/ozone/repair/om/package-info.java | 22 +
.../apache/hadoop/ozone/repair/package-info.java | 22 +
.../ozone/shell/tenant/GetUserInfoHandler.java | 44 +-
.../shell/tenant/TenantAssignAdminHandler.java | 21 +-
.../ozone/shell/tenant/TenantCreateHandler.java | 15 +-
.../ozone/shell/tenant/TenantDeleteHandler.java | 17 +-
.../ozone/shell/tenant/TenantListHandler.java | 31 +-
.../ozone/shell/tenant/TenantListUsersHandler.java | 19 +-
.../shell/tenant/TenantRevokeAdminHandler.java | 21 +-
pom.xml | 524 +---
530 files changed, 14606 insertions(+), 10620 deletions(-)
copy hadoop-ozone/dev-support/checks/integration.sh => .github/labeler.yml
(78%)
mode change 100755 => 100644
copy hadoop-ozone/dev-support/checks/unit.sh => .github/workflows/label-pr.yml
(73%)
mode change 100755 => 100644
create mode 100644 .github/workflows/populate-cache.yml
create mode 100644 dev-support/rat/rat-exclusions.txt
delete mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/AbstractFuture.java
delete mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/TimeoutFuture.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotFileSystemFso.java
=>
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/replication/TestGrpcReplicationServiceWithZeroCopy.java
(72%)
create mode 100644 hadoop-hdds/docs/.gitignore
create mode 100644 hadoop-hdds/docs/content/interface/Cli.zh.md
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/client/ScmTopologyClient.java
copy
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotFileSystemFso.java
=>
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/client/package-info.java
(73%)
create mode 100644
hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/graph/TestPrintableGraph.java
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/states/Node2ContainerMap.java
rename
hadoop-hdds/server-scm/src/{main/java/org/apache/hadoop/hdds/scm/node/states/Node2ObjectsMap.java
=> test/java/org/apache/hadoop/hdds/scm/container/Node2ContainerMap.java} (63%)
rename
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/{node/states =>
container}/TestNode2ContainerMap.java (99%)
create mode 100644 hadoop-ozone/dist/.gitignore
rename hadoop-ozone/dist/src/main/compose/{ozonescripts/.ssh/environment =>
ozone-balancer/.env} (85%)
create mode 100644
hadoop-ozone/dist/src/main/compose/ozone-balancer/docker-compose.yaml
create mode 100644
hadoop-ozone/dist/src/main/compose/ozone-balancer/docker-config
rename hadoop-ozone/dist/src/main/compose/{ozone-om-ha/.ssh/environment =>
ozone-balancer/test.sh} (65%)
create mode 100644
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot
delete mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestStorageContainerManagerHelper.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestGetClusterTreeInformation.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestOMSortDatanodes.java
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOmSnapshotDisabled.java (92%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOmSnapshotDisabledRestart.java (93%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOmSnapshotFileSystem.java (99%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOmSnapshotFileSystemFso.java (95%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOmSnapshotFileSystemLegacy.java (95%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestOzoneManagerHASnapshot.java (97%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestSnapshotBackgroundServices.java (98%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestSnapshotDeletingService.java (96%)
rename
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/{ =>
snapshot}/TestSnapshotDirectoryCleaningService.java (95%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/BucketUtilizationMetrics.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/multitenant/RangerRestMultiTenantAccessController.java
rename
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/{metrics
=> }/OzoneManagerDoubleBufferMetrics.java (95%)
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/package-info.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestBucketUtilizationMetrics.java
rename hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/{
=> snapshot}/TestSnapshotChain.java (98%)
rename hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/{
=> snapshot}/TestSnapshotInfo.java (96%)
rename hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/{
=> snapshot}/TestSstFilteringService.java (97%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/acl/OzoneNativeAclTestUtil.java
create mode 100644 hadoop-ozone/recon/.gitignore
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/handlers/OBSBucketHandler.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithOBS.java
copy
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/{TestNSSummaryEndpointWithLegacy.java
=> TestNSSummaryEndpointWithOBSAndLegacy.java} (56%)
copy
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/{TestNSSummaryTask.java
=> TestNSSummaryTaskWithLegacyOBSLayout.java} (55%)
copy
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/{TestNSSummaryTask.java
=> TestNSSummaryTaskWithOBS.java} (57%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/OzoneRepair.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/RDBRepair.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/SnapshotRepair.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/package-info.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/package-info.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
