This is an automated email from the ASF dual-hosted git repository.
ashishkr pushed a change to branch HDDS-7593
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 905834e0ee HDDS-10700. Merge HDDS-10780 into HBase feature branch.
(#6641)
add 0c59c18b52 HDDS-10668. Remove custom dependabot workflow (#6501)
add c1b27a87b5 HDDS-10630. Add missing parent directories deleted between
initiate and complete MPU (#6496)
add 83dbd1d17f HDDS-1056. Remove Standalone as the replication type in
Ozone client api. (#6466)
add cba8c85e22 HDDS-10681. EC Reconstruction does not issue put block to
data index if it is unused (#6514)
add a523fd9d23 HDDS-8450. Dedicated acceptance test suite for s3a (#6458)
add c63e2b9324 HDDS-10132.
TestStorageContainerManager.testScmProcessDatanodeHeartbeat fails with Java 17
(#6516)
add a5fccbc1d6 HDDS-10682. EC Reconstruction creates empty chunks at the
end of blocks with partial stripes (#6515)
add 8244a6f762 HDDS-10625. Remove unused netty-related config options from
SCM (#6528)
add 531216c2f9 HDDS-10598. Rename unit check to native (#6521)
add 1bd11f25a2 HDDS-10605. Add a configuration option for compliance mode
(#6470)
add 5151a5070b HDDS-10690. SCMStateMachine Override
LeaderEventApi.notifyLeaderReady (#6529)
add 93a2489ab9 HDDS-10452. Improve Recon Disk Usage to fetch and display
Top N records based on size. (#6318)
add f295b8ef89 HDDS-10695. Remove unused plugin maven-failsafe-plugin
(#6534)
add e0a403351e HDDS-10463. Fail Datanode Maintenance early (#6517)
add 92f2449cca HDDS-10615. ETag change detected in S3A contract test
(#6519)
add 89aa202a41 HDDS-10703. Bump commons-validator to 1.8.0 (#6526)
add f10262d39d Revert "HDDS-10703. Bump commons-validator to 1.8.0 (#6526)"
add b6f953bf70 HDDS-10671. Wrong size of response data in om-echo. (#6504)
add c18a34068c HDDS-10686. Bump npm packages only for security (#6522)
add 20c84d87e7 HDDS-10434. Added permission check for snapshotDiff related
operations (#6505)
add 72240fa726 HDDS-10692. ozone s3 getsecret prints some internal details
(#6531)
add fdc38b50fc HDDS-7252. Polled source Datanodes are wrongly not
re-considered for balancing in Container Balancer (#6305)
add 7bb7938062 HDDS-10680. Duplicate delete key blocks sent to SCM (#6513)
add 4f9b86ece1 HDDS-10704. Do not fail read of EC block if the last chunk
is empty (#6540)
add 233c069083 HDDS-10679. Enable ITestS3ACommitterMRJob (#6539)
add 3e97d8fd6a HDDS-10156. Optimize Snapshot Cache get and eviction (#6024)
add 4b8871f1d2 HDDS-10309. Speed up TestSnapshotDeletingService (#6541)
add 99a5703372 HDDS-10652. EC Reconstruction fails with "IOException: None
of the block data have checksum" after upgrade (#6520)
add cc023e7f92 HDDS-10614. Avoid decreasing cached space usage below zero
(#6508)
add 4d35d40f30 HDDS-10705. Avoid persist duplicate DeleteBlockCommands on
DN (#6547)
add 5dbd3cfe30 HDDS-10717. nodeFailureTimeoutMs should be initialized
before syncTimeoutRetry (#6560)
add 8310617008 HDDS-10718. Bump commons-daemon to 1.3.4 (#6545)
add 7fbbd82441 HDDS-10722. Bump aspectj to 1.9.22 (#6524)
add 31aa998ff7 HDDS-10724. Bump commons-io to 2.16.1 (#6565)
add e46fab8f90 HDDS-10727. Bump
com.gradle:common-custom-user-data-maven-extension to 1.13 (#6566)
add 4fc7635fc7 Revert "HDDS-10722. Bump aspectj to 1.9.22 (#6524)"
add aff383874f HDDS-10728. Bump jsch to 0.1.55 (#6564)
add bf1f644a29 HDDS-10729. Bump javassist to 3.30.2-GA (#6525)
add 96fc70e66c HDDS-10719. Avoid empty ETag for key created outside of S3
(#6563)
add 316eade978 HDDS-10716. Remove skipTrash option from IOException
message in OFS recursive volume delete (#6559)
add d99529c0c7 HDDS-10725. TestContentGenerator#writeWithHsync fails with
Java 17 (#6570)
add d10a822ddd HDDS-10653. Support custom metadata for MPU key (#6489)
add 8795bcb0fe HDDS-10545. Repeated tests stop if fork does not return
(#6393)
add 89afdd320c HDDS-10726. TestAuditParser.testLoadCommand fails with Java
11+ (#6571)
add dfe1ea5902 HDDS-10735. `repeat-acceptance-test` workflow always builds
the branch it is run from. (#6576)
add b80d2669d8 HDDS-10723. Trigger native check in more cases (#6569)
add dec977bde9 HDDS-10442. [hsync] Add a Freon tool to measure client to
DataNode round-trip latency (#6297) (#6562)
add 42f749ee3c HDDS-10712. Add a dashboard for chunks & volume IO. (#6554)
add 8c003445b9 HDDS-10713. Run JUnit tests with Java 17 (#6575)
add 95b2caac7b HDDS-10628. Display if safemode exit was via force exit
command. (#6494)
add fd30db5e22 HDDS-10741. Mark value objects as Immutable (#6579)
add dd86223dc9 HDDS-10582. Intermittent timeout during waitForReplicaCount
in TestReconAndAdminContainerCLI (#6585)
add d767b0dc26 HDDS-10691. CRYPTO_COMPLIANCE tag for cryptography
parameters (#6538)
add 383c83fa5f HDDS-10745. Do not use BitSet for OzoneAcl.aclBitSet.
(#6581)
add 97b5fa310e HDDS-10752. OmBucketInfo/OmMultipartKeyInfo/OmPrefixInfo
should implement CopyObject. (#6590)
add 32b16923fe HDDS-10753. OmKeyInfo#acls and WithMetadata#metadata are
not thread safe. (#6591)
add 25599d6663 HDDS-10701. Create config option for keystores (#6548)
add c306b25e21 HDDS-7791. Support key ownership (#4188)
add 0984893ae0 HDDS-10765. Bump maven-jar-plugin to 3.4.1 (#6601)
add 4a14ad7fba HDDS-10768. Bump guava to 32.1.3-jre (#6598)
add 294710a7e6 HDDS-10756. Avoid proto2 ByteString#copyFrom(byte[]).
(#6592)
add b37d99db42 HDDS-10771. Bump jetty to 9.4.54.v20240208 (#6600)
add 06c7cb419c HDDS-10761. Add raft close threshold config to OM
RaftProperties (#6594)
add 3d3f11c5fe HDDS-10732. Add CRYPTO_COMPLIANCE tag to client checksum
configs (#6606)
add f5c2d98a2e HDDS-10769. Integration check no longer needs Ozone repo
(#6604)
add 72ea38f766 HDDS-5570. security/SecuringOzoneHTTP.md translation (#6584)
add 1eaddc43d6 HDDS-10657. Design Doc for overwriting a key if it has not
changed (#6482)
add a67e43469c HDDS-10746. Make ssl.server.keystore.type and
ssl.server.truststore.type configurable (#6595)
add fdd203770e HDDS-10744. Standardize byte[] to String conversion for
LiveFileMetaData (#6580)
add 81a3634a23 HDDS-10751. Detect missing EC padding blocks (#6593)
add ce864a7bb4 HDDS-10773. Simplify OM RaftProperties formatting (#6605)
add 59f5879d0d HDDS-5571. tools/Admin.md translation (#6586)
add a0f8809b56 HDDS-10409. Decommissioning of datanodes - Duplicate ozone
nodes display as dead in Ozone Recon. (#6360)
add fe1b5b6335 HDDS-10709. Intermittent failure in
TestContainerBalancerOperations (#6582)
add 78a7e7ae1a HDDS-9889. Refactor tests related to dynamical adaptation
for datanode limits in ContainerBalancer (#5758)
add a658802d62 HDDS-10783. Close SstFileReaderIterator in
RocksDBCheckpointDiffer (#6616)
add a3687695ab HDDS-10784. Multipart upload to encrypted bucket fails with
ClassCastException (#6618)
add a433969641 HDDS-9761. Intermittent failure in
TestOzoneManagerHAWithStoppedNodes due to OMLeaderNotReadyException (#6621)
add f259bdae6b HDDS-10464. Add integration tests for fail-early (#6555)
add 8d781902fe HDDS-10792. Bump Netty to 4.1.109.Final (#6622)
add 1324e95f36 HDDS-10720. Datanode volume DU reserved percent should have
a non-zero default value. (#6561)
add 73b35ece7b HDDS-10796. Avoid dummy KeyOutputStream in
OzoneOutputStreamStub (#6623)
add 18a3294347 HDDS-10797. Remove unused UserGroupInformation object in
DataNode token verifier. (#6624)
add e1d2d9c832 HDDS-10780. NullPointerException in watchForCommit (#6627)
add c435a7f0e4 HDDS-10787. Updated rocksdb-checkpoint-differ to use
managed RocksDB objects (#6625)
add 4e9dc2faae HDDS-10798. OMLeaderNotReadyException exception on switch
leader (#6626)
add 8d2569da59 HDDS-10097. Intermittent ManagedChannel not shutdown
properly in TestWatchForCommit (#6620)
add 9cec258ce3 HDDS-10806. Bump Bouncy Castle to 1.78.1 (#6632)
add 46482459e1 HDDS-10807. Bump aspectj-maven-plugin to 1.15.0 (#6636)
add 2815d71e96 HDDS-10808. Bump dnsjava to 2.1.9 (#6633)
add a15cc3e382 HDDS-10803. HttpServer fails to start with wildcard
principal (#6631)
add 45b73794f2 HDDS-10800. Follower SCMs should not log Replication
Manager is not ready (#6629)
add 4f949f979c HDDS-10809. Bump maven-deploy-plugin to 3.1.2 (#6634)
add 21fa62fdc9 HDDS-10810. Bump maven-remote-resources-plugin to 1.7.0
(#6635)
add 1cbee607f8 HDDS-10802. Improve logging for signature verification
(#6630)
add fff2b6cbb6 HDDS-10815. Bump Spring Framework to 5.3.34 (#6643)
add f61f56d338 HDDS-10542. Replace remaining GSON usage with Jackson.
(#6500)
add d49a2b6b19 HDDS-10465. Change ozone.client.bytes.per.checksum default
to 16KB (#6331)
add db2e507504 HDDS-10793. HttpFS gateway should throw unsupported
operation for some operations (#6645)
add bab281cf0e HDDS-10814. Log exception stack trace in addSCM only at
debug level (#6642)
add ff78dc83a7 HDDS-10777. S3 Gateway error when parsing XML concurrently
(#6609)
add acb49b97f7 HDDS-10820. Freon tool DN-Echo to support GRPC and Ratis
read/write mode (#6647)
add 673c112ca0 HDDS-10834. Revert snapshot diff output change added in
HDDS-9360 (#6660)
add 7427026300 HDDS-10772. [Ozone-Streaming] Stream write metric is wrong
(#6610)
add 9074b8df0e HDDS-10608. Recon can't get full key when using Recon API.
(#6492)
add 3f1a7edf6e HDDS-10696. Fix test failure caused by empty snapshot
installation (#6659)
add 3096887c5c HDDS-10838. Merge latest master change into HDDS-7593
feature branch.
No new revisions were added by this update.
Summary of changes:
.github/dependabot.yml | 21 +-
.github/workflows/ci.yml | 35 +-
.github/workflows/dependabot-ci.yml | 78 ---
.github/workflows/repeat-acceptance.yml | 2 +
.mvn/extensions.xml | 2 +-
dev-support/ci/categorize_basic_checks.sh | 49 +-
dev-support/ci/selective_ci_checks.bats | 6 +-
dev-support/ci/selective_ci_checks.sh | 19 +-
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 10 +-
.../hadoop/hdds/scm/client/ClientTrustManager.java | 2 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 10 +-
.../hdds/scm/storage/ECBlockOutputStream.java | 24 +-
.../storage/TestBlockOutputStreamCorrectness.java | 83 +++
.../google/protobuf/Proto2Utils.java} | 29 +-
.../java/com/google/protobuf}/package-info.java | 4 +-
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 5 +
.../java/org/apache/hadoop/hdds/HddsUtils.java | 4 +-
.../hdds/client/DefaultReplicationConfig.java | 2 +
.../hadoop/hdds/client/ECReplicationConfig.java | 15 +-
.../hadoop/hdds/client/RatisReplicationConfig.java | 2 +
.../hdds/client/StandaloneReplicationConfig.java | 2 +
.../hadoop/hdds/fs/CachingSpaceUsageSource.java | 14 +-
.../org/apache/hadoop/hdds/scm/ScmConfigKeys.java | 12 +-
.../apache/hadoop/hdds/scm/client/ScmClient.java | 2 +-
.../protocol/StorageContainerLocationProtocol.java | 2 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 4 +-
.../security/ssl/PemFileBasedKeyStoresFactory.java | 13 +-
.../org/apache/hadoop/hdds/server/JsonUtils.java | 20 +
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 6 +-
.../common/src/main/resources/ozone-default.xml | 104 ++--
.../org/apache/hadoop/hdds/JsonTestUtils.java} | 69 +--
.../hdds/fs/TestCachingSpaceUsageSource.java | 14 +
.../org/apache/hadoop/hdds/conf/ConfigTag.java | 3 +-
.../container/common/impl/HddsDispatcher.java | 3 +-
.../commandhandler/DeleteBlocksCommandHandler.java | 36 +-
.../transport/server/ratis/XceiverServerRatis.java | 2 +-
.../ozone/container/common/volume/VolumeUsage.java | 39 +-
.../ECReconstructionCommandInfo.java | 13 +-
.../ECReconstructionCoordinator.java | 123 ++---
.../KeyValueContainerMetadataInspector.java | 178 +++----
.../container/keyvalue/impl/BlockManagerImpl.java | 2 +-
.../keyvalue/impl/KeyValueStreamDataChannel.java | 7 +-
.../keyvalue/impl/StreamDataChannelBase.java | 3 +
.../keyvalue/interfaces/ChunkManager.java | 2 +-
.../metadata/DatanodeStoreSchemaThreeImpl.java | 5 +-
.../commands/ReconstructECContainersCommand.java | 28 +-
.../container/common/impl/TestHddsDispatcher.java | 4 +-
.../TestDeleteBlocksCommandHandler.java | 45 +-
.../states/endpoint/TestHeartbeatEndpointTask.java | 4 +-
.../volume/TestCapacityVolumeChoosingPolicy.java | 4 +
.../common/volume/TestReservedVolumeSpace.java | 60 ++-
.../volume/TestRoundRobinVolumeChoosingPolicy.java | 5 +
.../TestKeyValueContainerMetadataInspector.java | 94 ++--
.../replication/TestReplicationSupervisor.java | 3 +-
.../TestReconstructionECContainersCommands.java | 13 +-
.../design/overwrite-key-only-if-unchanged.md | 199 +++++++
hadoop-hdds/docs/content/interface/ReconApi.md | 33 ++
...ecuringOzoneHTTP.md => SecuringOzoneHTTP.zh.md} | 96 ++--
hadoop-hdds/docs/content/tools/Admin.zh.md | 35 ++
.../static/swagger-resources/recon-api.yaml | 19 +-
.../apache/hadoop/hdds/conf/HddsConfServlet.java | 10 +-
...inerLocationProtocolClientSideTranslatorPB.java | 3 +-
.../security/token/CompositeTokenVerifier.java | 4 +-
.../hdds/security/token/NoopTokenVerifier.java | 4 +-
.../security/token/ShortLivedTokenVerifier.java | 8 +-
.../hadoop/hdds/security/token/TokenVerifier.java | 9 +-
.../hadoop/hdds/server/http/BaseHttpServer.java | 6 +-
.../hadoop/hdds/utils/RocksDBStoreMetrics.java | 4 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 2 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 8 +-
.../hadoop/hdds/conf/TestHddsConfServlet.java | 5 +-
.../token/TestOzoneBlockTokenSecretManager.java | 16 +-
.../hdds/security/token/TokenVerifierTests.java | 14 +-
.../apache/hadoop/hdds/server/TestJsonUtils.java | 5 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 8 +
hadoop-hdds/hadoop-dependency-test/pom.xml | 4 +
.../src/main/proto/ScmAdminProtocol.proto | 1 +
.../src/main/proto/DatanodeClientProtocol.proto | 1 +
.../utils/db/managed/ManagedSstFileReader.java | 21 +-
hadoop-hdds/rocks-native/pom.xml | 5 -
hadoop-hdds/rocksdb-checkpoint-differ/pom.xml | 37 --
.../org/apache/ozone/rocksdb/util/RdbUtil.java | 3 +-
.../ozone/rocksdb/util/SstFileSetReader.java | 33 +-
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 84 ++-
.../org/apache/ozone/rocksdiff/RocksDiffUtils.java | 9 +-
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 125 +++--
.../ContainerBalancerSelectionCriteria.java | 12 +-
.../container/balancer/ContainerBalancerTask.java | 37 +-
.../scm/container/balancer/FindSourceGreedy.java | 8 +-
.../scm/container/balancer/FindSourceStrategy.java | 10 +
.../replication/ECUnderReplicationHandler.java | 8 +-
.../container/replication/ReplicationManager.java | 13 +-
.../hadoop/hdds/scm/ha/SCMRatisServerImpl.java | 9 +-
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 21 +-
.../hadoop/hdds/scm/ha/io/BigIntegerCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/BooleanCodec.java | 11 +-
.../apache/hadoop/hdds/scm/ha/io/EnumCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/IntegerCodec.java | 4 +-
.../apache/hadoop/hdds/scm/ha/io/LongCodec.java | 4 +-
.../hdds/scm/ha/io/ManagedSecretKeyCodec.java | 2 +-
.../apache/hadoop/hdds/scm/ha/io/StringCodec.java | 4 +-
.../hdds/scm/ha/io/X509CertificateCodec.java | 4 +-
.../hdds/scm/node/NodeDecommissionManager.java | 95 +++-
.../apache/hadoop/hdds/scm/node/NodeManager.java | 11 +
.../hadoop/hdds/scm/node/NodeStateManager.java | 4 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 32 ++
.../hadoop/hdds/scm/node/states/NodeStateMap.java | 11 +
...inerLocationProtocolServerSideTranslatorPB.java | 2 +-
.../hdds/scm/safemode/SCMSafeModeManager.java | 30 +-
.../hdds/scm/server/SCMClientProtocolServer.java | 4 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 7 +
.../hdds/scm/server/StorageContainerManager.java | 7 +-
.../main/resources/webapps/scm/scm-overview.html | 4 +
.../hdds/scm/container/balancer/MockedSCM.java | 318 ++++++++++++
.../TestContainerBalancerDatanodeNodeLimit.java | 186 +++++++
.../balancer/TestContainerBalancerTask.java | 144 ++----
.../scm/container/balancer/TestableCluster.java | 253 +++++++++
.../replication/TestECUnderReplicationHandler.java | 7 +-
.../replication/TestReplicationManager.java | 5 +-
.../hdds/scm/ha/io/TestX509CertificateCodec.java | 3 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 365 ++++++++++++-
.../hdds/scm/cli/ContainerOperationClient.java | 4 +-
.../scm/cli/datanode/MaintenanceSubCommand.java | 8 +-
.../scm/cli/pipeline/CreatePipelineSubcommand.java | 10 +-
.../cli/datanode/TestMaintenanceSubCommand.java | 5 +-
.../org/apache/hadoop/ozone/client/BucketArgs.java | 2 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 66 +--
.../org/apache/hadoop/ozone/client/OzoneKey.java | 22 +-
.../hadoop/ozone/client/OzoneKeyDetails.java | 4 +-
.../org/apache/hadoop/ozone/client/VolumeArgs.java | 2 +
.../client/checksum/ECBlockChecksumComputer.java | 6 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 2 +-
.../ozone/client/protocol/ClientProtocol.java | 25 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 74 ++-
hadoop-ozone/common/pom.xml | 5 -
.../java/org/apache/hadoop/ozone/OzoneAcl.java | 143 ++++--
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 13 +-
.../hadoop/ozone/om/helpers/BasicOmKeyInfo.java | 47 +-
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 14 +-
.../hadoop/ozone/om/helpers/OmDirectoryInfo.java | 20 +
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 14 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 35 +-
.../ozone/om/helpers/OmMultipartKeyInfo.java | 4 +-
.../hadoop/ozone/om/helpers/OmVolumeArgs.java | 105 ++--
.../hadoop/ozone/om/helpers/S3SecretValue.java | 4 +-
.../apache/hadoop/ozone/om/helpers/VolumeArgs.java | 140 -----
.../hadoop/ozone/om/helpers/WithMetadata.java | 8 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 27 +-
.../ozone/security/acl/IAccessAuthorizer.java | 9 +-
.../ozone/snapshot/SnapshotDiffReportOzone.java | 16 +-
.../org/apache/hadoop/ozone/util/PayloadUtils.java | 66 +++
.../ozone/om/helpers/TestOmMultipartKeyInfo.java | 2 +-
.../hadoop/ozone/om/helpers/TestOzoneAclUtil.java | 4 +-
.../apache/hadoop/ozone/util/TestPayloadUtils.java | 32 +-
hadoop-ozone/dev-support/checks/_lib.sh | 15 +
.../dev-support/checks/_mvn_unit_report.sh | 4 +-
hadoop-ozone/dev-support/checks/acceptance.sh | 52 +-
hadoop-ozone/dev-support/checks/junit.sh | 2 +-
hadoop-ozone/dev-support/checks/native.sh | 2 +-
...on => Datanode Chunk Read_Write Dashboard.json} | 569 ++++++++++-----------
.../dist/src/main/compose/common/s3a-test.sh | 127 +++++
.../compose/ozone/{test-hadoop.sh => test-s3a.sh} | 5 +-
.../test.sh => ozonesecure-ha/test-s3a.sh} | 7 +-
.../src/main/compose/ozonesecure/docker-config | 5 +-
.../dist/src/main/compose/ozonesecure/test.sh | 9 +
hadoop-ozone/dist/src/main/compose/test-all.sh | 7 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 4 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 3 +
.../src/main/smoketest/admincli/pipeline.robot | 24 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 1 +
.../main/smoketest/httpfs/operations_tests.robot | 10 -
.../src/main/smoketest/s3/MultipartUpload.robot | 18 +-
.../dist/src/main/smoketest/s3/objectputget.robot | 5 +-
.../src/main/smoketest/security/S3-secret.robot | 2 +
.../smoketest/security/ozone-secure-owner-s3.robot | 63 +++
.../smoketest/security/ozone-secure-tenant.robot | 9 +-
.../apache/ozone/fs/http/server/HttpFSServer.java | 12 +-
.../fs/ozone/AbstractOzoneFileSystemTest.java | 97 ++++
.../hadoop/hdds/scm/TestContainerSmallFile.java | 2 +-
.../hdds/scm/TestStorageContainerManager.java | 7 +-
.../scm/node/TestDecommissionAndMaintenance.java | 219 +++++++-
.../hdds/scm/storage/TestContainerCommandsEC.java | 28 +-
.../ozone/TestContainerBalancerOperations.java | 20 +-
.../hadoop/ozone/TestOzoneConfigurationFields.java | 2 -
.../client/rpc/TestOzoneRpcClientAbstract.java | 186 ++++++-
.../ozone/client/rpc/TestWatchForCommit.java | 167 +++---
.../apache/hadoop/ozone/container/TestHelper.java | 2 +-
.../hadoop/ozone/freon/TestDNRPCLoadGenerator.java | 37 +-
.../hadoop/ozone/om/TestAddRemoveOzoneManager.java | 2 -
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 3 +-
.../hadoop/ozone/om/TestOmBlockVersioning.java | 1 +
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 2 +
.../om/TestOzoneManagerHAWithStoppedNodes.java | 2 +-
.../hadoop/ozone/om/snapshot/TestOmSnapshot.java | 16 +-
.../om/snapshot/TestOmSnapshotFileSystem.java | 2 +
.../om/snapshot/TestSnapshotDeletingService.java | 89 ++--
.../ozone/recon/TestReconAndAdminContainerCLI.java | 4 +-
.../ozone/recon/TestReconContainerEndpoint.java | 222 ++++++++
.../hadoop/ozone/recon/TestReconScmSnapshot.java | 50 ++
.../src/test/resources/core-site.xml | 13 +-
.../src/main/proto/OmClientProtocol.proto | 5 +
.../hadoop/ozone/om/helpers/OmPrefixInfo.java | 12 +-
.../hadoop/ozone/om/helpers/OzoneAclStorage.java | 4 +-
.../hadoop/ozone/om/helpers/TestOmPrefixInfo.java | 2 +-
.../apache/hadoop/ozone/common/PayloadUtils.java | 44 --
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 1 +
.../hadoop/ozone/om/OMDBCheckpointServlet.java | 13 +-
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 10 +-
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 18 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 26 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 225 ++++----
.../ozone/om/ratis/OzoneManagerStateMachine.java | 2 +-
.../hadoop/ozone/om/request/RequestAuditor.java | 7 +
.../om/request/file/OMDirectoryCreateRequest.java | 1 +
.../file/OMDirectoryCreateRequestWithFSO.java | 1 +
.../ozone/om/request/file/OMFileRequest.java | 3 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 1 +
.../S3InitiateMultipartUploadRequest.java | 3 +
.../S3InitiateMultipartUploadRequestWithFSO.java | 3 +
.../S3MultipartUploadCompleteRequest.java | 111 +++-
.../S3MultipartUploadCompleteRequestWithFSO.java | 39 +-
.../om/request/util/OMEchoRPCWriteRequest.java | 8 +-
.../S3MultipartUploadCompleteResponse.java | 8 +
.../S3MultipartUploadCompleteResponseWithFSO.java | 45 +-
.../om/service/AbstractKeyDeletingService.java | 20 +-
.../hadoop/ozone/om/snapshot/ReferenceCounted.java | 8 +-
.../om/snapshot/ReferenceCountedCallback.java} | 7 +-
.../hadoop/ozone/om/snapshot/SnapshotCache.java | 141 ++---
.../ozone/om/snapshot/SnapshotDiffManager.java | 2 +-
.../protocolPB/OzoneManagerRequestHandler.java | 7 +-
.../OzoneDelegationTokenSecretManager.java | 2 +
.../hadoop/ozone/om/TestBucketManagerImpl.java | 5 +
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 3 +
.../apache/hadoop/ozone/om/TestTrashService.java | 2 +
.../om/ratis/TestOzoneManagerStateMachine.java | 20 +
.../ozone/om/request/OMRequestTestUtils.java | 23 +-
.../TestS3InitiateMultipartUploadRequest.java | 12 +-
...estS3InitiateMultipartUploadRequestWithFSO.java | 11 +-
.../s3/multipart/TestS3MultipartRequest.java | 52 +-
.../TestS3MultipartUploadCompleteRequest.java | 21 +-
.../s3/multipart/TestS3MultipartResponse.java | 5 +-
.../ozone/om/service/TestKeyDeletingService.java | 2 +
.../service/TestMultipartUploadCleanupService.java | 3 +
.../om/service/TestOpenKeyCleanupService.java | 5 +
.../ozone/om/snapshot/TestSnapshotCache.java | 17 +-
.../ozone/om/snapshot/TestSnapshotDiffManager.java | 5 +-
.../ozone/om/snapshot/TestSstFilteringService.java | 3 +
.../security/acl/TestOzoneNativeAuthorizer.java | 1 +
.../hadoop/ozone/security/acl/TestParentAcl.java | 1 +
.../hadoop/ozone/security/acl/TestVolumeOwner.java | 2 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 2 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 2 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 2 +-
.../apache/hadoop/ozone/recon/ReconConstants.java | 5 +
.../org/apache/hadoop/ozone/recon/ReconUtils.java | 142 ++++-
.../hadoop/ozone/recon/api/ContainerEndpoint.java | 51 +-
.../hadoop/ozone/recon/api/NSSummaryEndpoint.java | 12 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 154 ++++++
.../recon/api/handlers/BucketEntityHandler.java | 13 +-
.../recon/api/handlers/DirectoryEntityHandler.java | 13 +-
.../ozone/recon/api/handlers/EntityHandler.java | 2 +-
.../ozone/recon/api/handlers/KeyEntityHandler.java | 2 +-
.../recon/api/handlers/RootEntityHandler.java | 15 +-
.../recon/api/handlers/UnknownEntityHandler.java | 2 +-
.../recon/api/handlers/VolumeEntityHandler.java | 13 +-
.../hadoop/ozone/recon/api/types/AclMetadata.java | 4 +-
.../ozone/recon/api/types/DatanodeMetadata.java | 16 +
.../ozone/recon/api/types/DatanodesResponse.java | 18 +
.../hadoop/ozone/recon/api/types/KeyMetadata.java | 11 +
.../hadoop/ozone/recon/api/types/KeysResponse.java | 9 +-
.../hadoop/ozone/recon/api/types/NSSummary.java | 15 +-
...se.java => RemoveDataNodesResponseWrapper.java} | 38 +-
.../hadoop/ozone/recon/codec/NSSummaryCodec.java | 17 +-
.../hadoop/ozone/recon/heatmap/HeatMapUtil.java | 2 +-
.../ozone/recon/scm/ReconDeadNodeHandler.java | 1 -
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 23 +
.../recon/spi/ReconNamespaceSummaryManager.java | 3 +
.../spi/impl/ReconNamespaceSummaryManagerImpl.java | 12 +-
.../recon/tasks/NSSummaryTaskDbEventHandler.java | 2 +
.../hadoop/ozone/recon/api/TestEndpoints.java | 144 +++++-
.../recon/api/TestNSSummaryDiskUsageOrdering.java | 421 +++++++++++++++
.../recon/api/TestNSSummaryEndpointWithFSO.java | 195 ++++++-
.../recon/api/TestNSSummaryEndpointWithLegacy.java | 76 ++-
.../api/TestNSSummaryEndpointWithOBSAndLegacy.java | 120 +++--
.../impl/TestReconNamespaceSummaryManagerImpl.java | 6 +-
.../recon/tasks/TestNSSummaryTaskWithFSO.java | 54 +-
hadoop-ozone/s3gateway/pom.xml | 5 -
.../hadoop/ozone/s3/commontypes/KeyMetadata.java | 13 +
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 10 +-
...CompleteMultipartUploadRequestUnmarshaller.java | 6 +-
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 16 +
.../endpoint/MultiDeleteRequestUnmarshaller.java | 6 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 42 +-
.../endpoint/PutBucketAclRequestUnmarshaller.java | 6 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 11 +-
.../hadoop/ozone/client/OzoneBucketStub.java | 86 +++-
.../hadoop/ozone/client/OzoneOutputStreamStub.java | 18 -
.../hadoop/ozone/s3/endpoint/TestBucketList.java | 34 +-
...CompleteMultipartUploadRequestUnmarshaller.java | 44 ++
.../s3/endpoint/TestMultipartUploadComplete.java | 59 ++-
.../ozone/s3/endpoint/TestPermissionCheck.java | 3 +-
.../hadoop/ozone/debug/FindMissingPadding.java | 277 ++++++++++
.../hadoop/ozone/freon/DNRPCLoadGenerator.java | 49 +-
.../apache/hadoop/ozone/freon/OmKeyGenerator.java | 2 +
.../hadoop/ozone/freon/OmRPCLoadGenerator.java | 20 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 2 +-
pom.xml | 51 +-
307 files changed, 7584 insertions(+), 2625 deletions(-)
delete mode 100644 .github/workflows/dependabot-ci.yml
copy
hadoop-hdds/common/src/main/java/{org/apache/hadoop/hdds/ComponentVersion.java
=> com/google/protobuf/Proto2Utils.java} (60%)
copy hadoop-hdds/{test-utils/src/main/java/org/apache/hadoop/test =>
common/src/main/java/com/google/protobuf}/package-info.java (91%)
copy
hadoop-hdds/common/src/{main/java/org/apache/hadoop/hdds/server/JsonUtils.java
=> test/java/org/apache/hadoop/hdds/JsonTestUtils.java} (54%)
create mode 100644
hadoop-hdds/docs/content/design/overwrite-key-only-if-unchanged.md
copy hadoop-hdds/docs/content/security/{SecuringOzoneHTTP.md =>
SecuringOzoneHTTP.zh.md} (54%)
create mode 100644 hadoop-hdds/docs/content/tools/Admin.zh.md
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/MockedSCM.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancerDatanodeNodeLimit.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestableCluster.java
delete mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/VolumeArgs.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/util/PayloadUtils.java
copy
hadoop-hdds/framework/src/test/java/org/apache/hadoop/ozone/audit/DummyAction.java
=>
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/util/TestPayloadUtils.java
(62%)
copy hadoop-ozone/dist/src/main/compose/common/grafana/dashboards/{Ozone - S3
Dashboard.json => Datanode Chunk Read_Write Dashboard.json} (69%)
create mode 100644 hadoop-ozone/dist/src/main/compose/common/s3a-test.sh
copy hadoop-ozone/dist/src/main/compose/ozone/{test-hadoop.sh => test-s3a.sh}
(92%)
mode change 100755 => 100644
copy hadoop-ozone/dist/src/main/compose/{ozonesecure-mr/test.sh =>
ozonesecure-ha/test-s3a.sh} (89%)
mode change 100755 => 100644
create mode 100644
hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-owner-s3.robot
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconContainerEndpoint.java
delete mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/common/PayloadUtils.java
copy
hadoop-ozone/{tools/src/main/java/org/apache/hadoop/ozone/repair/om/package-info.java
=>
ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCountedCallback.java}
(81%)
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/{KeysResponse.java
=> RemoveDataNodesResponseWrapper.java} (56%)
create mode 100644
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/api/TestNSSummaryDiskUsageOrdering.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/FindMissingPadding.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
