This is an automated email from the ASF dual-hosted git repository.
licheng pushed a change to branch HDDS-2823
in repository https://gitbox.apache.org/repos/asf/hadoop-ozone.git.
from 2e3e064 HDDS-3837 Add isLeader check in SCMHAManager. (#1191)
new ad71958 HDDS-3798. Display more accurate timestamp in recon Web
(#1201)
new c219647 HDDS-3765. Fluentd writing to secure Ozone S3 API fails with
500 Error. (#1179)
new f322918 HDDS-3967. Remove leftover debug setting (#1202)
new d19f322 HDDS-3968. LDB scan fails to read from transactionInfoTable.
(#1205)
new dddc85d HDDS-2767. security/SecuringTDE.md (#1184)
new c4f1446 HDDS-3958. Intermittent failure in Recon acceptance test due
to mixed stdout and stderr (#1200)
new f13a221 HDDS-3923. Display the safemode status on scm page (#1165)
new e9eb482 HDDS-3807. Propagate raft log disks info to SCM from
datanode. (#1107)
new 82c402e HDDS-3966. Disable flaky TestOMRatisSnapshots
new 168ba19 HDDS-3824: OM read requests should make SCM#refreshPipeline
outside BUCKET_LOCK (#1164)
new e7cac9a HDDS-3926. OM Token Identifier table should use in-house
serialization. (#1182)
new 97fdf92 HDDS-3612. Allow mounting bucket under other volume (#1104)
new d3dda10 HDDS-3964. Ratis config key mismatch (#1204)
new ed8df6b HDDS-3855. Add upgrade smoketest (#1142)
new 888abde HDDS-3965. SCM failed to start up for duplicated pipeline
detected. (#1210)
new db3cef0 HDDS-3741. Reload old OM state if Install Snapshot from
Leader fails (#1129)
new 51a3582 HDDS-3955. Unable to list intermediate paths on keys created
using S3G. (#1196)
new 5d6f832 HDDS-3806. Support recognize aws v2 Authorization header.
(#1098)
new 23dafc5 HDDS-3984. Support filter and search the columns in recon UI
(#1218)
new 26a38a2 HDDS-3987. Encrypted bucket creation failed with
INVALID_REQUEST Encryption cannot be set for bucket links (#1221)
new fb3cfe9 HDDS-3982. Disable moveToTrash in o3fs and ofs temporarily
(#1215)
new c53c0e0 Update ratis to 1.0.0 (#1222)
new 8d8c5cc HDDS-3813. Upgrade Ratis third-party, too (#1229)
new a265f7f HDDS-3986. Frequent failure in
TestCommitWatcher#testReleaseBuffersOnException (#1220)
new eb765fe HDDS-3989. Display revision and build date of DN in recon UI
(#1226)
new 7e8f2ee HDDS-3992. Remove project skeleton of in-place upgrade
feature (#1225)
new 093cea1 HDDS-3892. Datanode initialization is too slow when there are
thousan… (#1147)
new a0dcdd3 HDDS-3989. Addendum: revert proto.lock file (#1226)
new 4ff513f HDDS-3980. Correct the toString of RangeHeader (#1213)
new 7f699fa HDDS-3991. Ignore protobuf lock files (#1224)
new 364468c HDDS-3933. Fix memory leak because of too many Datanode State
Machine Thread (#1185)
new afe9988 HDDS-2770. security/SecurityAcls.md (#1190)
new 5c541bc HDDS-3718: Improve OmKeyLocationInfoGroup internal data
structure (#1023)
new 8cb1e85 Remove optional jersey-json dependency (#1238)
new d144473 HDDS-3993. Create volume required for S3G during OM startup.
(#1227)
new 12d1d35 HDDS-3969. Add validName check for FileSystem requests (#1211)
new 0d61e47 HDDS-4003. Delete the redundant word of the description
(#1240)
new a5b2093 HDDS-3827. Intermittent failure in
TestKeyManagerUnit#listMultipartUploads (#1239)
new 51a0c93 HDDS-3998. Shorten Ozone FS Hadoop compatibility module names
(#1237)
new 83bff20 HDDS-4006. Disallow MPU on encrypted buckets. (#1241)
new eff1f43 HDDS-4008. Recon should fallback to ozone.om.service.ids when
the internal service id is not defined. (#1243)
new 829143e HDDS-3658. Stop to persist container related pipeline info of
each ke… (#1012)
new 7ff95c0 HDDS-4018. Datanode log spammed by NPE (#1250)
new 5486793 HDDS-3999. OM Shutdown when Commit part tries to commit the
part, after abort upload. (#1244)
new b8b1f01 HDDS-3996. Missing TLS client configurations to allow
ozone.grpc.tls.… (#1234)
new c884133 HDDS-3997. Ozone certificate needs additional flags and SAN
extension… (#1235)
new c059c83 HDDS-4007. Generate encryption info for the bucket outside
bucket lock. (#1242)
new e41edd6 HDDS-4025. Add test for creating encrypted key (#1254)
new d757253 HDDS-3973. Update main feature design status. (#1207)
new 60b328b HDDS-3877. Do not fail CI check for log upload failure (#1209)
new 7545f18 HDDS-3975. Use Duration for time in RatisClientConfig (#1217)
new 749d874 HDDS-4022. Ozone s3 API return 400 Bad Request for
head-bucket for non existing bucket. (#1251)
new 2286ee5 HDDS-3905. Show status of OM in the OM web ui (#1152)
new 5dd1015 HDDS-4000. Split acceptance tests to reduce CI feedback time
(#1236)
new 64d76ed HDDS-4017. Acceptance check may run against wrong commit
(#1249)
new 1fa9e78 HDDS-4026. Dir rename failed when sets
'ozone.om.enable.filesystem.paths' to true (#1256)
new 2b78046 HDDS-4032. Run author check without docker (#1262)
new dd7eaf1 HDDS-4030. Remember the selected columns and make the X-axis
scrollable in recon datanodes UI (#1259)
new d6d7d81 HDDS-4011. Update S3 related documentation. (#1245)
new d80f873 HDDS-4038. Eliminate GitHub check warnings (#1268)
new 6b36e21 HDDS-4031. Run shell tests in CI (#1261)
new 91a43ec HDDS-4041. Ozone /conf endpoint triggers kerberos replay
error when SPNEGO is enabled. (#1267)
new dc72bb9 HDDS-3511. Fix javadoc comment in OmMetadataManager (#1247)
new 08c6512 HDDS-4019. Show the storageDir while need init om or scm
(#1248)
new 52fc2ac HDDS-3809. Make number of open containers on a datanode a
function of no of volumes reported by it. (#1081)
new acd3480 HDDS-4024. Avoid while loop too soon when exception happen
(#1253)
new 7537e34 HDDS-4027. Suppress ERROR message when SCM attempt to create
additional pipelines. (#1265)
new c17504b HDDS-3423. Enabling TestContainerReplicationEndToEnd and
addressing failures (#1260)
new 00afc8d HDDS-4033. Make the acceptance test reports hierarchical
(#1263)
new 8b7e0d3 HDDS-3970. Enabling TestStorageContainerManager with all
failures addressed (#1257)
new 19eaad1 HDDS-4045. Add more ignore rules to the RAT ignore list
(#1273)
new f511340 HDDS-3990. Test Kubernetes examples with acceptance tests
(#1223)
new 036c142 HDDS-4047. OzoneManager met NPE exception while
getServiceList (#1277)
new da4fccb HDDS-4052. Remove master/slave terminology from Ozone (#1281)
new 26ffe01 HDDS-4021. Organize Recon DBs into a 'DBDefinition'. (#1255)
new 979faa7 HDDS-4029. Recon unable to add a new container which is in
CLOSED state. (#1258)
new dd28dc9 HDDS-4058. Wrong use of AtomicBoolean in HddsDatanodeService
(#1284)
new 040577c HDDS-3446. Enable TestOzoneManagerRestart and address any
failure. (#1279)
new 4d186d4 HDDS-3994. Make retry policy can be set by configuration.
(#1231)
new a92a2f7 Update PipelineManagerV2 with getNumHealthyVolumes interface.
The 80 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/comments.yaml | 2 +-
.github/workflows/post-commit.yml | 154 ++++-
.../hadoop/hdds/protocol/DatanodeDetails.java | 88 ++-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 141 ++---
.../hadoop/hdds/ratis/conf/RatisClientConfig.java | 110 ++--
.../RequestTypeDependentRetryPolicyCreator.java | 120 ++++
.../retrypolicy/RetryLimitedPolicyCreator.java | 47 ++
.../hdds/ratis/retrypolicy/RetryPolicyCreator.java | 32 +-
.../hdds/ratis/retrypolicy}/package-info.java | 12 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 3 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 5 +
.../common/src/main/resources/ozone-default.xml | 27 +-
.../hdds/ratis/conf/TestRaftClientConfig.java | 62 ++
.../hdds/ratis/conf/TestRatisClientConfig.java | 68 +++
.../apache/hadoop/ozone/HddsDatanodeService.java | 18 +-
.../common/impl/StorageLocationReport.java | 11 +-
.../common/statemachine/DatanodeStateMachine.java | 48 +-
.../common/statemachine/StateContext.java | 31 +
.../CreatePipelineCommandHandler.java | 3 +-
.../states/datanode/RunningDatanodeState.java | 14 +-
.../common/transport/server/XceiverServerSpi.java | 12 +-
.../server/ratis/ContainerStateMachine.java | 12 +-
.../transport/server/ratis/XceiverServerRatis.java | 62 +-
.../container/common/utils/ContainerCache.java | 71 ++-
.../container/common/volume/MutableVolumeSet.java | 10 +-
.../ozone/container/ozoneimpl/ContainerReader.java | 2 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 30 +-
.../replication/GrpcReplicationClient.java | 29 +-
.../replication/SimpleContainerDownloader.java | 30 +-
.../ozone/container/common/TestContainerCache.java | 52 ++
.../common/statemachine/TestStateContext.java | 85 +++
.../states/datanode/TestRunningDatanodeState.java | 90 +++
.../container/ozoneimpl/TestContainerReader.java | 69 ++-
.../container/ozoneimpl/TestOzoneContainer.java | 51 +-
hadoop-hdds/docs/content/design/multiraft.md | 2 +-
hadoop-hdds/docs/content/design/ofs.md | 4 +
.../content/design/ozone-enhancement-proposals.md | 2 +-
hadoop-hdds/docs/content/design/recon2.md | 2 +-
hadoop-hdds/docs/content/design/scmha.md | 4 +-
hadoop-hdds/docs/content/design/trash.md | 7 +-
hadoop-hdds/docs/content/interface/CSI.md | 7 -
hadoop-hdds/docs/content/interface/OzoneFS.md | 3 +
hadoop-hdds/docs/content/interface/S3.md | 8 +-
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md | 1 -
hadoop-hdds/docs/content/security/SecuringTDE.md | 4 +-
.../docs/content/security/SecuringTDE.zh.md | 52 ++
hadoop-hdds/docs/content/security/SecurityAcls.md | 8 +-
.../docs/content/security/SecurityAcls.zh.md | 66 +++
hadoop-hdds/docs/content/start/OnPrem.md | 2 +-
hadoop-hdds/docs/content/start/OnPrem.zh.md | 2 +-
.../docs/content/start/StartFromDockerHub.md | 6 +-
.../hadoop/hdds/conf/DatanodeRatisGrpcConfig.java | 7 +-
.../hdds/conf/DatanodeRatisServerConfig.java | 32 +-
.../x509/certificate/authority/BaseApprover.java | 9 +
.../certificate/authority/DefaultApprover.java | 11 +
.../certificate/authority/DefaultCAServer.java | 42 +-
.../certificate/client/DNCertificateClient.java | 5 +-
.../certificates/utils/SelfSignedCertificate.java | 105 +++-
.../hadoop/hdds/server/http/HttpServer2.java | 21 +
.../apache/hadoop/hdds/utils/HddsServerUtil.java | 16 +-
.../hadoop/hdds/utils/db/RDBStoreIterator.java | 40 +-
.../x509/certificates/TestRootCertificate.java | 42 +-
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 10 +-
.../hadoop/hdds/utils/db/TestRDBTableStore.java | 61 ++
hadoop-hdds/hadoop-dependency-server/pom.xml | 4 +
.../src/main/{proto => resources}/proto.lock | 0
.../interface-client/src/main/proto/hdds.proto | 2 +
.../src/main/{proto => resources}/proto.lock | 0
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 6 +
.../src/main/{proto => resources}/proto.lock | 0
hadoop-hdds/pom.xml | 10 +-
.../hdds/scm/container/ContainerStateManager.java | 8 +-
.../hdds/scm/container/SCMContainerManager.java | 7 +-
.../apache/hadoop/hdds/scm/node/DatanodeInfo.java | 22 +
.../apache/hadoop/hdds/scm/node/NodeManager.java | 2 +
.../hadoop/hdds/scm/node/SCMNodeManager.java | 21 +
.../hadoop/hdds/scm/pipeline/PipelineManager.java | 2 +
.../hdds/scm/pipeline/PipelineManagerV2Impl.java | 10 +
.../hdds/scm/pipeline/PipelineStateManager.java | 5 +
.../scm/pipeline/PipelineStateManagerV2Impl.java | 5 +
.../hdds/scm/pipeline/SCMPipelineManager.java | 30 +-
.../hadoop/hdds/scm/pipeline/StateManager.java | 1 +
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 2 +
.../hdds/scm/server/StorageContainerManager.java | 13 +-
.../main/resources/webapps/scm/scm-overview.html | 11 +
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 46 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 66 +++
.../hadoop/hdds/scm/container/MockNodeManager.java | 11 +
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 51 +-
.../hdds/scm/pipeline/MockPipelineManager.java | 5 +
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 87 +++
...TestSCMStoreImplWithOldPipelineIDKeyFormat.java | 180 ++++++
.../testutils/ReplicationNodeManagerMock.java | 5 +
.../org/apache/hadoop/ozone/client/BucketArgs.java | 33 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 23 +-
.../client/io/BlockOutputStreamEntryPool.java | 6 +-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 11 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 47 ++
.../org/apache/hadoop/ozone/audit/OMAction.java | 5 -
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 7 +
.../hadoop/ozone/om/exceptions/OMException.java | 6 +-
.../ozone/om/helpers/BucketEncryptionKeyInfo.java | 4 +
.../hadoop/ozone/om/helpers/OmBucketInfo.java | 142 ++++-
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 18 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 37 +-
.../hadoop/ozone/om/helpers/OmKeyLocationInfo.java | 18 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 107 ++--
.../hadoop/ozone/om/helpers/OzoneFSUtils.java | 30 +
.../hadoop/ozone/om/helpers/RepeatedOmKeyInfo.java | 9 +-
.../ozone/security/OzoneTokenIdentifier.java | 56 +-
.../org/apache/hadoop/ozone/util/ExitManager.java} | 24 +-
.../java/org/apache/hadoop/ozone/TestOmUtils.java | 49 ++
.../hadoop/ozone/om/helpers/TestOmBucketInfo.java | 22 +-
.../om/helpers/TestOmKeyLocationInfoGroup.java | 59 ++
.../hadoop/ozone/om/helpers/TestOzoneFsUtils.java} | 29 +-
.../csi/src/main/{proto => resources}/proto.lock | 0
hadoop-ozone/dev-support/checks/acceptance.sh | 2 +
.../dev-support/checks/{acceptance.sh => bats.sh} | 29 +-
.../checks/{acceptance.sh => kubernetes.sh} | 4 +-
.../dist/dev-support/bin/dist-layout-stitching | 2 +
hadoop-ozone/dist/pom.xml | 9 +-
.../main/compose/{ozonesecure => ozone-mr}/test.sh | 60 +-
.../dist/src/main/compose/ozone-topology/test.sh | 4 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 20 +-
.../dist/src/main/compose/ozonesecure/test.sh | 17 +-
hadoop-ozone/dist/src/main/compose/test-all.sh | 25 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 61 +-
.../flekszible.yaml => compose/upgrade/.env} | 7 +-
.../dist/src/main/compose/upgrade/README.md | 29 +
.../src/main/compose/upgrade/docker-compose.yaml | 127 ++++
.../dist/src/main/compose/upgrade/docker-config | 33 ++
hadoop-ozone/dist/src/main/compose/upgrade/test.sh | 70 +++
.../src/main/compose/upgrade/versions/README.md | 18 +-
.../upgrade/versions/ozone-0.5.0.sh} | 4 +-
.../upgrade/versions/ozone-0.6.0.sh} | 4 +-
.../dist/src/main/k8s/.gitignore | 18 +-
.../src/main/k8s/definitions/ozone/config.yaml | 1 +
.../k8s/definitions/ozone/definitions/onenode.yaml | 2 +-
.../{pv-test => test-webserver}/flekszible.yaml | 0
.../webserver-deployment.yaml | 0
.../webserver-service.yaml | 0
.../webserver-volume.yaml | 0
.../examples/getting-started/config-configmap.yaml | 1 +
.../getting-started/datanode-statefulset.yaml | 10 -
.../getting-started/test.sh} | 34 +-
.../k8s/examples/minikube/config-configmap.yaml | 1 +
.../minikube/test.sh} | 34 +-
.../src/main/k8s/examples/ozone-dev/Flekszible | 4 +-
.../k8s/examples/ozone-dev/config-configmap.yaml | 1 +
.../examples/ozone-dev/datanode-statefulset.yaml | 10 -
.../ozone-dev/test.sh} | 34 +-
.../dist/src/main/k8s/examples/ozone/Flekszible | 2 +-
.../main/k8s/examples/ozone/config-configmap.yaml | 1 +
.../ozone/test.sh} | 34 +-
.../test.sh => k8s/examples/test-all.sh} | 47 +-
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh | 137 +++++
.../dist/src/main/smoketest/basic/links.robot | 152 +++++
.../dist/src/main/smoketest/commonlib.robot | 34 +-
.../smoketest/{commonlib.robot => lib/os.robot} | 36 +-
.../src/main/smoketest/lib/os_tests.robot} | 36 +-
.../dist/src/main/smoketest/ozone-lib/shell.robot | 53 ++
.../src/main/smoketest/ozone-lib/shell_tests.robot | 58 ++
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 2 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 16 +-
.../dist/src/main/smoketest/recon/recon-api.robot | 16 +-
hadoop-ozone/dist/src/main/smoketest/robot.robot | 81 +++
.../src/main/smoketest/s3/MultipartUpload.robot | 7 +-
.../dist/src/main/smoketest/s3/bucketdelete.robot | 12 +-
.../dist/src/main/smoketest/s3/buckethead.robot | 5 +-
.../dist/src/main/smoketest/s3/commonawslib.robot | 35 +-
.../bucket-encryption.robot} | 34 +-
.../main/smoketest/security/ozone-secure-s3.robot | 15 +
.../dist/src/main/smoketest/spnego/web.robot | 14 +
hadoop-ozone/dist/src/shell/hdds/hadoop-daemons.sh | 2 +-
.../dist/src/shell/hdds/hadoop-functions.sh | 6 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 5 -
.../upgrade/0.6.0.sh} | 20 +-
.../upgrade/0.6.0/01-migrate-scm-db.sh} | 21 +-
.../dist/src/test/shell/compose_testlib.bats | 36 +-
hadoop-ozone/dist/src/test/shell/gc_opts.bats | 6 +-
.../dist/src/test/shell/k8s_testlib.bats | 58 +-
.../dist/src/test/shell/test1/test.sh | 20 +-
.../dist/src/test/shell/test2/test.sh | 18 +-
.../dist/src/test/shell/test3/subtest1/test.sh | 18 +-
.../dist/src/test/shell/test4/test.sh | 18 +-
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 231 ++++++++
.../hadoop/fs/ozone/TestOzoneFileInterfaces.java | 11 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 158 ++++-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 43 +-
.../hadoop/fs/ozone/contract/OzoneContract.java | 18 +-
.../ozone/contract/rooted/RootedOzoneContract.java | 18 +-
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 2 +-
.../hadoop/ozone/TestStorageContainerManager.java | 33 +-
.../ozone/client/rpc/Test2WayCommitInRatis.java | 16 +-
.../rpc/TestBlockOutputStreamWithFailures.java | 28 +-
...estBlockOutputStreamWithFailuresFlushDelay.java | 28 +-
.../hadoop/ozone/client/rpc/TestCommitWatcher.java | 41 +-
.../rpc/TestContainerReplicationEndToEnd.java | 19 +-
.../TestContainerStateMachineFailureOnRead.java | 27 +-
.../rpc/TestContainerStateMachineFailures.java | 27 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 28 +-
.../client/rpc/TestFailureHandlingByClient.java | 28 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 28 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 28 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 124 +++-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 3 -
.../client/rpc/TestValidateBCSIDOnRestart.java | 28 +-
.../ozone/client/rpc/TestWatchForCommit.java | 28 +-
.../hadoop/ozone/freon/TestDataValidate.java | 18 +-
.../ozone/freon/TestFreonWithDatanodeRestart.java | 18 +-
.../ozone/freon/TestFreonWithPipelineDestroy.java | 17 +-
.../hadoop/ozone/freon/TestRandomKeyGenerator.java | 18 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 14 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 217 +++++--
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 110 ++--
.../ozone/om/TestOzoneManagerListVolumes.java | 18 +-
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 26 +-
.../snapshot/TestOzoneManagerSnapshotProvider.java | 26 +-
.../ozone/recon/TestReconWithOzoneManager.java | 14 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 2 +
.../src/test/resources/log4j.properties | 3 -
.../src/main/proto/OmClientProtocol.proto | 6 +
.../src/main/{proto => resources}/proto.lock | 0
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 112 ++--
.../org/apache/hadoop/ozone/om/KeyManager.java | 6 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 271 +++++----
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 34 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 652 +++++++++++++++------
.../org/apache/hadoop/ozone/om/ResolvedBucket.java | 111 ++++
.../hadoop/ozone/om/codec/OMDBDefinition.java | 19 +-
.../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 13 +-
.../ozone/om/codec/RepeatedOmKeyInfoCodec.java | 13 +-
.../ozone/om/codec/TokenIdentifierCodec.java | 16 +-
.../hadoop/ozone/om/ratis/OMTransactionInfo.java | 21 +-
.../ozone/om/ratis/OzoneManagerStateMachine.java | 6 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 37 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 76 +++
.../om/request/bucket/OMBucketCreateRequest.java | 14 +
.../om/request/file/OMDirectoryCreateRequest.java | 4 +
.../ozone/om/request/file/OMFileCreateRequest.java | 10 +-
.../ozone/om/request/file/OMFileRequest.java | 2 +-
.../om/request/key/OMAllocateBlockRequest.java | 15 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 31 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 80 ++-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 19 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 26 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 51 +-
.../ozone/om/request/key/OMKeysDeleteRequest.java | 48 +-
.../om/request/key/OMTrashRecoverRequest.java | 7 +
.../S3InitiateMultipartUploadRequest.java | 60 +-
.../multipart/S3MultipartUploadAbortRequest.java | 30 +-
.../S3MultipartUploadCommitPartRequest.java | 51 +-
.../S3MultipartUploadCompleteRequest.java | 76 ++-
.../ozone/om/response/key/OMKeyDeleteResponse.java | 2 +-
.../S3MultipartUploadCommitPartResponse.java | 3 +-
.../om/snapshot/OzoneManagerSnapshotProvider.java | 10 +-
.../protocolPB/OzoneManagerRequestHandler.java | 2 +-
.../main/resources/webapps/ozoneManager/main.html | 2 +
.../webapps/ozoneManager/om-overview.html | 22 +-
.../resources/webapps/ozoneManager/ozoneManager.js | 7 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 43 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 2 +-
.../hadoop/ozone/om/codec/TestOmKeyInfoCodec.java | 116 ++++
.../ozone/om/codec/TestRepeatedOmKeyInfoCodec.java | 121 ++++
...tOzoneManagerDoubleBufferWithDummyResponse.java | 2 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 2 +-
.../ozone/om/request/TestNormalizePaths.java | 109 ++++
.../request/file/TestOMDirectoryCreateRequest.java | 4 +
.../om/request/key/TestOMKeyCreateRequest.java | 182 +++++-
.../ozone/om/request/key/TestOMKeyRequest.java | 14 +
.../TestS3InitiateMultipartUploadRequest.java | 59 +-
.../s3/multipart/TestS3MultipartRequest.java | 15 +-
.../TestS3MultipartUploadCommitPartRequest.java | 2 +-
.../ozone/security/TestOzoneTokenIdentifier.java | 19 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 45 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 29 +
hadoop-ozone/ozonefs-hadoop2/pom.xml | 2 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 2 +-
hadoop-ozone/pom.xml | 16 +-
.../hadoop/ozone/recon/ReconControllerModule.java | 10 +-
.../hadoop/ozone/recon/api/NodeEndpoint.java | 2 +
.../ozone/recon/api/types/DatanodeMetadata.java | 28 +
.../DatanodeDetailsCodec.java} | 29 +-
.../ReconNodeDBKeyCodec.java} | 26 +-
.../hadoop/ozone/recon/codec}/package-info.java | 9 +-
.../ozone/recon/scm/ReconContainerManager.java | 61 +-
.../recon/scm/ReconContainerReportHandler.java | 2 +-
.../ReconIncrementalContainerReportHandler.java | 2 +-
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 55 +-
...DBDefinition.java => ReconSCMDBDefinition.java} | 29 +-
.../scm/ReconStorageContainerManagerFacade.java | 9 +-
.../spi/impl/ContainerDBServiceProviderImpl.java | 29 +-
.../recon/spi/impl/ReconContainerDBProvider.java | 28 +-
.../ozone/recon/spi/impl/ReconDBDefinition.java | 71 +++
.../webapps/recon/ozone-recon-web/api/db.json | 48 +-
.../components/autoReloadPanel/autoReloadPanel.tsx | 4 +-
.../ozone-recon-web/src/types/datanode.types.tsx | 4 +-
.../datanode.types.tsx => utils/columnSearch.less} | 21 +-
.../ozone-recon-web/src/utils/columnSearch.tsx | 94 +++
.../src/views/datanodes/datanodes.less | 1 +
.../src/views/datanodes/datanodes.tsx | 101 +++-
.../src/views/overview/overview.tsx | 2 +-
.../src/views/pipelines/pipelines.tsx | 32 +-
.../scm/AbstractReconContainerManagerTest.java | 51 +-
.../ozone/recon/scm/TestReconContainerManager.java | 95 ++-
...TestReconIncrementalContainerReportHandler.java | 63 ++
.../ozone/recon/scm/TestReconNodeManager.java | 20 +-
.../ozone/recon/scm/TestReconPipelineManager.java | 8 +-
...reProcessor.java => AWSSignatureProcessor.java} | 43 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 8 +-
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 8 +-
.../ozone/s3/header/AuthorizationHeaderV2.java | 2 +-
.../ozone/s3/header/AuthorizationHeaderV4.java | 4 +-
.../apache/hadoop/ozone/s3/util/RangeHeader.java | 6 +-
...ocessor.java => TestAWSSignatureProcessor.java} | 44 +-
.../hadoop/ozone/s3/endpoint/TestBucketHead.java | 11 +-
.../hadoop/ozone/s3/endpoint/TestBucketPut.java | 2 +-
.../ozone/s3/header/TestAuthorizationHeaderV4.java | 2 +-
hadoop-ozone/tools/pom.xml | 4 +
.../hadoop/ozone/debug/DBDefinitionFactory.java | 22 +-
.../org/apache/hadoop/ozone/debug/DBScanner.java | 25 +-
.../apache/hadoop/ozone/fsck/ContainerMapper.java | 2 +-
.../hadoop/ozone/shell/bucket/BucketCommands.java | 1 +
.../ozone/shell/bucket/LinkBucketHandler.java | 79 +++
.../ozone/shell/volume/CreateVolumeHandler.java | 2 +-
.../ozone/debug/TestDBDefinitionFactory.java | 59 ++
hadoop-ozone/upgrade/pom.xml | 57 --
.../org/apache/hadoop/ozone/upgrade/Balance.java | 38 --
.../hadoop/ozone/upgrade/InPlaceUpgrade.java | 45 --
pom.xml | 61 +-
330 files changed, 8454 insertions(+), 2516 deletions(-)
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/retrypolicy/RequestTypeDependentRetryPolicyCreator.java
create mode 100644
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/retrypolicy/RetryLimitedPolicyCreator.java
copy
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/DBDefinitionFactory.java
=>
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/retrypolicy/RetryPolicyCreator.java
(52%)
copy {hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade =>
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/retrypolicy}/package-info.java
(81%)
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/conf/TestRaftClientConfig.java
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/ratis/conf/TestRatisClientConfig.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/states/datanode/TestRunningDatanodeState.java
create mode 100644 hadoop-hdds/docs/content/security/SecuringTDE.zh.md
create mode 100644 hadoop-hdds/docs/content/security/SecurityAcls.zh.md
rename hadoop-hdds/interface-admin/src/main/{proto => resources}/proto.lock
(100%)
rename hadoop-hdds/interface-client/src/main/{proto => resources}/proto.lock
(100%)
rename hadoop-hdds/interface-server/src/main/{proto => resources}/proto.lock
(100%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestSCMStoreImplWithOldPipelineIDKeyFormat.java
rename
hadoop-ozone/{upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Execute.java
=> common/src/main/java/org/apache/hadoop/ozone/util/ExitManager.java} (64%)
create mode 100644
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmKeyLocationInfoGroup.java
rename
hadoop-ozone/{upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Plan.java
=>
common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOzoneFsUtils.java}
(57%)
rename hadoop-ozone/csi/src/main/{proto => resources}/proto.lock (100%)
copy hadoop-ozone/dev-support/checks/{acceptance.sh => bats.sh} (62%)
copy hadoop-ozone/dev-support/checks/{acceptance.sh => kubernetes.sh} (92%)
copy hadoop-ozone/dist/src/main/compose/{ozonesecure => ozone-mr}/test.sh (50%)
mode change 100755 => 100644
copy hadoop-ozone/dist/src/main/{k8s/definitions/pv-test/flekszible.yaml =>
compose/upgrade/.env} (81%)
create mode 100644 hadoop-ozone/dist/src/main/compose/upgrade/README.md
create mode 100644
hadoop-ozone/dist/src/main/compose/upgrade/docker-compose.yaml
create mode 100644 hadoop-ozone/dist/src/main/compose/upgrade/docker-config
create mode 100644 hadoop-ozone/dist/src/main/compose/upgrade/test.sh
copy hadoop-hdds/docs/content/design/scmha.md =>
hadoop-ozone/dist/src/main/compose/upgrade/versions/README.md (60%)
copy hadoop-ozone/dist/src/main/{k8s/definitions/pv-test/flekszible.yaml =>
compose/upgrade/versions/ozone-0.5.0.sh} (91%)
copy hadoop-ozone/dist/src/main/{k8s/definitions/pv-test/flekszible.yaml =>
compose/upgrade/versions/ozone-0.6.0.sh} (91%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/main/k8s/.gitignore (70%)
rename hadoop-ozone/dist/src/main/k8s/definitions/{pv-test =>
test-webserver}/flekszible.yaml (100%)
rename hadoop-ozone/dist/src/main/k8s/definitions/{pv-test =>
test-webserver}/webserver-deployment.yaml (100%)
copy hadoop-ozone/dist/src/main/k8s/definitions/{pv-test =>
test-webserver}/webserver-service.yaml (100%)
rename hadoop-ozone/dist/src/main/k8s/definitions/{pv-test =>
test-webserver}/webserver-volume.yaml (100%)
copy
hadoop-ozone/dist/src/main/k8s/{definitions/pv-test/webserver-service.yaml =>
examples/getting-started/test.sh} (70%)
mode change 100644 => 100755
copy
hadoop-ozone/dist/src/main/k8s/{definitions/pv-test/webserver-service.yaml =>
examples/minikube/test.sh} (70%)
mode change 100644 => 100755
copy
hadoop-ozone/dist/src/main/k8s/{definitions/pv-test/webserver-service.yaml =>
examples/ozone-dev/test.sh} (70%)
mode change 100644 => 100755
copy
hadoop-ozone/dist/src/main/k8s/{definitions/pv-test/webserver-service.yaml =>
examples/ozone/test.sh} (70%)
mode change 100644 => 100755
copy hadoop-ozone/dist/src/main/{compose/ozonesecure/test.sh =>
k8s/examples/test-all.sh} (54%)
create mode 100644 hadoop-ozone/dist/src/main/k8s/examples/testlib.sh
create mode 100644 hadoop-ozone/dist/src/main/smoketest/basic/links.robot
copy hadoop-ozone/dist/src/main/smoketest/{commonlib.robot => lib/os.robot}
(57%)
copy hadoop-ozone/{dev-support/checks/acceptance.sh =>
dist/src/main/smoketest/lib/os_tests.robot} (55%)
mode change 100755 => 100644
create mode 100644 hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell.robot
create mode 100644
hadoop-ozone/dist/src/main/smoketest/ozone-lib/shell_tests.robot
create mode 100644 hadoop-ozone/dist/src/main/smoketest/robot.robot
copy hadoop-ozone/dist/src/main/smoketest/{s3/bucketdelete.robot =>
security/bucket-encryption.robot} (51%)
copy
hadoop-ozone/dist/src/{main/k8s/definitions/pv-test/webserver-service.yaml =>
shell/upgrade/0.6.0.sh} (69%)
mode change 100644 => 100755
rename
hadoop-ozone/dist/src/{main/k8s/definitions/pv-test/webserver-service.yaml =>
shell/upgrade/0.6.0/01-migrate-scm-db.sh} (65%)
mode change 100644 => 100755
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/compose_testlib.bats (60%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/k8s_testlib.bats (58%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/test1/test.sh (67%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/test2/test.sh (71%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/test3/subtest1/test.sh (71%)
copy .github/workflows/comments.yaml =>
hadoop-ozone/dist/src/test/shell/test4/test.sh (71%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFSWithObjectStoreCreate.java
rename hadoop-ozone/interface-client/src/main/{proto => resources}/proto.lock
(100%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java
copy hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html
=>
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/om-overview.html
(60%)
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/codec/TestOmKeyInfoCodec.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/codec/TestRepeatedOmKeyInfoCodec.java
create mode 100644
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/TestNormalizePaths.java
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/{scm/ReconDBDefinition.java
=> codec/DatanodeDetailsCodec.java} (52%)
copy
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/{scm/ReconDBDefinition.java
=> codec/ReconNodeDBKeyCodec.java} (59%)
rename hadoop-ozone/{upgrade/src/main/java/org/apache/hadoop/ozone/upgrade =>
recon/src/main/java/org/apache/hadoop/ozone/recon/codec}/package-info.java (87%)
rename
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/{ReconDBDefinition.java
=> ReconSCMDBDefinition.java} (55%)
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconDBDefinition.java
copy
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/{types/datanode.types.tsx
=> utils/columnSearch.less} (77%)
create mode 100644
hadoop-ozone/recon/src/main/resources/webapps/recon/ozone-recon-web/src/utils/columnSearch.tsx
rename
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/{AWSV4SignatureProcessor.java
=> AWSSignatureProcessor.java} (91%)
rename
hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/{TestAWSV4SignatureProcessor.java
=> TestAWSSignatureProcessor.java} (69%)
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/bucket/LinkBucketHandler.java
create mode 100644
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/debug/TestDBDefinitionFactory.java
delete mode 100644 hadoop-ozone/upgrade/pom.xml
delete mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/Balance.java
delete mode 100644
hadoop-ozone/upgrade/src/main/java/org/apache/hadoop/ozone/upgrade/InPlaceUpgrade.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
