This is an automated email from the ASF dual-hosted git repository.
rakeshr pushed a change to branch HDDS-2939
in repository https://gitbox.apache.org/repos/asf/ozone.git.
from 723cdaa Merge remote-tracking branch 'origin/master' into HDDS-2939
add e50fe52 HDDS-5041. Use getShortUserName in getTrashRoot(s) (#2244)
add fc7b0bd HDDS-5260. Bump node to v16.2.0 for Recon (#2276)
add 9080fc3 HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen
in SCM log. (#2267)
add e00fa80 HDDS-5206. Addendum: Support revoking S3 secret (#2270)
add 8d29be1 HDDS-5142. Make generic streaming client/service for
container re-replication, data read, scm/om snapshot download (#2256)
add ab8f07d HDDS-5249. Race Condition between Full and Incremental
Container Reports (#2268)
add 931a041 HDDS-5266 Misspelt words in
S3MultipartUploadCommitPartRequest.java line 202 (#2279)
add 355096b HDDS-5236. Require block token for more operations (#2254)
add 19f5bb2 HDDS-5250. Build integration tests with Maven cache (#2269)
add 0b4779c HDDS-5073. Use ReplicationConfig on client side (#2136)
add 2dbf04e HDDS-5191. Increase default pvc storage size (#2219)
add 000e7c5 HDDS-5252. PipelinePlacementPolicy filter out datanodes with
not enough space. (#2271)
add 2671b48 HDDS-5219. Limit number of bad volumes by
dfs.datanode.failed.volumes.tolerated. (#2243)
add d9ae1b3 HDDS-5241. SCM UI should have leader/follower and Primordial
SCM information (#2260)
add 53bbabf HDDS-5275. Datanode Report Publisher publishes one extra
report after DN shutdown (#2283)
add 4301315 HDDS-5158. Add documentation for SCM HA Security. (#2205)
add d567360 HDDS-5273. Handle unsecure cluster convert to secure cluster
for SCM. (#2281)
add 00b675d HDDS-4927. Determine over and under utilized datanodes in
Container Balancer. (#2230)
add 8586815 HDDS-4043. allow deletion from Trash directory without
-skipTrash option (#2110)
add fa4ddbc HDDS-4936. Change ozone groupId from org.apache.hadoop to
org.apache.ozone (#2018)
add 3ffbdd0 HDDS-4993. Add guardrail for reserved buffer size when DN
reads a chunk (#2058)
add e5a774f HDDS-5123. Use the pre-created apache/ozone-testkrb5 image
during secure acceptance tests (#2165)
add 5fd9780 HDDS-5272. Make ozonefs.robot execution repeatable (#2280)
add 12e2918 HDDS-5280. Make XceiverClientManager creation when necessary
in ContainerOperationClient (#2289)
add 6b09219 HDDS-5264. SCM should send token for CloseContainer command
(#2277)
add e35dde9 HDDS-5151. Support ByteBuffer read in OzoneInputStream (#2203)
add 96d9ed7 HDDS-5240. Fix out of bound exception when loading
auditparser (#2258)
add 69b92a0 HDDS-5281. Add reinitialize() for SequenceIdGenerator. (#2292)
add 183f4b4 HDDS-5242. Skip `failing` acceptance suite by default (#2263)
add 18f0fe1 HDDS-4733. Remove leveldb from codebase (#2274)
add 656339e HDDS-5204. Missing type-level Javadoc comments (#2262)
add 90a8d7c HDDS-5276. Use built-in cancel support for duplicates (#2284)
add f3f258a HDDS-5277. Recon shows operational status as
"DECOMMISSIONING" for "DECOMMISSIONED" DNs (#2286)
add b6a035a HDDS-5288. Recon UI cannot sort DN by operational state
(#2298)
add 3030c9f HDDS-5282. Relocate classes copied from Hadoop (#2293)
add 095d09b HDDS-4483. Datanodes should send last processed CRL sequence
ID in heartbeats (#2285)
add 25d66d7 HDDS-5154. Add SCM GRPC server to publish CRL update. (#2216)
add c2ae90b HDDS-5295. testCRLStatusReportPublisher fails to create
CRLInfo (#2302)
add 8c1de61 HDDS-5274. Revert "HDDS-5153. Decommissioning a dead node
should complete immediately (#2190)" (#2282)
add bb8eb39 HDDS-5286. Document SCM HA classes (#2297)
add 405925f HDDS-5168. Use ReplicationConfig in OmKeyArgs (#2287).
Contributed by elek.
add 7bc1a25 HDDS-5292. Introduce the WritableContainerInterface to SCM
(#2300)
add 9ad2e98 HDDS-5298. Use netty-bom to ensure consistent Netty version
(#2304)
new 86bcebb Merge remote-tracking branch 'origin/master' into HDDS-2939
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/buildenv/Dockerfile | 20 -
.github/workflows/cancel-ci.yaml | 8 -
.github/workflows/post-commit.yml | 108 ++--
hadoop-hdds/client/pom.xml | 30 +-
.../hadoop/hdds/scm/storage/BlockInputStream.java | 29 +-
.../hadoop/hdds/scm/storage/ByteArrayReader.java | 67 +++
.../hadoop/hdds/scm/storage/ByteBufferReader.java | 71 +++
.../hdds/scm/storage/ByteReaderStrategy.java} | 27 +-
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 38 +-
.../hdds/scm/storage/TestBlockInputStream.java | 37 +-
.../hdds/scm/storage/TestChunkInputStream.java | 2 +-
hadoop-hdds/common/pom.xml | 31 +-
.../apache/hadoop/hdds/DFSConfigKeysLegacy.java | 5 -
.../org/apache/hadoop/hdds/HddsConfigKeys.java | 8 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 99 +++-
.../hadoop/hdds/client/RatisReplicationConfig.java | 15 +
.../hadoop/hdds/client/ReplicationConfig.java | 55 ++
.../hdds/client/StandaloneReplicationConfig.java | 9 +
.../hdds/scm/container/ContainerException.java | 14 +-
.../scm/container/ContainerNotFoundException.java | 4 +-
.../ContainerReplicaNotFoundException.java | 4 +-
.../hadoop/hdds/scm/exceptions/SCMException.java | 6 +
.../scm/ha/RetriableWithNoFailoverException.java | 4 +
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 26 +-
.../scm/pipeline/PipelineNotFoundException.java | 8 +-
.../pipeline/UnknownPipelineStateException.java | 8 +-
.../hadoop/hdds/scm/utils/ClientCommandsUtils.java | 3 +
.../hadoop/hdds/security/x509/crl/CRLInfo.java | 39 ++
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 4 +-
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 2 +-
.../ozone/common/ChecksumByteBufferFactory.java | 3 +
.../ozone/common/ChecksumByteBufferImpl.java | 5 +-
.../hadoop/ozone/common/utils/BufferUtils.java | 3 +
.../java/org/apache/hadoop/ozone/ha/ConfUtils.java | 3 +
.../common/src/main/resources/ozone-default.xml | 8 +
.../java/org/apache/hadoop/hdds/TestHddsUtils.java | 2 +-
.../hadoop/hdds/client/TestReplicationConfig.java | 147 +++++
.../hdds/fs/TestCachingSpaceUsageSource.java | 2 +-
.../java/org/apache/hadoop/hdds/fs/TestDU.java | 2 +-
.../org/apache/hadoop/hdds/fs/TestDUFactory.java | 2 +-
.../hdds/fs/TestDedicatedDiskSpaceUsage.java | 2 +-
.../fs/TestDedicatedDiskSpaceUsageFactory.java | 2 +-
.../hadoop/hdds/fs/TestSaveSpaceUsageToFile.java | 4 +-
.../hadoop/hdds/fs/TestSpaceUsageFactory.java | 2 +-
.../apache/hadoop/hdds/scm/ha/TestSCMNodeInfo.java | 3 +
.../hadoop/hdds/tracing/TestStringCodec.java | 2 +-
.../hadoop/hdds/utils/TestResourceLimitCache.java | 2 +-
.../common/TestChecksumImplsComputeSameValues.java | 4 +
.../ozone/container/ContainerTestHelper.java | 101 +++-
.../apache/hadoop/ozone/lock/TestLockManager.java | 2 +-
hadoop-hdds/config/pom.xml | 10 +-
hadoop-hdds/container-service/pom.xml | 54 +-
.../apache/hadoop/ozone/HddsDatanodeService.java | 2 +-
.../common/report/CRLStatusReportPublisher.java | 82 +++
.../container/common/report/ReportPublisher.java | 4 +-
.../common/report/ReportPublisherFactory.java | 2 +
.../common/statemachine/DatanodeConfiguration.java | 30 ++
.../common/statemachine/DatanodeStateMachine.java | 27 +-
.../common/statemachine/StateContext.java | 21 +-
.../CloseContainerCommandHandler.java | 9 +-
.../states/endpoint/HeartbeatEndpointTask.java | 4 +
.../container/common/volume/HddsVolumeChecker.java | 11 +-
.../container/common/volume/MutableVolumeSet.java | 42 +-
.../container/keyvalue/KeyValueContainerCheck.java | 4 +-
.../container/keyvalue/helpers/BlockUtils.java | 2 +-
.../container/keyvalue/helpers/ChunkUtils.java | 12 +
.../keyvalue/impl/ChunkManagerDummyImpl.java | 6 +-
.../keyvalue/impl/FilePerBlockStrategy.java | 5 +-
.../keyvalue/impl/FilePerChunkStrategy.java | 2 +
.../background/BlockDeletingService.java | 3 +
.../ozone/container/ozoneimpl/OzoneContainer.java | 5 +-
.../container/replication/ReplicationServer.java | 3 +
.../replication/ReplicationSupervisor.java | 3 +
.../stream/DirectoryServerDestination.java} | 28 +-
.../container/stream/DirectoryServerSource.java | 57 ++
.../container/stream/DirstreamClientHandler.java | 143 +++++
.../container/stream/DirstreamServerHandler.java | 135 +++++
.../ozone/container/stream/StreamingClient.java | 101 ++++
.../container/stream/StreamingDestination.java} | 24 +-
.../ozone/container/stream/StreamingServer.java | 94 ++++
.../ozone/container/stream/StreamingSource.java} | 28 +-
.../ozone/container/stream/package-info.java} | 14 +-
.../hadoop/ozone/protocol/commands/SCMCommand.java | 10 +
.../metadata/TestDatanodeCRLStoreImpl.java | 2 +-
.../hadoop/ozone/TestHddsDatanodeService.java | 2 +-
.../hadoop/ozone/TestHddsSecureDatanodeInit.java | 4 +-
.../ozone/container/common/SCMTestUtils.java | 2 +-
.../container/common/TestBlockDeletingService.java | 9 +-
.../container/common/TestDatanodeStateMachine.java | 15 +-
.../TestSchemaOneBackwardsCompatibility.java | 6 +-
.../common/helpers/TestDatanodeVersionFile.java | 2 +-
.../common/impl/TestContainerDataYaml.java | 2 +-
.../impl/TestContainerDeletionChoosingPolicy.java | 2 +-
.../common/impl/TestContainerPersistence.java | 4 +-
.../container/common/impl/TestContainerSet.java | 2 +-
.../container/common/impl/TestHddsDispatcher.java | 7 +-
.../common/report/TestReportPublisher.java | 52 +-
.../common/report/TestReportPublisherFactory.java | 13 +-
.../statemachine/TestDatanodeConfiguration.java | 12 +
.../common/statemachine/TestStateContext.java | 74 +--
.../common/volume/TestHddsVolumeChecker.java | 2 +-
.../volume/TestRoundRobinVolumeChoosingPolicy.java | 2 +-
.../container/common/volume/TestVolumeSet.java | 9 +-
.../common/volume/TestVolumeSetDiskChecks.java | 13 +-
.../keyvalue/TestKeyValueBlockIterator.java | 4 +-
.../container/keyvalue/TestKeyValueContainer.java | 2 +-
.../keyvalue/TestKeyValueContainerCheck.java | 4 +-
.../TestKeyValueContainerMarkUnhealthy.java | 2 +-
.../container/keyvalue/TestKeyValueHandler.java | 5 +-
.../container/keyvalue/TestTarContainerPacker.java | 2 +-
.../container/keyvalue/helpers/TestChunkUtils.java | 4 +-
.../keyvalue/impl/CommonChunkManagerTestCases.java | 35 +-
.../keyvalue/impl/TestBlockManagerImpl.java | 2 +-
.../container/ozoneimpl/TestContainerReader.java | 2 +-
.../container/ozoneimpl/TestOzoneContainer.java | 5 +-
.../replication/TestReplicationSupervisor.java | 2 +-
.../replication/TestSimpleContainerDownloader.java | 4 +-
.../container/stream/TestStreamingServer.java | 127 +++++
hadoop-hdds/dev-support/checkstyle/checkstyle.xml | 3 +
hadoop-hdds/docs/content/feature/SCM-HA.md | 68 ++-
hadoop-hdds/docs/content/feature/scm-secure-ha.png | Bin 0 -> 46757 bytes
hadoop-hdds/docs/content/interface/O3fs.md | 4 +-
hadoop-hdds/docs/content/interface/O3fs.zh.md | 8 +-
hadoop-hdds/docs/content/interface/Ofs.md | 4 +-
hadoop-hdds/docs/content/interface/S3.md | 21 +
hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md | 6 +-
.../docs/content/recipe/SparkOzoneFSK8S.zh.md | 6 +-
hadoop-hdds/docs/pom.xml | 6 +-
hadoop-hdds/framework/pom.xml | 34 +-
.../java/org/apache/hadoop/hdds/NodeDetails.java | 5 +-
.../scm/update/client/CRLClientUpdateHandler.java | 201 +++++++
.../hadoop/hdds/scm/update/client/CRLStore.java} | 24 +-
.../hdds/scm/update/client/ClientCRLStore.java | 98 ++++
.../scm/update/client/ClientUpdateHandler.java} | 23 +-
.../client/SCMUpdateClientConfiguration.java | 53 ++
.../update/client/SCMUpdateServiceGrpcClient.java | 218 ++++++++
.../scm/update/client/UpdateServiceConfig.java} | 42 +-
.../hdds/scm/update/client/package-info.java} | 24 +-
.../hdds/scm/update/server/CRLClientInfo.java | 56 ++
.../scm/update/server/SCMUpdateClientInfo.java | 67 +++
.../hdds/scm/update/server/package-info.java} | 24 +-
.../hdds/security/token/BlockTokenVerifier.java | 39 +-
.../security/token/CompositeTokenVerifier.java | 4 +-
...nVerifier.java => ContainerTokenGenerator.java} | 49 +-
.../token/ContainerTokenSecretManager.java | 30 +-
.../security/token/ContainerTokenVerifier.java | 4 +-
.../hdds/security/token/NoopTokenVerifier.java | 6 +-
.../security/token/ShortLivedTokenVerifier.java | 9 +-
.../hadoop/hdds/security/token/TokenVerifier.java | 11 +-
.../x509/certificate/client/CertificateClient.java | 26 +-
.../client/DefaultCertificateClient.java | 65 +++
.../certificate/client/OMCertificateClient.java | 11 +-
.../org/apache/hadoop/hdds/utils/LevelDBStore.java | 400 --------------
.../hadoop/hdds/utils/LevelDBStoreIterator.java | 64 ---
.../apache/hadoop/hdds/utils/MetadataStore.java | 233 --------
.../hadoop/hdds/utils/MetadataStoreBuilder.java | 151 ------
.../org/apache/hadoop/hdds/utils/RocksDBStore.java | 397 --------------
.../hadoop/hdds/utils/RocksDBStoreIterator.java | 66 ---
.../hadoop/hdds/utils/db/cache/TableCache.java | 3 +
.../hdds/scm/protocol/TestReplicationConfig.java | 66 ---
.../token/TestOzoneBlockTokenIdentifier.java | 2 +-
.../certificate/authority/TestDefaultCAServer.java | 2 +-
.../client/TestCertificateClientInit.java | 2 +-
.../client/TestDefaultCertificateClient.java | 4 +-
.../security/x509/keys/TestHDDSKeyGenerator.java | 2 +-
.../hdds/security/x509/keys/TestKeyCodec.java | 2 +-
.../hadoop/hdds/utils/TestMetadataStore.java | 592 ---------------------
.../hadoop/hdds/utils/TestRocksDBStoreMBean.java | 230 --------
.../hdds/utils/db/TestTypedRDBTableStore.java | 2 +-
.../hadoop/hdds/utils/db/cache/TestTableCache.java | 2 +-
hadoop-hdds/hadoop-dependency-client/pom.xml | 10 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 10 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 6 +-
hadoop-hdds/interface-admin/pom.xml | 10 +-
hadoop-hdds/interface-client/pom.xml | 8 +-
.../src/main/proto/SCMUpdateProtocol.proto | 98 ++++
hadoop-hdds/interface-server/pom.xml | 10 +-
.../proto/ScmServerDatanodeHeartbeatProtocol.proto | 13 +
.../src/main/proto/ScmServerProtocol.proto | 6 +
hadoop-hdds/pom.xml | 78 +--
hadoop-hdds/server-scm/pom.xml | 56 +-
.../hadoop/hdds/scm/block/BlockManagerImpl.java | 89 +---
.../scm/block/DeletedBlockLogStateManager.java | 3 +
.../scm/block/DeletedBlockLogStateManagerImpl.java | 3 +
.../scm/container/CloseContainerEventHandler.java | 9 +
.../hdds/scm/container/ContainerReportHandler.java | 53 +-
.../IncrementalContainerReportHandler.java | 54 +-
.../hdds/scm/container/ReplicationManager.java | 12 +-
.../scm/container/balancer/ContainerBalancer.java | 345 ++++++++++--
.../balancer/ContainerBalancerConfiguration.java | 52 +-
.../balancer/ContainerBalancerMetrics.java | 130 +++++
.../apache/hadoop/hdds/scm/ha/HASecurityUtils.java | 5 +-
.../hdds/scm/ha/MockSCMHADBTransactionBuffer.java | 4 +
.../org/apache/hadoop/hdds/scm/ha/SCMContext.java | 16 +-
.../hdds/scm/ha/SCMDBCheckpointProvider.java | 4 +-
.../hadoop/hdds/scm/ha/SCMHAManagerImpl.java | 1 +
.../hadoop/hdds/scm/ha/SCMHANodeDetails.java | 8 +
.../hadoop/hdds/scm/ha/SequenceIdGenerator.java | 29 +-
.../apache/hadoop/hdds/scm/ha/io/BooleanCodec.java | 3 +
.../org/apache/hadoop/hdds/scm/ha/io/Codec.java | 3 +
.../apache/hadoop/hdds/scm/ha/io/CodecFactory.java | 3 +
.../apache/hadoop/hdds/scm/ha/io/EnumCodec.java | 3 +
.../hdds/scm/ha/io/GeneratedMessageCodec.java | 3 +
.../apache/hadoop/hdds/scm/ha/io/ListCodec.java | 3 +
.../apache/hadoop/hdds/scm/ha/io/LongCodec.java | 3 +
.../apache/hadoop/hdds/scm/ha/io/StringCodec.java | 3 +
.../hadoop/hdds/scm/node/DatanodeUsageInfo.java | 3 +
.../hadoop/hdds/scm/node/DeadNodeHandler.java | 3 +
.../hdds/scm/node/NodeDecommissionManager.java | 31 +-
.../apache/hadoop/hdds/scm/node/NodeStatus.java | 2 +-
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 29 +-
.../scm/pipeline/WritableContainerFactory.java | 58 ++
.../scm/pipeline/WritableContainerProvider.java | 57 ++
.../pipeline/WritableRatisContainerProvider.java | 158 ++++++
.../hadoop/hdds/scm/server/SCMCertStore.java | 3 +
.../hdds/scm/server/SCMClientProtocolServer.java | 17 +-
.../hadoop/hdds/scm/server/SCMConfigurator.java | 20 +
.../hdds/scm/server/SCMDatanodeProtocolServer.java | 4 +-
.../apache/hadoop/hdds/scm/server/SCMMXBean.java | 10 +
.../hdds/scm/server/SCMSecurityProtocolServer.java | 25 +-
.../hdds/scm/server/StorageContainerManager.java | 68 ++-
.../hadoop/hdds/scm/update/server/SCMCRLStore.java | 51 ++
.../scm/update/server/SCMCRLUpdateHandler.java | 151 ++++++
.../scm/update/server/SCMUpdateClientManager.java | 149 ++++++
.../hdds/scm/update/server/SCMUpdateHandler.java | 53 ++
.../update/server/SCMUpdateServiceGrpcServer.java | 99 ++++
.../scm/update/server/SCMUpdateServiceImpl.java | 119 +++++
.../Codec.java => update/server/package-info.java} | 24 +-
.../main/resources/webapps/scm/scm-overview.html | 8 +
.../scm/TestStorageContainerManagerHttpServer.java | 2 +-
.../hadoop/hdds/scm/block/TestBlockManager.java | 2 +-
.../hadoop/hdds/scm/block/TestDeletedBlockLog.java | 2 +-
.../command/TestCommandStatusReportHandler.java | 2 +-
.../hadoop/hdds/scm/container/MockNodeManager.java | 88 ++-
.../container/TestCloseContainerEventHandler.java | 2 +-
.../scm/container/TestContainerManagerImpl.java | 2 +-
.../scm/container/TestContainerReportHandler.java | 4 +-
.../TestIncrementalContainerReportHandler.java | 98 +++-
.../hdds/scm/container/TestReplicationManager.java | 2 +-
.../scm/container/TestSCMContainerManager.java | 2 +-
.../container/balancer/TestContainerBalancer.java | 222 ++++++++
.../apache/hadoop/hdds/scm/ha/TestSCMContext.java | 6 +-
.../hadoop/hdds/scm/ha/TestSCMHAConfiguration.java | 3 +
.../hadoop/hdds/scm/ha/TestSCMServiceManager.java | 7 +-
.../hdds/scm/ha/TestSequenceIDGenerator.java | 3 +
.../hadoop/hdds/scm/node/TestDeadNodeHandler.java | 8 +-
.../hdds/scm/node/TestNodeDecommissionManager.java | 35 +-
.../hdds/scm/node/TestNodeReportHandler.java | 2 +-
.../hadoop/hdds/scm/node/TestSCMNodeManager.java | 2 +-
.../hdds/scm/node/TestSCMNodeStorageStatMap.java | 2 +-
.../hadoop/hdds/scm/node/TestStatisticsUpdate.java | 2 +-
.../hdds/scm/pipeline/TestPipelineManagerImpl.java | 2 +-
.../scm/pipeline/TestPipelinePlacementPolicy.java | 25 +
.../hdds/scm/pipeline/TestSCMPipelineManager.java | 4 +-
.../safemode/TestHealthyPipelineSafeModeRule.java | 2 +-
.../TestOneReplicaPipelineSafeModeRule.java | 2 +-
.../hdds/scm/safemode/TestSCMSafeModeManager.java | 4 +-
.../scm/server/TestSCMBlockProtocolServer.java | 2 +-
.../scm/server/TestSCMSecurityProtocolServer.java | 4 +-
.../hdds/scm/update/server/MockCRLStore.java | 138 +++++
.../server/TestSCMUpdateServiceGrpcServer.java | 301 +++++++++++
.../ozone/container/common/TestEndPoint.java | 4 +-
hadoop-hdds/test-utils/pom.xml | 6 +-
.../{hadoop => ozone}/test/GenericTestUtils.java | 2 +-
.../{hadoop => ozone}/test/LambdaTestUtils.java | 2 +-
.../test/TimedOutTestsListener.java | 2 +-
.../java/org/apache/ozone/test/package-info.java} | 12 +-
hadoop-hdds/tools/pom.xml | 26 +-
.../hdds/scm/cli/ContainerOperationClient.java | 42 +-
.../org/apache/hadoop/hdds/scm/cli/ScmOption.java | 14 +-
hadoop-ozone/client/pom.xml | 14 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 112 ++--
.../org/apache/hadoop/ozone/client/OzoneKey.java | 41 +-
.../hadoop/ozone/client/OzoneKeyDetails.java | 20 +
.../hadoop/ozone/client/OzoneMultipartUpload.java | 34 +-
.../client/OzoneMultipartUploadPartListParts.java | 34 +-
.../client/io/BlockOutputStreamEntryPool.java | 7 +-
.../hadoop/ozone/client/io/KeyInputStream.java | 51 +-
.../hadoop/ozone/client/io/KeyOutputStream.java | 30 +-
.../client/io/MultipartCryptoKeyInputStream.java | 3 +
.../hadoop/ozone/client/io/OzoneInputStream.java | 15 +-
.../ozone/client/protocol/ClientProtocol.java | 58 ++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 85 +--
.../hadoop/ozone/client/TestOzoneClient.java | 2 +-
hadoop-ozone/common/pom.xml | 26 +-
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 3 +-
.../hadoop/ozone/client/io/LengthInputStream.java | 17 +-
.../apache/hadoop/ozone/om/helpers/OmKeyArgs.java | 46 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 57 +-
.../ozone/om/helpers/OmMultipartKeyInfo.java | 59 +-
.../hadoop/ozone/om/helpers/OmMultipartUpload.java | 32 +-
.../om/helpers/OmMultipartUploadListParts.java | 25 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 71 ++-
.../org/apache/hadoop/ozone/TestOzoneAcls.java | 2 +-
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 10 +-
.../ozone/om/helpers/TestOmMultipartKeyInfo.java | 5 +-
hadoop-ozone/csi/pom.xml | 16 +-
hadoop-ozone/datanode/pom.xml | 14 +-
hadoop-ozone/dev-support/checks/coverage.sh | 3 +-
hadoop-ozone/dev-support/checks/integration.sh | 2 +-
hadoop-ozone/dev-support/checks/unit.sh | 2 +-
hadoop-ozone/dev-support/docker/Dockerfile | 2 +-
.../intellij/runConfigurations/CsiServer.xml | 4 +-
.../intellij/runConfigurations/Datanode1.xml | 2 +-
.../intellij/runConfigurations/Datanode2.xml | 2 +-
.../intellij/runConfigurations/Datanode3.xml | 2 +-
.../intellij/runConfigurations/FreonStandalone.xml | 4 +-
.../intellij/runConfigurations/OzoneFsShell.xml | 2 +-
.../intellij/runConfigurations/OzoneManager.xml | 2 +-
.../runConfigurations/OzoneManagerInit.xml | 2 +-
.../intellij/runConfigurations/OzoneShell.xml | 2 +-
.../intellij/runConfigurations/Recon.xml | 4 +-
.../intellij/runConfigurations/S3Gateway.xml | 4 +-
.../runConfigurations/StorageContainerManager.xml | 2 +-
.../StorageContainerManagerInit.xml | 2 +-
.../dist/dev-support/bin/dist-layout-stitching | 3 +
.../dist/dev-support/bin/dist-tar-stitching | 6 +-
hadoop-ozone/dist/pom.xml | 73 +--
.../docker-image/docker-krb5/Dockerfile-krb5 | 35 --
.../common/docker-image/docker-krb5/README.md | 34 --
.../common/docker-image/docker-krb5/kadm5.acl | 20 -
.../common/docker-image/docker-krb5/launcher.sh | 25 -
.../dist/src/main/compose/ozone-ha/docker-config | 1 +
.../main/compose/ozone-mr/hadoop27/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop31/docker-config | 4 +-
.../main/compose/ozone-mr/hadoop32/docker-config | 4 +-
.../dist/src/main/compose/ozonesecure-ha/.env | 1 +
.../compose/ozonesecure-ha/docker-compose.yaml | 44 +-
.../src/main/compose/ozonesecure-ha/docker-config | 24 +-
.../compose/ozonesecure-ha/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-ha/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-ha/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-ha/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-ha/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-ha/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-ha/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-ha/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-ha}/krb5.conf | 4 +-
.../dist/src/main/compose/ozonesecure-mr/.env | 1 +
.../dist/src/main/compose/ozonesecure-mr/README.md | 2 +-
.../compose/ozonesecure-mr/docker-compose.yaml | 30 +-
.../src/main/compose/ozonesecure-mr/docker-config | 25 +-
.../compose/ozonesecure-mr/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../main/compose/ozonesecure-mr/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure-mr/keytabs/om.keytab | Bin 0 -> 278 bytes
.../compose/ozonesecure-mr/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure-mr/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure-mr/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure-mr/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../ozonesecure-mr/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure-mr}/krb5.conf | 6 +-
.../dist/src/main/compose/ozonesecure/.env | 1 +
.../main/compose/ozonesecure/docker-compose.yaml | 23 +-
.../src/main/compose/ozonesecure/docker-config | 24 +-
.../main/compose/ozonesecure/keytabs/HTTP.keytab | Bin 0 -> 144 bytes
.../src/main/compose/ozonesecure/keytabs/dn.keytab | Bin 0 -> 278 bytes
.../src/main/compose/ozonesecure/keytabs/om.keytab | Bin 0 -> 278 bytes
.../main/compose/ozonesecure/keytabs/recon.keytab | Bin 0 -> 296 bytes
.../main/compose/ozonesecure/keytabs/s3g.keytab | Bin 0 -> 434 bytes
.../main/compose/ozonesecure/keytabs/scm.keytab | Bin 0 -> 586 bytes
.../compose/ozonesecure/keytabs/testuser.keytab | Bin 0 -> 152 bytes
.../compose/ozonesecure/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../docker-krb5 => ozonesecure}/krb5.conf | 4 +-
hadoop-ozone/dist/src/main/compose/testlib.sh | 20 +-
.../definitions/ozone/definitions/persistence.yaml | 4 +-
.../examples/ozone-ha/datanode-statefulset.yaml | 2 +-
.../k8s/examples/ozone/datanode-statefulset.yaml | 2 +-
hadoop-ozone/dist/src/main/keytabs/HTTP.keytab | Bin 0 -> 432 bytes
hadoop-ozone/dist/src/main/keytabs/dn.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/hadoop.keytab | Bin 0 -> 146 bytes
hadoop-ozone/dist/src/main/keytabs/jhs.keytab | Bin 0 -> 142 bytes
hadoop-ozone/dist/src/main/keytabs/nm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/om.keytab | Bin 0 -> 278 bytes
hadoop-ozone/dist/src/main/keytabs/recon.keytab | Bin 0 -> 296 bytes
hadoop-ozone/dist/src/main/keytabs/rm.keytab | Bin 0 -> 138 bytes
hadoop-ozone/dist/src/main/keytabs/s3g.keytab | Bin 0 -> 434 bytes
hadoop-ozone/dist/src/main/keytabs/scm.keytab | Bin 0 -> 586 bytes
hadoop-ozone/dist/src/main/keytabs/testuser.keytab | Bin 0 -> 450 bytes
.../dist/src/main/keytabs/testuser2.keytab | Bin 0 -> 154 bytes
.../dist/src/main/keytabs/update-keytabs.sh | 77 +++
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 3 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 59 +-
.../dist/src/main/license/update-jar-report.sh | 2 +-
.../src/main/smoketest/admincli/container.robot | 11 +-
.../dist/src/main/smoketest/basic/links.robot | 10 +-
.../src/main/smoketest/basic/ozone-shell-lib.robot | 6 +-
.../dist/src/main/smoketest/cli/classpath.robot | 14 +-
.../dist/src/main/smoketest/commonlib.robot | 8 +-
.../dist/src/main/smoketest/ozonefs/setup.robot | 10 +-
.../main/smoketest/security/ozone-secure-fs.robot | 18 +-
hadoop-ozone/dist/src/shell/ozone/ozone | 42 +-
.../dist/src/test/shell/compose_testlib.bats | 13 +
.../dist/src/test/shell/failing1/test.sh | 7 +-
.../fault-injection-test/mini-chaos-tests/pom.xml | 24 +-
.../apache/hadoop/ozone/MiniOzoneChaosCluster.java | 2 +-
.../fault-injection-test/network-tests/pom.xml | 6 +-
hadoop-ozone/fault-injection-test/pom.xml | 6 +-
hadoop-ozone/insight/pom.xml | 42 +-
.../hadoop/ozone/insight/BaseInsightPoint.java | 4 +-
hadoop-ozone/integration-test/pom.xml | 70 +--
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 7 +-
.../fs/ozone/TestOzoneFileSystemMissingParent.java | 6 +-
.../apache/hadoop/fs/ozone/TestOzoneFsHAURLs.java | 2 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 4 +-
.../hadoop/hdds/scm/TestRatisPipelineLeader.java | 2 +-
.../hadoop/hdds/scm/TestSCMInstallSnapshot.java | 2 +-
.../apache/hadoop/hdds/scm/TestSCMSnapshot.java | 3 +
.../metrics/TestSCMContainerManagerMetrics.java | 2 +-
.../hdds/scm/pipeline/TestLeaderChoosePolicy.java | 4 +-
.../hadoop/hdds/scm/pipeline/TestNodeFailure.java | 2 +-
.../hdds/scm/pipeline/TestPipelineClose.java | 2 +-
.../TestRatisPipelineCreateAndDestroy.java | 2 +-
.../safemode/TestSCMSafeModeWithPipelineRules.java | 2 +-
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 2 +-
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 2 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 12 +-
.../org/apache/hadoop/ozone/OzoneTestUtils.java | 2 +-
.../hadoop/ozone/TestContainerOperations.java | 21 +-
.../apache/hadoop/ozone/TestDelegationToken.java | 6 +-
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 8 +-
.../hadoop/ozone/TestMiniOzoneOMHACluster.java | 2 +-
.../hadoop/ozone/TestSecureOzoneCluster.java | 34 +-
.../hadoop/ozone/TestStorageContainerManager.java | 2 +-
.../ozone/client/CertificateClientTestImpl.java | 14 +
.../ozone/client/rpc/Test2WayCommitInRatis.java | 2 +-
.../apache/hadoop/ozone/client/rpc/TestBCSID.java | 9 +-
.../rpc/TestCloseContainerHandlingByClient.java | 39 +-
.../rpc/TestContainerReplicationEndToEnd.java | 2 +-
.../client/rpc/TestContainerStateMachine.java | 2 +-
.../TestContainerStateMachineFailureOnRead.java | 2 +-
.../rpc/TestContainerStateMachineFailures.java | 2 +-
.../rpc/TestContainerStateMachineFlushDelay.java | 2 +-
.../client/rpc/TestDeleteWithSlowFollower.java | 10 +-
.../client/rpc/TestFailureHandlingByClient.java | 28 +-
.../rpc/TestFailureHandlingByClientFlushDelay.java | 7 +-
.../rpc/TestMultiBlockWritesWithDnFailures.java | 13 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 2 +-
.../client/rpc/TestOzoneRpcClientAbstract.java | 4 +-
.../rpc/TestOzoneRpcClientForAclAuditLog.java | 2 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 4 +-
.../ozone/client/rpc/TestWatchForCommit.java | 2 +-
.../ozone/client/rpc/read/TestInputStreamBase.java | 5 +-
.../ozone/client/rpc/read/TestKeyInputStream.java | 71 ++-
.../ozone/container/TestContainerReplication.java | 6 +-
.../apache/hadoop/ozone/container/TestHelper.java | 2 +-
.../commandhandler/TestBlockDeletion.java | 13 +-
.../TestCloseContainerByPipeline.java | 23 +-
.../commandhandler/TestCloseContainerHandler.java | 9 +-
.../commandhandler/TestDeleteContainerHandler.java | 8 +-
.../transport/server/ratis/TestCSMMetrics.java | 2 +-
.../container/metrics/TestContainerMetrics.java | 4 +-
.../ozoneimpl/TestOzoneContainerWithTLS.java | 2 +-
.../ozoneimpl/TestSecureOzoneContainer.java | 2 +-
.../hadoop/ozone/container/package-info.java | 30 +-
.../container/server/TestContainerServer.java | 2 +-
.../server/TestSecureContainerServer.java | 75 ++-
.../apache/hadoop/ozone/dn/DatanodeTestUtils.java | 33 +-
.../ozone/dn/ratis/TestDnRatisLogParser.java | 2 +-
.../TestDatanodeHddsVolumeFailureDetection.java | 6 +
.../TestDatanodeHddsVolumeFailureToleration.java | 131 +++++
.../ozone/freon/TestHadoopDirTreeGenerator.java | 2 +-
.../ozone/freon/TestHadoopNestedDirGenerator.java | 4 +-
.../hadoop/ozone/fsck/TestContainerMapper.java | 2 +-
.../ozone/om/TestContainerReportWithKeys.java | 6 +-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 35 +-
.../org/apache/hadoop/ozone/om/TestKeyPurging.java | 2 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 2 +-
.../org/apache/hadoop/ozone/om/TestOmAcls.java | 2 +-
.../hadoop/ozone/om/TestOmBlockVersioning.java | 4 +
.../ozone/om/TestOzoneManagerConfiguration.java | 2 +-
.../apache/hadoop/ozone/om/TestOzoneManagerHA.java | 2 +-
.../ozone/om/TestOzoneManagerHAKeyDeletion.java | 5 +-
.../ozone/om/TestOzoneManagerHAMetadataOnly.java | 2 +-
.../ozone/om/TestOzoneManagerHAWithData.java | 2 +-
.../hadoop/ozone/om/TestOzoneManagerRestart.java | 2 +-
.../ozone/om/TestOzoneManagerRocksDBLogging.java | 2 +-
.../apache/hadoop/ozone/om/TestScmSafeMode.java | 4 +-
.../hadoop/ozone/om/TestSecureOzoneManager.java | 4 +-
.../ozone/om/parser/TestOMRatisLogParser.java | 2 +-
.../hadoop/ozone/recon/TestReconAsPassiveScm.java | 4 +-
.../apache/hadoop/ozone/recon/TestReconTasks.java | 2 +-
.../ozone/recon/TestReconWithOzoneManager.java | 5 +-
.../ozone/recon/TestReconWithOzoneManagerHA.java | 2 +-
.../hadoop/ozone/scm/TestCloseContainer.java | 2 +-
.../hadoop/ozone/scm/TestFailoverWithSCMHA.java | 2 +-
.../ozone/scm/TestSCMInstallSnapshotWithHA.java | 2 +-
.../ozone/scm/TestStorageContainerManagerHA.java | 9 +-
.../hadoop/ozone/scm/TestXceiverClientManager.java | 2 +-
.../hadoop/ozone/scm/TestXceiverClientMetrics.java | 2 +-
.../scm/node/TestDecommissionAndMaintenance.java | 2 +-
.../hadoop/ozone/scm/node/TestQueryNode.java | 2 +-
.../TestSCMPipelineBytesWrittenMetrics.java | 2 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 103 +++-
hadoop-ozone/interface-client/pom.xml | 10 +-
hadoop-ozone/interface-storage/pom.xml | 30 +-
.../hadoop/ozone/om/codec/TestOmKeyInfoCodec.java | 5 +-
.../om/codec/TestOmMultipartKeyInfoCodec.java | 7 +-
.../ozone/om/codec/TestOmPrefixInfoCodec.java | 2 +-
.../ozone/om/codec/TestRepeatedOmKeyInfoCodec.java | 5 +-
.../ozone/om/codec/TestS3SecretValueCodec.java | 2 +-
.../ozone/om/codec/TestTransactionInfoCodec.java | 2 +-
hadoop-ozone/native-client/README.md | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 60 +--
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 7 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 81 ++-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 14 +-
.../java/org/apache/hadoop/ozone/om/ScmClient.java | 11 +
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 2 +-
.../apache/hadoop/ozone/om/TrashPolicyOzone.java | 24 +
.../om/request/file/OMDirectoryCreateRequest.java | 6 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 2 +-
.../request/file/OMFileCreateRequestWithFSO.java | 2 +-
.../ozone/om/request/file/OMFileRequest.java | 6 +-
.../om/request/key/OMAllocateBlockRequest.java | 2 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 2 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 2 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 2 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 2 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 27 +-
.../S3InitiateMultipartUploadRequest.java | 12 +-
.../S3InitiateMultipartUploadRequestWithFSO.java | 10 +-
.../multipart/S3MultipartUploadAbortRequest.java | 2 +-
.../S3MultipartUploadCommitPartRequest.java | 4 +-
.../S3MultipartUploadCompleteRequest.java | 15 +-
.../om/request/s3/security/S3GetSecretRequest.java | 9 +-
.../request/s3/security/S3RevokeSecretRequest.java | 10 +-
.../protocolPB/OzoneManagerRequestHandler.java | 16 +-
.../hadoop/ozone/om/TestBucketManagerImpl.java | 10 +-
.../hadoop/ozone/om/TestKeyDeletingService.java | 6 +-
.../apache/hadoop/ozone/om/TestKeyManagerUnit.java | 21 +-
.../org/apache/hadoop/ozone/om/TestOMStorage.java | 2 +-
.../ozone/om/TestOzoneManagerHttpServer.java | 2 +-
.../apache/hadoop/ozone/om/TestTrashService.java | 4 +
.../hadoop/ozone/om/failover/TestOMFailovers.java | 2 +-
...tOzoneManagerDoubleBufferWithDummyResponse.java | 2 +-
...TestOzoneManagerDoubleBufferWithOMResponse.java | 2 +-
.../om/ratis/TestOzoneManagerRatisServer.java | 2 +-
.../ozone/om/request/TestOMRequestUtils.java | 12 +-
.../request/bucket/TestOMBucketCreateRequest.java | 2 +-
.../bucket/TestOMBucketSetPropertyRequest.java | 2 +-
.../request/volume/TestOMVolumeCreateRequest.java | 4 +-
.../volume/TestOMVolumeSetQuotaRequest.java | 2 +-
.../s3/multipart/TestS3MultipartResponse.java | 25 +-
.../security/TestOzoneBlockTokenSecretManager.java | 4 +-
.../TestOzoneDelegationTokenSecretManager.java | 2 +-
.../ozone/security/TestOzoneManagerBlockToken.java | 2 +-
.../ozone/security/TestOzoneTokenIdentifier.java | 2 +-
.../security/acl/TestOzoneNativeAuthorizer.java | 7 +-
.../hadoop/ozone/security/acl/TestVolumeOwner.java | 10 +-
hadoop-ozone/ozonefs-common/pom.xml | 42 +-
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 35 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 35 +-
.../apache/hadoop/fs/ozone/OzoneFSInputStream.java | 30 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 16 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 14 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 11 +-
hadoop-ozone/ozonefs/pom.xml | 14 +-
hadoop-ozone/pom.xml | 142 ++---
hadoop-ozone/recon-codegen/pom.xml | 10 +-
hadoop-ozone/recon/pom.xml | 40 +-
.../ozone/recon/persistence/ContainerHistory.java | 3 +
.../hadoop/ozone/recon/scm/PipelineSyncTask.java | 46 +-
.../ozone/recon/scm/ReconDeadNodeHandler.java | 79 +++
.../hadoop/ozone/recon/scm/ReconNodeManager.java | 20 +
.../scm/ReconStorageContainerManagerFacade.java | 5 +-
.../recon/spi/StorageContainerServiceProvider.java | 6 +
.../impl/StorageContainerServiceProviderImpl.java | 8 +
.../hadoop/ozone/recon/tasks/ReconTaskConfig.java | 4 +-
.../src/views/datanodes/datanodes.tsx | 2 +-
.../ozone/recon/OMMetadataManagerTestUtils.java | 7 +-
.../hadoop/ozone/recon/api/TestEndpoints.java | 2 +-
.../ozone/recon/fsck/TestContainerHealthTask.java | 2 +-
.../recovery/TestReconOmMetadataManagerImpl.java | 9 +-
...TestReconIncrementalContainerReportHandler.java | 2 +-
.../ozone/recon/scm/TestReconNodeManager.java | 41 +-
.../recon/tasks/TestContainerKeyMapperTask.java | 5 +-
.../ozone/recon/tasks/TestOMDBUpdatesHandler.java | 5 +-
hadoop-ozone/s3gateway/pom.xml | 42 +-
.../org/apache/hadoop/ozone/s3/endpoint/S3Acl.java | 6 +
.../hadoop/ozone/s3/endpoint/S3BucketAcl.java | 6 +
.../apache/hadoop/ozone/s3/endpoint/S3Owner.java | 3 +
.../hadoop/ozone/s3/signature/SignatureInfo.java | 3 +
.../hadoop/ozone/client/OzoneBucketStub.java | 5 +-
.../ozone/s3/TestEmptyContentTypeFilter.java | 5 +-
.../ozone/s3/TestVirtualHostStyleFilter.java | 2 +-
.../signature/TestAuthorizationV4HeaderParser.java | 2 +-
.../signature/TestAuthorizationV4QueryParser.java | 5 +-
.../hadoop/ozone/s3/util/TestOzoneS3Util.java | 2 +-
hadoop-ozone/tools/pom.xml | 50 +-
.../ozone/debug/container/ContainerCommands.java | 2 +-
.../ozone/debug/container/ExportSubcommand.java | 3 +
.../ozone/debug/container/InfoSubcommand.java | 3 +
.../ozone/debug/container/ListSubcommand.java | 3 +
.../hadoop/ozone/freon/ChunkManagerDiskWrite.java | 2 +-
.../ozone/freon/ClosedContainerReplicator.java | 3 +-
.../java/org/apache/hadoop/ozone/freon/Freon.java | 3 +-
.../apache/hadoop/ozone/freon/OmKeyGenerator.java | 5 +-
.../hadoop/ozone/freon/StreamingGenerator.java | 147 +++++
.../freon/containergenerator/BaseGenerator.java | 3 +
.../containergenerator/GeneratorDatanode.java | 2 +-
.../freon/containergenerator/GeneratorOm.java | 11 +-
.../hadoop/ozone/genesis/BenchMarkCRCBatch.java | 3 +
.../ozone/genesis/BenchMarkCRCStreaming.java | 3 +
.../ozone/genesis/BenchMarkDatanodeDispatcher.java | 2 +-
.../ozone/genesis/BenchMarkMetadataStoreReads.java | 70 ---
.../genesis/BenchMarkMetadataStoreWrites.java | 62 ---
.../ozone/genesis/BenchMarkOMKeyAllocation.java | 6 +-
.../ozone/genesis/BenchMarkOzoneManager.java | 13 +-
.../ozone/genesis/BenchMarkRocksDbStore.java | 119 -----
.../apache/hadoop/ozone/genesis/GenesisUtil.java | 32 --
.../org/apache/hadoop/ozone/scm/cli/SQLCLI.java | 562 -------------------
.../hadoop/ozone/shell/keys/CopyKeyHandler.java | 28 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 27 +-
.../hadoop/ozone/audit/parser/TestAuditParser.java | 31 +-
.../TestGenerateOzoneRequiredConfigurations.java | 2 +-
.../tools/src/test/resources/testaudit.log | 3 +
.../resources/{testaudit.log => testloadaudit.log} | 24 +-
pom.xml | 16 +-
619 files changed, 8758 insertions(+), 5790 deletions(-)
delete mode 100644 .github/buildenv/Dockerfile
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteArrayReader.java
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteBufferReader.java
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/utils/EntryConsumer.java
=>
client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteReaderStrategy.java}
(66%)
create mode 100644
hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/client/TestReplicationConfig.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/CRLStatusReportPublisher.java
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java
=>
container-service/src/main/java/org/apache/hadoop/ozone/container/stream/DirectoryServerDestination.java}
(61%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/stream/DirectoryServerSource.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/stream/DirstreamClientHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/stream/DirstreamServerHandler.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingClient.java
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java
=>
container-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingDestination.java}
(68%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingServer.java
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java
=>
container-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingSource.java}
(60%)
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java
=>
container-service/src/main/java/org/apache/hadoop/ozone/container/stream/package-info.java}
(77%)
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/stream/TestStreamingServer.java
create mode 100644 hadoop-hdds/docs/content/feature/scm-secure-ha.png
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/CRLClientUpdateHandler.java
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/Codec.java
=>
framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/CRLStore.java}
(65%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/ClientCRLStore.java
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/Codec.java
=>
framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/ClientUpdateHandler.java}
(64%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/SCMUpdateClientConfiguration.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/SCMUpdateServiceGrpcClient.java
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/LongCodec.java
=>
framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/UpdateServiceConfig.java}
(53%)
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/Codec.java
=>
framework/src/main/java/org/apache/hadoop/hdds/scm/update/client/package-info.java}
(64%)
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/server/CRLClientInfo.java
create mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMUpdateClientInfo.java
copy
hadoop-hdds/{server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/io/Codec.java
=>
framework/src/main/java/org/apache/hadoop/hdds/scm/update/server/package-info.java}
(64%)
copy
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/security/token/{CompositeTokenVerifier.java
=> ContainerTokenGenerator.java} (50%)
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStore.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/LevelDBStoreIterator.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStore.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/MetadataStoreBuilder.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStore.java
delete mode 100644
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/RocksDBStoreIterator.java
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/scm/protocol/TestReplicationConfig.java
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestMetadataStore.java
delete mode 100644
hadoop-hdds/framework/src/test/java/org/apache/hadoop/hdds/utils/TestRocksDBStoreMBean.java
create mode 100644
hadoop-hdds/interface-client/src/main/proto/SCMUpdateProtocol.proto
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerMetrics.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableContainerFactory.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableContainerProvider.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableRatisContainerProvider.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMCRLStore.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMCRLUpdateHandler.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMUpdateClientManager.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMUpdateHandler.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMUpdateServiceGrpcServer.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/update/server/SCMUpdateServiceImpl.java
copy
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/{ha/io/Codec.java
=> update/server/package-info.java} (64%)
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/balancer/TestContainerBalancer.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/update/server/MockCRLStore.java
create mode 100644
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/update/server/TestSCMUpdateServiceGrpcServer.java
rename hadoop-hdds/test-utils/src/main/java/org/apache/{hadoop =>
ozone}/test/GenericTestUtils.java (99%)
rename hadoop-hdds/test-utils/src/main/java/org/apache/{hadoop =>
ozone}/test/LambdaTestUtils.java (99%)
rename hadoop-hdds/test-utils/src/main/java/org/apache/{hadoop =>
ozone}/test/TimedOutTestsListener.java (99%)
copy
hadoop-hdds/{common/src/main/java/org/apache/hadoop/hdds/scm/ha/RetriableWithNoFailoverException.java
=> test-utils/src/main/java/org/apache/ozone/test/package-info.java} (77%)
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/Dockerfile-krb5
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/README.md
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/kadm5.acl
delete mode 100644
hadoop-ozone/dist/src/main/compose/common/docker-image/docker-krb5/launcher.sh
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure-ha}/krb5.conf (96%)
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure-mr/keytabs/testuser2.keytab
copy hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure-mr}/krb5.conf (91%)
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/HTTP.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/dn.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/om.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/recon.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/s3g.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/scm.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser.keytab
create mode 100755
hadoop-ozone/dist/src/main/compose/ozonesecure/keytabs/testuser2.keytab
rename hadoop-ozone/dist/src/main/compose/{common/docker-image/docker-krb5 =>
ozonesecure}/krb5.conf (96%)
create mode 100755 hadoop-ozone/dist/src/main/keytabs/HTTP.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/dn.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/hadoop.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/jhs.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/nm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/om.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/recon.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/rm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/s3g.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/scm.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/testuser2.keytab
create mode 100755 hadoop-ozone/dist/src/main/keytabs/update-keytabs.sh
rename .github/buildenv/entrypoint.sh =>
hadoop-ozone/dist/src/test/shell/failing1/test.sh (85%)
mode change 100755 => 100644
rename
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/EntryConsumer.java
=>
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/package-info.java
(54%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/volume/TestDatanodeHddsVolumeFailureToleration.java
create mode 100644
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconDeadNodeHandler.java
create mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/StreamingGenerator.java
delete mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkMetadataStoreReads.java
delete mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkMetadataStoreWrites.java
delete mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkRocksDbStore.java
delete mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/scm/cli/SQLCLI.java
copy hadoop-ozone/tools/src/test/resources/{testaudit.log =>
testloadaudit.log} (69%)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
