This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git.
omit 871f2cd HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
omit 976b141 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
omit 4480394 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
omit 26dd5cd HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
omit 51664a3 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
omit 64dab05 HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
omit c6950c7 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
omit f6f8715 HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
omit 8f9924e HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
omit b9368a5 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
omit 6211d74 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
omit 7787247 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
omit 471cadb HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add a5d6707 HDDS-5901. Delete on volume/bucket throws fatal error on ofs
(#2776)
add 26824ca HDDS-5884. OM Validate S3 Auth for write requests. (#2778)
add d018d00 HDDS-5805. remove containerStateManager V1 code (#2727)
add f153236 HDDS-5863. Error message having null fields on volume
creation (#2752)
add 5c76856 HDDS-5885. OM Auth Validate for read requests and handle non
ratis enabled code. (#2779)
add 5f0a588 HDDS-5908. MPU getKey can fail, if completeMPU result is
still in cache. (#2787)
add e427b71 HDDS-5883 Change S3G client to set S3 Auth per req (#2775)
add 73a081a HDDS-5864. Retry when DN connection issue during
getBlock/ReadChunk call during Ozone key Read (#2746)
add 83f8b04 HDDS-5736. Update navbar from Hadoop to Ozone on doc page
(#2792)
add 47cb26a HDDS-3983 Ozone RocksDB Iterator wrapper should not expose
key() and value() API. (#2402)
add 565972c Support more detailed error log when handleFlush in Ozone
client (#2795)
add 1cf43ac HDDS-5931. Broken link in dist/README (#2799)
add 9cea16d HDDS-5933. Ozone Filesystem shaded jars include unnecessary
dependencies (#2798)
add eb7136d HDDS-3369. Cleanup old write-path of volume in OM. (#2780)
add c088cd1 HDDS-5925. Fix SecuringOzoneHTTP doc (#2796)
add ac86851 HDDS-5943. Shutdown ResultHandlerExecutorService for
StorageVolumeChecker. (#2806)
add aa7e083 HDDS-5944. Fix typo in OzoneManagerServiceProviderImpl.java
(#2807)
add 6ebb316 HDDS-5945. bump rocksdb version to 6.25.3 (#2809)
add 4c2181f HDDS-5935. Bump Spring to 5.2.18 (#2801)
add f3b482a HDDS-5922. ignore deletetransaction when container is not
found (#2793)
add c63479c HDDS-5839. Make sure buckets created from OFS are in
FILE_SYSTEM_OPTIMIZED layout (#2730)
add 96e2d7f HDDS-5947. Remove unused mina-core and sshd-core dependencies
(#2811)
add 252fee4 HDDS-5910 Add additional verification for S3 Auth. (#2817)
add 9d9e780 HDDS-5956. Speed up
TestOzoneRpcClientAbstract#testZReadKeyWithUnhealthyContainerReplica (#2820)
add 0eedab9 HDDS-5929. Make FSO and OBS bucket layouts independent of
normalization config flag (#2819)
add ba3ac30 HDDS-5960. Change option name to bucketlayout instead of type
(#2825)
add 4558e40 HDDS-5970. Remove OMKeyRequest#getBucketLayout overridden
method in subclasses (#2830)
add 9384bd7 HDDS-5820. Intermittent failure in
TestPipelinePlacementPolicy#testPickLowestLoadAnchor (#2757)
add d0e9140 HDDS-5958. Refine container scrub log message. (#2823)
add 45c1899 HDDS-5937. Inaccurate bucket info returned from bucket list
command in Shell (#2832)
add c2db377 HDDS-5872. Do not failover on some RpcExceptions (#2772)
add 71f47d4 HDDS-5981. Refactor usage of bucket type. (#2836)
add 026d878 HDDS-5982. Remove isBucketFSOptimized flag. (#2838)
new f508854 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new e166e91 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
new 0cc389a HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new 51d1013 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
new 6c36e82 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new 51f88cb HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
new d1cb3f9 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new bdd2726 HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new 5156051 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new ce6f566 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
new 0936be3 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
new 9037764 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
new 1d9f870 HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (871f2cd)
\
N -- N -- N refs/heads/HDDS-4454 (1d9f870)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 13 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../hadoop/hdds/scm/storage/BlockInputStream.java | 52 +-
.../hadoop/hdds/scm/storage/BlockOutputStream.java | 8 +
.../hadoop/hdds/scm/storage/ChunkInputStream.java | 2 +-
.../storage/DummyBlockInputStreamWithRetry.java | 14 +-
.../hdds/scm/storage/TestBlockInputStream.java | 32 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 51 ++
.../java/org/apache/hadoop/hdds/NodeDetails.java | 0
.../org/apache/hadoop/hdds/scm/ha/SCMHAUtils.java | 31 +-
.../hdds/scm/storage/ContainerProtocolCalls.java | 5 -
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 6 +
.../common/src/main/resources/ozone-default.xml | 15 +
.../common/volume/StorageVolumeChecker.java | 1 +
.../container/keyvalue/KeyValueContainerCheck.java | 2 +-
.../common/volume/TestStorageVolumeChecker.java | 4 +
hadoop-hdds/docs/content/feature/Topology.md | 2 +-
.../docs/content/security/SecuringOzoneHTTP.md | 20 +-
hadoop-hdds/docs/content/start/FromSource.md | 2 +-
.../themes/ozonedoc/layouts/partials/navbar.html | 6 +-
.../themes/ozonedoc/layouts/partials/sidebar.html | 4 +-
.../hadoop/hdds/utils/db/RDBStoreIterator.java | 19 +-
.../apache/hadoop/hdds/utils/db/TableIterator.java | 12 -
.../apache/hadoop/hdds/utils/db/TypedTable.java | 18 -
.../hadoop/hdds/utils/db/TestRDBStoreIterator.java | 19 +-
.../scm/block/DeletedBlockLogStateManagerImpl.java | 10 -
.../hdds/scm/container/ContainerManager.java | 2 +-
.../hdds/scm/container/ContainerManagerImpl.java | 4 +-
.../hdds/scm/container/ContainerStateManager.java | 570 ++++-----------------
.../scm/container/ContainerStateManagerImpl.java | 15 +-
.../scm/container/ContainerStateManagerV2.java | 189 -------
.../hdds/scm/pipeline/PipelinePlacementPolicy.java | 24 +-
.../java/org/apache/hadoop/hdds/scm/TestUtils.java | 18 +-
.../scm/container/TestContainerReportHandler.java | 267 +++++-----
.../scm/container/TestContainerStateManager.java | 91 +++-
.../TestIncrementalContainerReportHandler.java | 126 +++--
.../hdds/scm/container/TestReplicationManager.java | 370 +++++++------
.../scm/container/TestUnknownContainerReport.java | 47 +-
.../hdds/scm/ha/TestReplicationAnnotation.java | 8 +-
.../org/apache/hadoop/ozone/client/BucketArgs.java | 8 +-
.../ozone/client/protocol/ClientProtocol.java | 19 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 50 +-
hadoop-ozone/common/pom.xml | 4 -
.../main/java/org/apache/hadoop/ozone/OFSPath.java | 3 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 25 -
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 2 +-
.../ozone/om/ha/OMFailoverProxyProvider.java | 11 +-
.../hadoop/ozone/om/helpers/BucketLayout.java | 17 +
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 3 +-
.../hadoop/ozone/om/helpers/OzoneFSUtils.java | 7 -
.../OmDeleteKeys.java => protocol/S3Auth.java} | 43 +-
...nsport.java => OzoneManagerClientProtocol.java} | 33 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 59 ++-
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 41 +-
hadoop-ozone/dist/README.md | 6 +-
.../dist/src/main/compose/ozone-mr/common-config | 1 +
.../src/main/compose/ozonesecure/docker-config | 1 +
.../main/smoketest/security/ozone-secure-fs.robot | 2 +-
.../ozone/TestDirectoryDeletingServiceWithFSO.java | 13 +-
.../fs/ozone/TestOzoneFSWithObjectStoreCreate.java | 3 +
.../hadoop/fs/ozone/TestOzoneFileInterfaces.java | 3 +-
.../fs/ozone/TestOzoneFileInterfacesWithFSO.java | 25 +-
.../hadoop/fs/ozone/TestOzoneFileSystem.java | 5 +-
.../fs/ozone/TestOzoneFileSystemMetrics.java | 3 +
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 12 +-
.../fs/ozone/TestRootedOzoneFileSystemWithFSO.java | 7 -
.../TestContainerStateManagerIntegration.java | 2 +-
.../rpc/TestOzoneClientMultipartUploadWithFSO.java | 5 -
.../client/rpc/TestOzoneRpcClientAbstract.java | 10 +-
.../ozone/client/rpc/TestSecureOzoneRpcClient.java | 100 +++-
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 16 +-
.../hadoop/ozone/om/TestOMRatisSnapshots.java | 5 +-
.../hadoop/ozone/om/TestObjectStoreWithFSO.java | 7 +-
.../ozone/recon/TestReconWithOzoneManagerFSO.java | 6 +-
.../hadoop/ozone/shell/TestOzoneShellHA.java | 2 +-
.../src/main/proto/OmClientProtocol.proto | 2 +-
.../hadoop/ozone/om/codec/OmKeyInfoCodec.java | 5 +-
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 55 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 4 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 71 ++-
.../hadoop/ozone/om/TrashOzoneFileSystem.java | 5 +-
.../org/apache/hadoop/ozone/om/VolumeManager.java | 40 +-
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 300 -----------
.../ozone/om/ratis/OzoneManagerRatisServer.java | 6 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 158 +++---
.../hadoop/ozone/om/request/OMClientRequest.java | 22 +-
.../om/request/file/OMDirectoryCreateRequest.java | 9 +-
.../file/OMDirectoryCreateRequestWithFSO.java | 6 +-
.../ozone/om/request/file/OMFileCreateRequest.java | 5 +
.../request/file/OMFileCreateRequestWithFSO.java | 9 +-
.../ozone/om/request/file/OMFileRequest.java | 38 +-
.../om/request/key/OMAllocateBlockRequest.java | 15 +-
.../request/key/OMAllocateBlockRequestWithFSO.java | 10 +-
.../ozone/om/request/key/OMKeyCommitRequest.java | 12 +-
.../om/request/key/OMKeyCommitRequestWithFSO.java | 13 +-
.../ozone/om/request/key/OMKeyCreateRequest.java | 19 +-
.../om/request/key/OMKeyCreateRequestWithFSO.java | 10 +-
.../ozone/om/request/key/OMKeyDeleteRequest.java | 12 +-
.../om/request/key/OMKeyDeleteRequestWithFSO.java | 10 +-
.../ozone/om/request/key/OMKeyRenameRequest.java | 5 +
.../om/request/key/OMKeyRenameRequestWithFSO.java | 10 +-
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 10 +-
.../om/request/key/OMPathsPurgeRequestWithFSO.java | 10 +-
.../S3InitiateMultipartUploadRequest.java | 15 +-
.../S3InitiateMultipartUploadRequestWithFSO.java | 10 +-
.../multipart/S3MultipartUploadAbortRequest.java | 18 +-
.../S3MultipartUploadAbortRequestWithFSO.java | 10 +-
.../S3MultipartUploadCommitPartRequest.java | 19 +-
.../S3MultipartUploadCommitPartRequestWithFSO.java | 10 +-
.../S3MultipartUploadCompleteRequest.java | 23 +-
.../S3MultipartUploadCompleteRequestWithFSO.java | 10 +-
...OzoneManagerProtocolServerSideTranslatorPB.java | 48 +-
.../hadoop/ozone/security/S3SecurityUtil.java | 82 +++
.../hadoop/ozone/om/TestOmMetadataManager.java | 4 +-
.../ozone/om/request/TestOMRequestUtils.java | 22 +-
.../file/TestOMDirectoryCreateRequestWithFSO.java | 63 ++-
.../file/TestOMFileCreateRequestWithFSO.java | 17 +-
.../om/request/key/TestOMAllocateBlockRequest.java | 7 +-
.../key/TestOMAllocateBlockRequestWithFSO.java | 7 +-
.../om/request/key/TestOMKeyAclRequestWithFSO.java | 11 -
.../om/request/key/TestOMKeyCommitRequest.java | 59 ++-
.../request/key/TestOMKeyCommitRequestWithFSO.java | 17 +-
.../om/request/key/TestOMKeyCreateRequest.java | 38 +-
.../request/key/TestOMKeyCreateRequestWithFSO.java | 11 +-
.../om/request/key/TestOMKeyDeleteRequest.java | 17 +-
.../request/key/TestOMKeyDeleteRequestWithFSO.java | 3 +-
.../TestS3InitiateMultipartUploadRequest.java | 27 +-
...estS3InitiateMultipartUploadRequestWithFSO.java | 6 +-
.../s3/multipart/TestS3MultipartRequest.java | 33 +-
.../TestS3MultipartUploadAbortRequest.java | 3 +-
.../TestS3MultipartUploadAbortRequestWithFSO.java | 7 +-
.../TestS3MultipartUploadCommitPartRequest.java | 18 +-
...tS3MultipartUploadCommitPartRequestWithFSO.java | 13 +-
.../TestS3MultipartUploadCompleteRequest.java | 22 +-
...estS3MultipartUploadCompleteRequestWithFSO.java | 28 +-
.../file/TestOMFileCreateResponseWithFSO.java | 13 -
.../key/TestOMAllocateBlockResponseWithFSO.java | 13 -
.../key/TestOMKeyCommitResponseWithFSO.java | 13 -
.../key/TestOMKeyCreateResponseWithFSO.java | 13 -
.../key/TestOMKeyDeleteResponseWithFSO.java | 14 -
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 3 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 17 +-
.../fs/ozone/BasicRootedOzoneFileSystem.java | 4 +-
hadoop-ozone/pom.xml | 2 +-
.../spi/impl/OzoneManagerServiceProviderImpl.java | 4 +-
.../ozone/recon/api/TestNSSummaryEndpoint.java | 3 -
.../ozone/recon/tasks/TestNSSummaryTask.java | 2 -
.../hadoop/ozone/s3/OzoneClientProducer.java | 71 +--
.../hadoop/ozone/s3/endpoint/BucketEndpoint.java | 5 +
.../hadoop/ozone/s3/endpoint/EndpointBase.java | 29 +-
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 1 +
.../hadoop/ozone/s3/endpoint/RootEndpoint.java | 5 +
.../hadoop/ozone/s3/TestOzoneClientProducer.java | 4 +-
.../apache/hadoop/ozone/debug/PrefixParser.java | 2 -
.../ozone/genesis/BenchMarkOMKeyAllocation.java | 137 -----
.../org/apache/hadoop/ozone/genesis/Genesis.java | 2 +-
.../ozone/shell/bucket/CreateBucketHandler.java | 4 +-
pom.xml | 12 +-
156 files changed, 2076 insertions(+), 2471 deletions(-)
rename hadoop-hdds/{framework =>
common}/src/main/java/org/apache/hadoop/hdds/NodeDetails.java (100%)
delete mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/ContainerStateManagerV2.java
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/{helpers/OmDeleteKeys.java
=> protocol/S3Auth.java} (56%)
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/{OmTransport.java
=> OzoneManagerClientProtocol.java} (56%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/security/S3SecurityUtil.java
delete mode 100644
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/genesis/BenchMarkOMKeyAllocation.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
