This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch HDDS-4454
in repository https://gitbox.apache.org/repos/asf/ozone.git.
omit c117885 HDDS-5743.[Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
omit a8a25b8 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
omit 869f943 HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
omit 9c68709 HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
omit 1d9f870 HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
omit 9037764 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
omit 0936be3 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
omit ce6f566 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
omit 5156051 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
omit bdd2726 HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
omit d1cb3f9 HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
omit 51f88cb HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
omit 6c36e82 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
omit 51d1013 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
omit 0cc389a HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
omit e166e91 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
omit f508854 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
add 346b55d HDDS-5906. Fix DBScanner to support Datanode DB (#2781)
add 1b7072c HDDS-5983. Prefix Parser tool should only work for FSO
buckets. (#2839)
add 5ef59b4 HDDS-5957. Bump proto-backwards-compatibility from 1.0.5 to
1.0.7 (#2821)
add b4a785c HDDS-5891. OFS mkdir -p does not work as expected for bucket
creation when volume exists due to volume create ACL check (#2814)
add 6ea9916 HDDS-5964. [S3-TDE] Range Reads not working on encrypted
buckets (#2827)
add 83e5aa3 HDDS-5848. Introduce more replication metrics (#2758)
add 4348d1d HDDS-5980. Get on a key which is created/overwrite through
MPU on a TDE bucket, the data does not match with uploaded data. (#2844)
add 78e725c HDDS-5969. TestRootedOzoneFileSystem parameterized test is
not initialized properly (#2843)
add 281f3b0 HDDS-5870. Make Datanode usageinfo command output more
readable (#2816)
add 07b54e2 HDDS-5534. Verify config is updated on all OMs before
proceeding with Bootstrap (#2491)
add 722fd59 HDDS-5898 S3G in secure mode checks OM version. (#2822)
add 52e619c HDDS-5517. Support multiple container moves from a source
datanode in one balance iteration (#2808)
add 6372776 HDDS-5976. add raftlog IOException notifier for SCM HA (#2840)
add bc6029e [FSO] Allow existing/old buckets with any layout during OM
startup (#2571)
add ffe53e7 HDDS-6019. Fix s3g create key error in cluster with
replication one. (#2855)
add f780851 HDDS-6003. Update Ozone master branch version after 1.2.0
release (#2863)
add 4f56b1a HDDS-6043. Buckets created via link command do not mirror
layout of s… (#2867)
add 7afd64f HDDS-5897. Support configuration for including/excluding
datanodes for balancing (#2786)
add 97d900e HDDS-5989. Binary content printed for failed chunk write
(#2846)
add 1e76ffc HDDS-6011. Freon datanode chunk generator fails with NPE in
secure cluster (#2853)
add 5472ea4 HDDS-5497. Save log from init containers (#2547)
add 8b4d4a9 HDDS-5804. SCMContainerPlacementCapacity to choose lower
utilized node (#2784)
add 4e91226 HDDS-5917. S3 acceptance test failure due to too wide
assertion (#2789)
new 3fe52be HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new 58b7209 HDDS-5452. Add link method to ContainerStateMachine for Ratis
streaming (#2422)
new 8e8af7f HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new 91a4d82 HDDS-5480. [Ozone-Streaming] Client and server should support
stream setup. (#2452)
new d3226b6 HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new 8fd34de HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer
to avoid buffer copying (#2557)
new 7126b3b HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new 27079cc HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new 836a67c HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new d90fd93 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
new ed05656 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
new 573f718 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
new 64c9933 HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
new b3d995b HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
new 153ef3e HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
new f400954 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
new 320ae54 HDDS-5743.[Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (c117885)
\
N -- N -- N refs/heads/HDDS-4454 (320ae54)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 17 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
hadoop-hdds/client/pom.xml | 4 +-
.../apache/hadoop/hdds/scm/XceiverClientGrpc.java | 38 +--
hadoop-hdds/common/pom.xml | 4 +-
.../java/org/apache/hadoop/hdds/HddsUtils.java | 71 ++++++
.../apache/hadoop/hdds/scm/XceiverClientSpi.java | 6 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 6 +
.../common/src/main/resources/ozone-default.xml | 18 ++
hadoop-hdds/config/pom.xml | 4 +-
hadoop-hdds/container-service/pom.xml | 4 +-
.../container/common/helpers/ContainerUtils.java | 68 -----
.../container/common/impl/HddsDispatcher.java | 4 +-
.../metadata/DatanodeSchemaOneDBDefinition.java | 6 +-
.../container/replication/MeasuredReplicator.java | 34 ++-
.../replication/ReplicationSupervisor.java | 1 -
.../replication/ReplicationSupervisorMetrics.java | 1 -
.../common/helpers/TestContainerUtils.java | 2 +-
.../replication/TestMeasuredReplicator.java | 34 ++-
hadoop-hdds/docs/pom.xml | 4 +-
hadoop-hdds/framework/pom.xml | 4 +-
.../java/org/apache/hadoop/hdds/ExitManager.java | 12 +-
hadoop-hdds/hadoop-dependency-client/pom.xml | 4 +-
hadoop-hdds/hadoop-dependency-server/pom.xml | 4 +-
hadoop-hdds/hadoop-dependency-test/pom.xml | 4 +-
hadoop-hdds/interface-admin/pom.xml | 4 +-
hadoop-hdds/interface-client/pom.xml | 4 +-
hadoop-hdds/interface-server/pom.xml | 4 +-
hadoop-hdds/pom.xml | 4 +-
hadoop-hdds/server-scm/pom.xml | 4 +-
.../hdds/scm/container/ReplicationManager.java | 30 ++-
.../scm/container/balancer/ContainerBalancer.java | 209 +++++++--------
.../balancer/ContainerBalancerConfiguration.java | 75 +++++-
.../ContainerBalancerSelectionCriteria.java | 22 +-
.../scm/container/balancer/FindSourceGreedy.java | 158 ++++++++++++
.../scm/container/balancer/FindSourceStrategy.java | 67 +++++
.../scm/container/balancer/FindTargetGreedy.java | 121 ++++++++-
.../scm/container/balancer/FindTargetStrategy.java | 32 +--
.../algorithms/SCMContainerPlacementCapacity.java | 2 +-
.../replication/ReplicationManagerMetrics.java | 41 +++
.../apache/hadoop/hdds/scm/ha/SCMStateMachine.java | 7 +
.../hadoop/hdds/scm/container/MockNodeManager.java | 6 +-
.../hdds/scm/container/TestReplicationManager.java | 10 +
.../container/balancer/TestContainerBalancer.java | 74 ++++++
.../TestSCMContainerPlacementCapacity.java | 6 +-
.../placement/TestContainerPlacement.java | 4 +-
hadoop-hdds/test-utils/pom.xml | 4 +-
hadoop-hdds/tools/pom.xml | 4 +-
.../hdds/scm/cli/datanode/UsageInfoSubcommand.java | 41 ++-
hadoop-ozone/client/pom.xml | 4 +-
.../apache/hadoop/ozone/client/OzoneBucket.java | 10 +-
.../hadoop/ozone/client/io/OzoneInputStream.java | 33 ++-
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 44 +++-
.../hadoop/ozone/client/rpc/RpcClientTest.java | 217 ++++++++++++++++
hadoop-ozone/common/pom.xml | 4 +-
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 82 ++++++
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 11 +
.../hadoop/ozone/om/helpers/OMNodeDetails.java | 50 +++-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 8 +
.../hadoop/ozone/om/helpers/ServiceInfo.java | 41 ++-
...erServiceProtocol.java => OMAdminProtocol.java} | 9 +-
.../hadoop/ozone/om/protocol/OMConfiguration.java | 92 +++++++
.../protocolPB/OMAdminProtocolClientSideImpl.java | 136 ++++++++++
...rviceProtocolPB.java => OMAdminProtocolPB.java} | 9 +-
hadoop-ozone/csi/pom.xml | 4 +-
hadoop-ozone/datanode/pom.xml | 4 +-
hadoop-ozone/dist/pom.xml | 4 +-
.../src/main/compose/ozonesecure-mr/docker-config | 3 +
hadoop-ozone/dist/src/main/k8s/examples/testlib.sh | 5 +-
.../dist/src/main/smoketest/s3/objectdelete.robot | 6 +-
.../fault-injection-test/mini-chaos-tests/pom.xml | 4 +-
.../fault-injection-test/network-tests/pom.xml | 2 +-
hadoop-ozone/fault-injection-test/pom.xml | 4 +-
hadoop-ozone/insight/pom.xml | 4 +-
hadoop-ozone/integration-test/pom.xml | 4 +-
.../hadoop/fs/ozone/TestRootedOzoneFileSystem.java | 140 ++++++++--
.../fs/ozone/TestRootedOzoneFileSystemWithFSO.java | 5 +-
.../hadoop/ozone/MiniOzoneHAClusterImpl.java | 150 ++++++++---
.../hadoop/ozone/TestOzoneConfigurationFields.java | 5 +
.../rpc/TestContainerStateMachineFailures.java | 20 +-
.../client/rpc/TestOzoneAtRestEncryption.java | 11 +
.../ozone/om/TestOMStartupWithBucketLayout.java | 161 ++++++++++++
.../apache/hadoop/ozone/om/TestObjectStore.java | 128 ++++++++++
.../hadoop/ozone/om/TestOzoneManagerBootstrap.java | 184 +++++++++++---
hadoop-ozone/interface-client/pom.xml | 4 +-
...ServiceProtocol.proto => OMAdminProtocol.proto} | 43 ++--
.../src/main/proto/OmClientProtocol.proto | 1 +
hadoop-ozone/interface-storage/pom.xml | 4 +-
hadoop-ozone/ozone-manager/pom.xml | 4 +-
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 79 +++++-
.../apache/hadoop/ozone/om/OMPolicyProvider.java | 5 +-
.../apache/hadoop/ozone/om/OMStarterInterface.java | 2 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 283 ++++++++++++++-------
.../hadoop/ozone/om/OzoneManagerStarter.java | 45 +++-
.../apache/hadoop/ozone/om/ha/OMHANodeDetails.java | 6 +-
.../ozone/om/ratis/OzoneManagerRatisServer.java | 24 +-
.../om/ratis/utils/OzoneManagerRatisUtils.java | 64 ++++-
.../S3MultipartUploadCompleteRequest.java | 11 +
.../protocolPB/OMAdminProtocolServerSideImpl.java | 66 +++++
.../protocolPB/OzoneManagerRequestHandler.java | 2 +-
.../hadoop/ozone/protocolPB/RequestHandler.java | 4 +-
.../hadoop/ozone/om/TestBucketManagerImpl.java | 55 +++-
.../hadoop/ozone/om/TestOzoneManagerStarter.java | 4 +-
hadoop-ozone/ozonefs-common/pom.xml | 4 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 8 +-
hadoop-ozone/ozonefs-hadoop2/pom.xml | 4 +-
hadoop-ozone/ozonefs-hadoop3/pom.xml | 4 +-
hadoop-ozone/ozonefs-shaded/pom.xml | 4 +-
hadoop-ozone/ozonefs/pom.xml | 4 +-
hadoop-ozone/pom.xml | 4 +-
hadoop-ozone/recon-codegen/pom.xml | 2 +-
hadoop-ozone/recon/pom.xml | 2 +-
hadoop-ozone/s3gateway/pom.xml | 4 +-
.../hadoop/ozone/s3/OzoneClientProducer.java | 5 +
.../hadoop/ozone/s3/endpoint/ObjectEndpoint.java | 19 +-
.../hadoop/ozone/s3/io/S3WrapperInputStream.java | 84 ------
.../apache/hadoop/ozone/s3/io/package-info.java | 23 --
.../apache/hadoop/ozone/s3/util/S3StorageType.java | 22 +-
.../s3/endpoint/TestAbortMultipartUpload.java | 2 +
.../s3/endpoint/TestInitiateMultipartUpload.java | 2 +
.../hadoop/ozone/s3/endpoint/TestListParts.java | 2 +
.../s3/endpoint/TestMultipartUploadComplete.java | 2 +
.../s3/endpoint/TestMultipartUploadWithCopy.java | 2 +
.../hadoop/ozone/s3/endpoint/TestObjectDelete.java | 2 +
.../hadoop/ozone/s3/endpoint/TestObjectGet.java | 2 +
.../hadoop/ozone/s3/endpoint/TestObjectHead.java | 2 +
.../hadoop/ozone/s3/endpoint/TestObjectPut.java | 2 +
.../hadoop/ozone/s3/endpoint/TestPartUpload.java | 2 +
.../ozone/s3/endpoint/TestPermissionCheck.java | 4 +
hadoop-ozone/tools/pom.xml | 4 +-
.../hadoop/ozone/debug/DBDefinitionFactory.java | 17 +-
.../org/apache/hadoop/ozone/debug/DBScanner.java | 6 +
.../apache/hadoop/ozone/debug/PrefixParser.java | 14 +
.../hadoop/ozone/freon/DatanodeChunkGenerator.java | 7 +-
.../ozone/debug/TestDBDefinitionFactory.java | 7 +-
pom.xml | 8 +-
134 files changed, 3046 insertions(+), 808 deletions(-)
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceGreedy.java
create mode 100644
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/FindSourceStrategy.java
create mode 100644
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/RpcClientTest.java
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/{OMInterServiceProtocol.java
=> OMAdminProtocol.java} (80%)
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OMConfiguration.java
create mode 100644
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OMAdminProtocolClientSideImpl.java
copy
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/{OMInterServiceProtocolPB.java
=> OMAdminProtocolPB.java} (81%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMStartupWithBucketLayout.java
copy
hadoop-ozone/interface-client/src/main/proto/{OmInterServiceProtocol.proto =>
OMAdminProtocol.proto} (58%)
create mode 100644
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OMAdminProtocolServerSideImpl.java
delete mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/io/S3WrapperInputStream.java
delete mode 100644
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/io/package-info.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
