This is an automated email from the ASF dual-hosted git repository.
szetszwo pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 517e3e15c5 HDDS-7371. Create properties for all dependency versions
(#4059)
new fdf219f179 HDDS-5366. [Ozone-Streaming] Implement stream method to
ContainerStateMachine. (#2358). Contributed by mingchao zhao
new 0afbbc541d HDDS-5452. Add link method to ContainerStateMachine for
Ratis streaming (#2422)
new 9eafca6ee8 HDDS-5481. Fix stream() and link() method in
ContainerStateMachine. (#2451)
new da95602eba HDDS-5480. [Ozone-Streaming] Client and server should
support stream setup. (#2452)
new 82153a26ed HDDS-5488. [Ozone-Streaming] Add a new
BlockOutputStream/KeyOutputStream to support streaming api (#2495)
new 5a294b7d54 HDDS-5599. [Ozone-Streaming]drop BufferPool and
ChunkBuffer to avoid buffer copying (#2557)
new ff59eeb27f HDDS-5705. [Ozone-Streaming] Change ByteBufStreamOutput to
ByteBufferStreamOutput (#2603)
new f27388a5cc HDDS-5742. Avoid unnecessary Bytebuffer conversions (#2673)
new 94f87891a7 HDDS-5486. [Ozone-Streaming] Streaming supports writing in
Pipline mode (#2682)
new 9d01c02c98 HDDS-5849. [Ozone-Streaming]Write exceptions occur after
checksum is enabled (#2729)
new 122f96c029 HDDS-5674.[Ozone-Streaming] Handle client retries on
exception (#2701)
new e063706ed3 HDDS-5895. [Ozone-Streaming] Make
raft.server.data-stream.client.pool.size configurable (#2766)
new 75efaae8cf HDDS-5763. Provide an Executor for each LocalStream in
ContainerStateMachine (#2782)
new 2e86e93321 HDDS-5987. [Ozone-Streaming] Add XceiverClientRatis stream
config (#2841)
new 9a7edb0240 HDDS-5961. [Ozone-Streaming] update the usage space of
Containers in the stream write (#2833)
new 237e0afed3 HDDS-5879. [Ozone-Streaming] OzoneBucket add the
createMultipartStreamKey method (#2760)
new 95e317b551 HDDS-5743. [Ozone-Streaming] Add option to write files via
streaming api in ofs and o3fs. (#2770)
new 9a2de9a530 HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer
fixed boundary for streaming writes. (#2866)
new ac9921d65d HDDS-6039. Define a minimum packet size during streaming
writes. (#2883)
new a67569fe83 HDDS-6130. [Ozone-Streaming] When releaseBuffers will get
“Couldn 't find the required future” (#2939)
new f85b42dcee HDDS-6139. [Ozone-Streaming] Fix incorrect computation of
totalAckDataLength. (#2978)
new 0e8c8004a1 HDDS-6178. [Ozone-Streaming] Fix NPE in HDDS-6139. (#2984)
new 2657f5ad95 HDDS-6138.[Ozone-Streaming] Define a limit on the size of
the retry bufferList. (#2946)
new 14ffbd9cb1 HDDS-6298. Add XceiverServerRatis stream config (#3070)
new 872e3061fe HDDS-5487. [Ozone-Streaming] BlockDataStreamOutput support
FlushDelay. (#3002)
new 92054a4d2f HDDS-6282. Fix BlockDataStreamOutput#doFlushIfNeeded NPE
(#3060)
new 8a5f2d6192 HDDS-6229. [Ozone-Streaming] Data Channel abstraction on
datanode (#3023)
new 8f68101eb8 HDDS-6355. [Ozone-Streaming] Fix CheckStyle problem (#3119)
new bb3af31757 HDDS-6388. [Ozone-Streaming] Streaming write support both
pipeline model and star model (#3145)
new dfefdf20eb HDDS-5798. [Ozone-Streaming] Setup TlsConf parameters.
(#3207)
new 95dbea64a0 HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput.
(#3195)
new 6a6384233e HDDS-6500. [Ozone-Streaming] Buffer the PutBlockRequest at
the end of the stream. (#3229)
new 3d821e31ff HDDS-5666. Add option to createKey via streaming api in
Freon (#2574)
new 7f22916889 HDDS-6592. [Ozone-Streaming] Fix
ContainerStateMachine#applyTransaction assert error (#3315)
new 45954da28b HDDS-6842. [Ozone-Streaming] Reduce the number of watch
requests in StreamCommitWatcher. (#3492)
new 7184025ffb HDDS-6867. [Ozone-Streaming] PutKeyHandler should not use
streaming to put EC key. (#3516)
new 07bf1dbcf0 HDDS-6955. [Ozone-streaming] Add explicit stream flag in
ozone shell (#3559)
new 13da2a7c4a HDDS-7431. [Ozone-Streaming] Disable data steam by default.
(#3900)
new 8a0e347105 HDDS-7438. [Ozone-Streaming] Add a createStreamKey method
to OzoneBucket. (#3914)
new 9bbf5a1ada HDDS-7425. Add documentation for the new Streaming Pipeline
feature. (#3913)
new 1e155f3820 HDDS-7478. [Ozone-Streaming] NPE in when creating a file
with o3fs. (#3949)
new 0650ad9b08 HDDS-7426. Add a new acceptance test for Streaming
Pipeline. (#4019)
The 42 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/hadoop/hdds/scm/OzoneClientConfig.java | 63 ++
.../apache/hadoop/hdds/scm/XceiverClientRatis.java | 7 +-
.../hdds/scm/storage/AbstractDataStreamOutput.java | 131 ++++
...utputStream.java => BlockDataStreamOutput.java} | 695 ++++++++++-----------
.../hdds/scm/storage/ByteBufferStreamOutput.java | 57 ++
.../hadoop/hdds/scm/storage/StreamBuffer.java} | 47 +-
...CommitWatcher.java => StreamCommitWatcher.java} | 211 +++----
.../hadoop/hdds/client/ReplicationConfig.java | 11 +
.../hadoop/hdds/protocol/DatanodeDetails.java | 9 +-
.../org/apache/hadoop/hdds/ratis/RatisHelper.java | 92 ++-
.../hdds/scm/storage/ContainerProtocolCalls.java | 14 +-
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 26 +
.../org/apache/hadoop/ozone/audit/DNAction.java | 3 +-
.../org/apache/hadoop/ozone/common/Checksum.java | 5 +
.../helpers/ContainerCommandRequestPBHelper.java | 1 +
.../common/src/main/resources/ozone-default.xml | 29 +
.../ozone/container/ContainerTestHelper.java | 12 +
.../org/apache/hadoop/hdds/conf/ConfigTag.java | 3 +-
.../container/common/impl/HddsDispatcher.java | 21 +-
.../common/interfaces/ContainerDispatcher.java | 10 +
.../ozone/container/common/interfaces/Handler.java | 5 +
.../server/ratis/ContainerStateMachine.java | 79 ++-
.../common/transport/server/ratis/LocalStream.java | 58 ++
.../transport/server/ratis/XceiverServerRatis.java | 43 +-
.../ozone/container/keyvalue/KeyValueHandler.java | 52 ++
.../keyvalue/impl/ChunkManagerDispatcher.java | 16 +
.../keyvalue/impl/FilePerBlockStrategy.java | 20 +
.../keyvalue/impl/KeyValueStreamDataChannel.java | 277 ++++++++
.../keyvalue/impl/StreamDataChannelBase.java | 96 +++
.../keyvalue/interfaces/ChunkManager.java | 13 +
.../container/common/TestDatanodeStateMachine.java | 8 +-
.../impl/TestKeyValueStreamDataChannel.java | 313 ++++++++++
.../upgrade/TestDatanodeUpgradeToSchemaV3.java | 4 +
.../content/feature/Streaming-Write-Pipeline.md | 136 ++++
.../hdds/conf/DatanodeRatisServerConfig.java | 35 ++
.../src/main/proto/DatanodeClientProtocol.proto | 5 +-
.../ozone/container/common/TestEndPoint.java | 8 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 58 ++
.../client/io/BlockDataStreamOutputEntry.java | 287 +++++++++
...ol.java => BlockDataStreamOutputEntryPool.java} | 218 ++-----
...yOutputStream.java => KeyDataStreamOutput.java} | 334 +++-------
...utputStream.java => OzoneDataStreamOutput.java} | 49 +-
.../ozone/client/protocol/ClientProtocol.java | 38 ++
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 145 +++++
.../intellij/runConfigurations/Datanode2.xml | 2 +-
.../intellij/runConfigurations/Datanode3.xml | 2 +-
.../dist/src/main/compose/ozone/docker-config | 2 +
.../src/main/compose/ozonesecure/docker-config | 2 +
.../dist/src/main/smoketest/ozonefs/ozonefs.robot | 8 +
.../fs/ozone/TestOzoneFileSystemWithStreaming.java | 158 +++++
.../org/apache/hadoop/ozone/MiniOzoneCluster.java | 20 +
.../apache/hadoop/ozone/MiniOzoneClusterImpl.java | 21 +
.../apache/hadoop/ozone/TestMiniOzoneCluster.java | 4 +
.../client/rpc/TestBlockDataStreamOutput.java | 273 ++++++++
.../rpc/TestContainerStateMachineStream.java | 219 +++++++
.../client/rpc/TestOzoneRpcClientAbstract.java | 1 +
.../client/rpc/TestOzoneRpcClientWithRatis.java | 129 ++++
.../apache/hadoop/ozone/container/TestHelper.java | 42 +-
.../server/TestSecureContainerServer.java | 4 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 24 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 8 +
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 32 +
.../fs/ozone/BasicRootedOzoneFileSystem.java | 8 +
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 3 +
.../hadoop/fs/ozone/OzoneFSDataStreamOutput.java | 103 +++
.../hadoop/ozone/client/ClientProtocolStub.java | 24 +
.../hadoop/ozone/freon/ContentGenerator.java | 18 +
.../ozone/freon/OzoneClientKeyGenerator.java | 29 +-
.../hadoop/ozone/shell/keys/PutKeyHandler.java | 63 +-
69 files changed, 3943 insertions(+), 1000 deletions(-)
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/AbstractDataStreamOutput.java
copy
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/{BlockOutputStream.java
=> BlockDataStreamOutput.java} (52%)
create mode 100644
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ByteBufferStreamOutput.java
copy
hadoop-hdds/client/src/{test/java/org/apache/hadoop/hdds/scm/storage/TestBufferPool.java
=> main/java/org/apache/hadoop/hdds/scm/storage/StreamBuffer.java} (54%)
copy
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/{CommitWatcher.java
=> StreamCommitWatcher.java} (51%)
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/LocalStream.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java
create mode 100644
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/StreamDataChannelBase.java
create mode 100644
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/impl/TestKeyValueStreamDataChannel.java
create mode 100644 hadoop-hdds/docs/content/feature/Streaming-Write-Pipeline.md
create mode 100644
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntry.java
copy
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/{BlockOutputStreamEntryPool.java
=> BlockDataStreamOutputEntryPool.java} (59%)
copy
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/{KeyOutputStream.java
=> KeyDataStreamOutput.java} (60%)
copy
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/{OzoneOutputStream.java
=> OzoneDataStreamOutput.java} (52%)
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemWithStreaming.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestBlockDataStreamOutput.java
create mode 100644
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestContainerStateMachineStream.java
create mode 100644
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneFSDataStreamOutput.java
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
