Gargi-jais11 commented on code in PR #235:
URL: https://github.com/apache/ozone-site/pull/235#discussion_r2682912642
##########
docs/05-administrator-guide/02-configuration/04-performance/04-streaming-write-pipeline.md:
##########
@@ -0,0 +1,130 @@
+---
+sidebar_label: Streaming Write Pipeline
+---
+
+# Streaming Write Pipeline
+
+This document discusses the Streaming Write Pipeline feature in Ozone. It is
implemented with the Ratis Streaming API. Note that the existing Ozone Write
Pipeline is implemented with the Ratis Async API. We refer the new Streaming
Write Pipeline as Write Pipeline V2 and the existing Async Write Pipeline as
Write Pipeline V1.
Review Comment:
It is good to highlight the **Note** part.
```suggestion
This document discusses the Streaming Write Pipeline feature in Ozone. It is
implemented with the Ratis Streaming API.
:::note Write Pipeline Versions
Note that the existing Ozone Write Pipeline is implemented with the Ratis
Async API. We refer to the new Streaming Write Pipeline as Write Pipeline V2
and the existing Async Write Pipeline as Write Pipeline V1.
:::
```
##########
docs/05-administrator-guide/02-configuration/04-performance/04-streaming-write-pipeline.md:
##########
@@ -0,0 +1,130 @@
+---
+sidebar_label: Streaming Write Pipeline
+---
+
+# Streaming Write Pipeline
+
+This document discusses the Streaming Write Pipeline feature in Ozone. It is
implemented with the Ratis Streaming API. Note that the existing Ozone Write
Pipeline is implemented with the Ratis Async API. We refer the new Streaming
Write Pipeline as Write Pipeline V2 and the existing Async Write Pipeline as
Write Pipeline V1.
+
+The Streaming Write Pipeline V2 increases the performance by providing better
network topology awareness and removing the performance bottlenecks in V1. The
V2 implementation also avoids unnecessary buffer copying (by Netty zero copy)
and has a better utilization of the CPUs and the disks in each Datanode.
+
+For detailed architectural information about write pipelines, see the [Write
Pipelines documentation](../../../core-concepts/replication/write-pipelines).
+
+## Configuration Properties
+
+Set the following properties to the Ozone configuration file `ozone-site.xml`.
+
+### Enable Streaming Write Pipeline
+
+To enable the Streaming Write Pipeline feature, set the following property to
true:
+
+```xml
+<property>
+ <name>hdds.container.ratis.datastream.enabled</name>
+ <value>false</value>
+ <description>Enable data stream of container</description>
+</property>
+```
+
+### Configure Datastream Port
+
+Datanodes listen to the following port for the streaming traffic:
+
+```xml
+<property>
+ <name>hdds.container.ratis.datastream.port</name>
+ <value>9855</value>
+ <description>The datastream port number of container</description>
+</property>
+```
+
+### Enable Filesystem Streaming
+
+To use Streaming in FileSystem API, set the following property to true:
Review Comment:
Its better to highlight `true` value here.
##########
docs/05-administrator-guide/02-configuration/04-performance/04-streaming-write-pipeline.md:
##########
@@ -0,0 +1,130 @@
+---
+sidebar_label: Streaming Write Pipeline
+---
+
+# Streaming Write Pipeline
+
+This document discusses the Streaming Write Pipeline feature in Ozone. It is
implemented with the Ratis Streaming API. Note that the existing Ozone Write
Pipeline is implemented with the Ratis Async API. We refer the new Streaming
Write Pipeline as Write Pipeline V2 and the existing Async Write Pipeline as
Write Pipeline V1.
+
+The Streaming Write Pipeline V2 increases the performance by providing better
network topology awareness and removing the performance bottlenecks in V1. The
V2 implementation also avoids unnecessary buffer copying (by Netty zero copy)
and has a better utilization of the CPUs and the disks in each Datanode.
+
+For detailed architectural information about write pipelines, see the [Write
Pipelines documentation](../../../core-concepts/replication/write-pipelines).
+
+## Configuration Properties
+
+Set the following properties to the Ozone configuration file `ozone-site.xml`.
+
+### Enable Streaming Write Pipeline
+
+To enable the Streaming Write Pipeline feature, set the following property to
true:
Review Comment:
Its better to highlight `true` value here.
##########
docs/05-administrator-guide/02-configuration/04-performance/04-streaming-write-pipeline.md:
##########
@@ -0,0 +1,130 @@
+---
+sidebar_label: Streaming Write Pipeline
+---
+
+# Streaming Write Pipeline
+
+This document discusses the Streaming Write Pipeline feature in Ozone. It is
implemented with the Ratis Streaming API. Note that the existing Ozone Write
Pipeline is implemented with the Ratis Async API. We refer the new Streaming
Write Pipeline as Write Pipeline V2 and the existing Async Write Pipeline as
Write Pipeline V1.
+
+The Streaming Write Pipeline V2 increases the performance by providing better
network topology awareness and removing the performance bottlenecks in V1. The
V2 implementation also avoids unnecessary buffer copying (by Netty zero copy)
and has a better utilization of the CPUs and the disks in each Datanode.
+
+For detailed architectural information about write pipelines, see the [Write
Pipelines documentation](../../../core-concepts/replication/write-pipelines).
+
+## Configuration Properties
+
+Set the following properties to the Ozone configuration file `ozone-site.xml`.
+
+### Enable Streaming Write Pipeline
+
+To enable the Streaming Write Pipeline feature, set the following property to
true:
+
+```xml
+<property>
+ <name>hdds.container.ratis.datastream.enabled</name>
+ <value>false</value>
+ <description>Enable data stream of container</description>
+</property>
+```
+
+### Configure Datastream Port
+
+Datanodes listen to the following port for the streaming traffic:
+
+```xml
+<property>
+ <name>hdds.container.ratis.datastream.port</name>
+ <value>9855</value>
+ <description>The datastream port number of container</description>
+</property>
+```
+
+### Enable Filesystem Streaming
+
+To use Streaming in FileSystem API, set the following property to true:
+
+```xml
+<property>
+ <name>ozone.fs.datastream.enabled</name>
+ <value>false</value>
+ <description>
+ Enable filesystem write via ratis streaming.
+ </description>
+</property>
+```
+
+## Client APIs
+
+### OzoneDataStreamOutput
+
+The new `OzoneDataStreamOutput` class is very similar to the existing
`OzoneOutputStream` class, except that `OzoneDataStreamOutput` uses
`ByteBuffer` as a parameter in the `write` methods while `OzoneOutputStream`
uses `byte[]`. The reason of using a `ByteBuffer`, instead of a `byte[]`, is to
support zero buffer copying. A typical `write` method is shown below:
+
+OzoneDataStreamOutput:
+
+```java
+public void write(ByteBuffer b, int off, int len) throws IOException;
+```
+
+OzoneOutputStream:
+
+```java
+public void write(byte[] b, int off, int len) throws IOException;
+```
+
+Using `ByteBuffer` enables zero-copy operations, reducing CPU overhead and
improving throughput.
+
+### OzoneBucket
+
+The following new methods are added to `OzoneBucket` for creating keys using
the Streaming Write Pipeline.
+
+#### createStreamKey
+
+```java
+public OzoneDataStreamOutput createStreamKey(String key, long size)
+ throws IOException;
+```
+
+```java
+public OzoneDataStreamOutput createStreamKey(String key, long size,
+ ReplicationConfig replicationConfig, Map<String, String> keyMetadata)
+ throws IOException;
+```
Review Comment:
I checked apache ozone codebase , the documentation should include the third
`createStreamKey` overload that accepts tags:
```
public OzoneDataStreamOutput createStreamKey(String key, long size,
ReplicationConfig replicationConfig, Map<String, String> keyMetadata,
Map<String, String> tags) throws IOException;
```
FYI, Tags are commonly used in S3-compatible storage systems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
