This is an automated email from the ASF dual-hosted git repository. roryqi pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-uniffle.git
The following commit(s) were added to refs/heads/master by this push:
new b7962b162 [#1383] [DOCS] Improve Netty's documentation (#1384)
b7962b162 is described below
commit b7962b162e4cfba82659f8d3d76d59953d3dee7e
Author: RickyMa <[email protected]>
AuthorDate: Tue Dec 19 20:52:06 2023 +0800
[#1383] [DOCS] Improve Netty's documentation (#1384)
### What changes were proposed in this pull request?
Improve Netty's documentation. (Add the parameter `rss.rpc.server.type` and
change all "Netty" to uppercase.)
### Why are the changes needed?
For [#1383](https://github.com/apache/incubator-uniffle/issues/1383)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
No need to be tested.
---
docs/client_guide/client_guide.md | 4 ++--
docs/server_guide.md | 8 ++++----
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/client_guide/client_guide.md
b/docs/client_guide/client_guide.md
index a5c167ef4..375d8ceea 100644
--- a/docs/client_guide/client_guide.md
+++ b/docs/client_guide/client_guide.md
@@ -96,10 +96,10 @@ spark.rss.data.replica.read 2
### Netty Setting
| Property Name | Default | Description
|
|-----------------------------------------------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| <client_type>.rss.client.type | GRPC | The
default is GRPC, we can set it to GRPC_NETTY to enable the netty on the client
|
+| <client_type>.rss.client.type | GRPC | The
default is GRPC, we can set it to GRPC_NETTY to enable the Netty on the client
|
| <client_type>.rss.client.netty.io.mode | NIO |
Netty EventLoopGroup backend, available options: NIO, EPOLL.
|
| <client_type>.rss.client.netty.client.connection.timeout.ms | 600000 |
Connection active timeout.
|
-| <client_type>.rss.client.netty.client.threads | 0 |
Number of threads used in the client thread pool. Default is 0, netty will use
the number of (available logical cores * 2) as the number of threads.
|
+| <client_type>.rss.client.netty.client.threads | 0 |
Number of threads used in the client thread pool. Default is 0, Netty will use
the number of (available logical cores * 2) as the number of threads.
|
| <client_type>.rss.client.netty.client.prefer.direct.bufs | true | If
true, we will prefer allocating off-heap byte buffers within Netty.
|
| <client_type>.rss.client.netty.client.connections.per.peer | 2 |
Suppose there are 100 executors,
spark.rss.client.netty.client.connections.per.peer = 2, then each ShuffleServer
will establish a total of (100 * 2) connections with multiple clients.
|
| <client_type>.rss.client.netty.client.receive.buffer | 0 |
Receive buffer size (SO_RCVBUF). Note: the optimal size for receive buffer and
send buffer should be latency * network_bandwidth. Assuming latency = 1ms,
network_bandwidth = 10Gbps, buffer size should be ~ 1.25MB. Default is 0, the
operating system automatically estimates the receive buffer size based on
default settings. |
diff --git a/docs/server_guide.md b/docs/server_guide.md
index 0e70dab72..c5b0d75dc 100644
--- a/docs/server_guide.md
+++ b/docs/server_guide.md
@@ -69,11 +69,11 @@ This document will introduce how to deploy Uniffle shuffle
servers.
| rss.coordinator.quorum | -
| Coordinator quorum
[...]
| rss.rpc.server.port | -
| RPC port for Shuffle
server, if set zero, grpc server start on random port.
[...]
| rss.jetty.http.port | -
| Http port for Shuffle
server
[...]
-| rss.server.netty.port | -1
| Netty port for Shuffle
server, if set zero, netty server start on random port.
[...]
-| rss.server.netty.epoll.enable | false
| If enable epoll model
with netty server.
[...]
+| rss.server.netty.port | -1
| Netty port for Shuffle
server, if set zero, Netty server start on random port.
[...]
+| rss.server.netty.epoll.enable | false
| If enable epoll model
with Netty server.
[...]
| rss.server.netty.accept.thread | 10
| Accept thread count in
netty.
[...]
| rss.server.netty.worker.thread | 100
| Worker thread count in
netty.
[...]
-| rss.server.netty.connect.backlog | 0
| For netty server,
requested maximum length of the queue of incoming connections.
[...]
+| rss.server.netty.connect.backlog | 0
| For Netty server,
requested maximum length of the queue of incoming connections.
[...]
| rss.server.netty.connect.timeout | 5000
| Timeout for connection in
netty.
[...]
| rss.server.netty.receive.buf | 0
| Receive buffer size
(SO_RCVBUF). Note: the optimal size for receive buffer and send buffer should
be latency * network_bandwidth. Assuming latency = 1ms, network_bandwidth =
10Gbps, buffer size should be ~ 1.25MB. Default is 0, the operating system
automatically estimates the receive buffer size based on default settings.
[...]
| rss.server.netty.send.buf | 0
| Send buffer size
(SO_SNDBUF).
[...]
@@ -132,7 +132,7 @@ For HADOOP FS, the conf value of
`rss.server.single.buffer.flush.threshold` shou
Finally, to improve the speed of writing to HDFS for a single partition, the
value of `rss.server.max.concurrency.of.per-partition.write` and
`rss.server.flush.hdfs.threadPool.size` could be increased to 50 or 100.
### Netty
-In version 0.0.8, we introduced Netty. Enabling netty on ShuffleServer can
significantly reduce GC time in high-throughput scenarios. We can enable netty
through the parameter `rss.server.netty.port`. Note: After enabling netty, the
ShuffleServer The node will be automatically tagged with `grpc_netty`, that is,
the node can only be assigned to clients of `spark.rss.client.type=GRPC_NETTY`.
+In version 0.8.0, we introduced Netty. Enabling Netty on ShuffleServer can
significantly reduce GC time in high-throughput scenarios. We can enable Netty
through the parameters `rss.server.netty.port` and `rss.rpc.server.type`. Note:
After setting the parameter `rss.rpc.server.type` to `GRPC_NETTY`,
ShuffleServer will be tagged with `GRPC_NETTY`, that is, the node can only be
assigned to clients with `spark.rss.client.type=GRPC_NETTY`.
When enabling Netty, we should also consider memory related configuration, the
following is an example.
