[
https://issues.apache.org/jira/browse/HDDS-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-16728:
--------------------------------
Target Version/s: 2.3.0
> Limit the Ratis data stream client to 16 event loops by default
> ---------------------------------------------------------------
>
> Key: HDDS-16728
> URL: https://issues.apache.org/jira/browse/HDDS-16728
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
>
> *Background*
> A Ratis data stream client sends and receives data on netty event loops. All
> the Ratis data stream clients in a JVM share one group of event loops.
> {{raft.netty.dataStream.client.worker-group.size}} sets the size of this
> group. The Ratis default is two times the number of CPU cores (768 on a
> 384-core host). Ozone can set it with
> {{hdds.ratis.raft.netty.dataStream.client.worker-group.size}}, but Ozone does
> not set a default.
> The Ratis data stream read of HDDS-9904 reads closed containers through their
> read pipelines. SCM gives the read pipeline of each closed container a random
> ID. Thus, the client opens one Ratis client and one connection for each
> closed container that it reads.
> *Problem*
> Netty puts each connection on one event loop. The netty pooled allocator
> gives each event loop thread a separate arena of 4 MiB chunks. A reply is a
> slice of the buffer that received it. Thus, a chunk stays allocated until the
> client releases all the replies in that chunk. An arena cannot use the free
> space of a different arena.
> *Effect*
> In a TPC-H SF100 test with HBase on a 384-core host, each RegionServer opened
> about 150 data stream connections. The connections used different event
> loops, and each event loop used its own arena.
> * At idle, the pool kept 30.5 GiB of direct memory in 7,632 chunks. 98.5% of
> this memory was free.
> * The RegionServers stopped with {{OutOfMemoryError: Cannot reserve ... bytes
> of direct buffer memory}}.
> * A limit on the read-ahead did not prevent this.
> *Changes*
> * Set the default of
> {{hdds.ratis.raft.netty.dataStream.client.worker-group.size}} to 16.
> * In the same test, 16 event loops kept the peak direct memory of SF100 q1
> and q6 at 3 to 9 GiB. The query times were the same as with the gRPC read.
> *Open points*
> * Datanodes use the same setting for the clients that forward streamed writes
> to the other datanodes of a pipeline. Thus, a default of 16 can decrease the
> write throughput of a busy datanode. A write benchmark must examine this.
> Alternatively, Ozone can set the default only for clients, in
> {{RatisHelper.createRaftClientProperties}}.
> * If the client shares read connections for each datanode, it opens fewer
> connections. Then this limit becomes less important.
> *Tests*
> * A unit test examines the default.
> * A streaming write benchmark compares 16 event loops with the Ratis default.
> This sub-task contains the event loop limit of
> https://github.com/apache/ozone/pull/10743.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
