[
https://issues.apache.org/jira/browse/HDDS-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HDDS-17871:
-----------------------------------
Description:
Netty's default write buffer watermarks for a connection are 32 KiB (low) and
64 KiB (high). One positioned-read reply is bigger than the high watermark: a
64 KiB range plus the response's framing.
Netty's HTTP/2 flow controller hands DATA to the connection only while the
connection is writable. In each pass it writes at most
`max(bytesBeforeUnwritable, lowWaterMark)` bytes
(`DefaultHttp2RemoteFlowController.writePendingBytes` /
`maxUsableChannelBytes`). So for every reply it:
1. writes about 64 KiB, the connection turns unwritable, and the rest of the
reply stays queued in the flow controller;
2. sends the tail only after the socket drains below 32 KiB. At that point
`Http2ConnectionHandler.channelWritabilityChanged` sees pending data and
flushes again.
Every positioned-read reply therefore needs an extra event-loop pass after a
socket drain before the client gets its last bytes. With several concurrent
reads on one connection, each writable cycle releases only about 32–64 KiB,
split across the streams.
We might set the child channels' WRITE_BUFFER_WATER_MARK to something bigger,
so several concurrent replies fit below the low watermark.
was:
Netty's default write buffer limit for a connection is 64 KiB. One
positioned-read reply is bigger than that: 64 KiB of data plus headers. So
every reply the datanode sends marks the client connection as "not writable",
and then "writable" again once the reply is sent.
Each of these switches makes the datanode go through every stream open on that
connection. With many concurrent reads on one connection, the datanode wastes
event-loop time on this for every reply.
We might set the child channels' WRITE_BUFFER_WATER_MARK to something bigger,
so several concurrent replies fit below the low watermark.
> Datanode gRPC connection holds back the tail of every positioned-read reply
> ---------------------------------------------------------------------------
>
> Key: HDDS-17871
> URL: https://issues.apache.org/jira/browse/HDDS-17871
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Affects Versions: 2.3.0
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Minor
>
> Netty's default write buffer watermarks for a connection are 32 KiB (low) and
> 64 KiB (high). One positioned-read reply is bigger than the high watermark: a
> 64 KiB range plus the response's framing.
> Netty's HTTP/2 flow controller hands DATA to the connection only while the
> connection is writable. In each pass it writes at most
> `max(bytesBeforeUnwritable, lowWaterMark)` bytes
> (`DefaultHttp2RemoteFlowController.writePendingBytes` /
> `maxUsableChannelBytes`). So for every reply it:
> 1. writes about 64 KiB, the connection turns unwritable, and the rest of the
> reply stays queued in the flow controller;
> 2. sends the tail only after the socket drains below 32 KiB. At that point
> `Http2ConnectionHandler.channelWritabilityChanged` sees pending data and
> flushes again.
> Every positioned-read reply therefore needs an extra event-loop pass after a
> socket drain before the client gets its last bytes. With several concurrent
> reads on one connection, each writable cycle releases only about 32–64 KiB,
> split across the streams.
> We might set the child channels' WRITE_BUFFER_WATER_MARK to something bigger,
> so several concurrent replies fit below the low watermark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]