[ 
https://issues.apache.org/jira/browse/HDDS-17871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Soldatov updated HDDS-17871:
-----------------------------------
    Description: 
Netty's default write buffer watermarks for a connection are 32 KiB (low) and 
64 KiB (high). One positioned-read reply is bigger than the high watermark: a 
64 KiB range plus the response's framing.

Netty's HTTP/2 flow controller hands DATA to the connection only while the 
connection is writable 
({{DefaultHttp2RemoteFlowController.writePendingBytes}}). So when the datanode 
flushes a reply:
# the flow controller writes about 64 KiB, the connection turns unwritable, and 
the flow controller stops;
# the connection writes those bytes to the socket; the buffer drops below 32 
KiB and the connection turns writable again (the event fires synchronously, 
inside the socket write);
# {{Http2ConnectionHandler.channelWritabilityChanged}} sees pending data and 
flushes again, and the rest of the reply goes out in a separate socket write.

Every positioned-read reply therefore costs two writability events through the 
pipeline, a second flow-controller pass, and an extra socket write. With N 
concurrent replies in one flush, the write is split into about N+1 socket 
writes and 2N writability events, instead of one write.

gRPC's server creates its flow controller without a listener 
({{NettyServerHandler}}), so a writability change does not walk the 
connection's streams; the cost is the split write above.

h3. Measured
Netty's HTTP/2 handler set up like gRPC's server 
({{UniformStreamByteDistributor}}, 1 MiB window), with an {{EmbeddedChannel}} 
standing in for a socket that accepts all bytes.

||Watermark||Replies||Events while sending||
|32K / 64K (default)|1|unwritable, write 65381 B, writable, write 1233 B|
|32K / 64K (default)|4|4 × (unwritable, write ~64 KiB, writable), then write 
4932 B|
|512K / 1M|1|one write of 66614 B|
|512K / 1M|4|one write of 266456 B|


  was:
Netty's default write buffer watermarks for a connection are 32 KiB (low) and 
64 KiB (high). One positioned-read reply is bigger than the high watermark: a 
64 KiB range plus the response's framing.

Netty's HTTP/2 flow controller hands DATA to the connection only while the 
connection is writable. In each pass it writes at most 
`max(bytesBeforeUnwritable, lowWaterMark)` bytes 
(`DefaultHttp2RemoteFlowController.writePendingBytes` / 
`maxUsableChannelBytes`). So for every reply it:

1. writes about 64 KiB, the connection turns unwritable, and the rest of the 
reply stays queued in the flow controller;
2. sends the tail only after the socket drains below 32 KiB. At that point 
`Http2ConnectionHandler.channelWritabilityChanged` sees pending data and 
flushes again.

Every positioned-read reply therefore needs an extra event-loop pass after a 
socket drain before the client gets its last bytes. With several concurrent 
reads on one connection, each writable cycle releases only about 32–64 KiB, 
split across the streams.

We might set the child channels' WRITE_BUFFER_WATER_MARK to something bigger, 
so several concurrent replies fit below the low watermark.


> Datanode gRPC connection flips writability on every positioned-read reply
> -------------------------------------------------------------------------
>
>                 Key: HDDS-17871
>                 URL: https://issues.apache.org/jira/browse/HDDS-17871
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Datanode
>    Affects Versions: 2.3.0
>            Reporter: Sergey Soldatov
>            Assignee: Sergey Soldatov
>            Priority: Minor
>
> Netty's default write buffer watermarks for a connection are 32 KiB (low) and 
> 64 KiB (high). One positioned-read reply is bigger than the high watermark: a 
> 64 KiB range plus the response's framing.
> Netty's HTTP/2 flow controller hands DATA to the connection only while the 
> connection is writable 
> ({{DefaultHttp2RemoteFlowController.writePendingBytes}}). So when the 
> datanode flushes a reply:
> # the flow controller writes about 64 KiB, the connection turns unwritable, 
> and the flow controller stops;
> # the connection writes those bytes to the socket; the buffer drops below 32 
> KiB and the connection turns writable again (the event fires synchronously, 
> inside the socket write);
> # {{Http2ConnectionHandler.channelWritabilityChanged}} sees pending data and 
> flushes again, and the rest of the reply goes out in a separate socket write.
> Every positioned-read reply therefore costs two writability events through 
> the pipeline, a second flow-controller pass, and an extra socket write. With 
> N concurrent replies in one flush, the write is split into about N+1 socket 
> writes and 2N writability events, instead of one write.
> gRPC's server creates its flow controller without a listener 
> ({{NettyServerHandler}}), so a writability change does not walk the 
> connection's streams; the cost is the split write above.
> h3. Measured
> Netty's HTTP/2 handler set up like gRPC's server 
> ({{UniformStreamByteDistributor}}, 1 MiB window), with an {{EmbeddedChannel}} 
> standing in for a socket that accepts all bytes.
> ||Watermark||Replies||Events while sending||
> |32K / 64K (default)|1|unwritable, write 65381 B, writable, write 1233 B|
> |32K / 64K (default)|4|4 × (unwritable, write ~64 KiB, writable), then write 
> 4932 B|
> |512K / 1M|1|one write of 66614 B|
> |512K / 1M|4|one write of 266456 B|



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to