Chu Cheng Li created HDDS-17801:
-----------------------------------
Summary: Limit the stream ReadBlock buffer on the datanode to the
requested range
Key: HDDS-17801
URL: https://issues.apache.org/jira/browse/HDDS-17801
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
h3. Problem
* For each stream {{ReadBlock}} request, {{KeyValueHandler.readBlockImpl}}
allocates a heap buffer of the full response size.
* The client asks for 1 MiB responses by default
({{ozone.client.stream.read.response-data-size}}). Thus, a 4 KiB positioned
read allocates 1 MiB on the datanode.
* At 2,000 positioned reads per second, the datanode allocates and clears about
2 GB of heap per second. This makes other reads on the same datanode slower.
h3. Changes
* {{BlockReadCursor.responseDataSize()}} is now at most the length of the range
that the request reads. {{readBlockImpl}} allocates its buffer with this size.
* The range starts and ends on checksum boundaries. Thus, one checksum interval
always fits in the buffer.
* A request that is larger than the response size gets the same buffer and the
same responses as before.
h3. Tests
* A small request gets a buffer of the size of its range. This includes a range
in a short last chunk and a range in a chunk without checksums.
* A zero-length request gets an empty buffer.
* A request with more than one response gets a buffer of the response size and
returns the correct data.
h3. Benchmark
freon {{ocpr}} on one host, each process pinned to its own CPUs, 256 MiB key,
stream read path. One sequential reader runs together with positioned reads.
The table shows the speed of the sequential reader in MB/s.
||Positioned reads||Before||After||
|None|1,158|1,135|
|8 threads x 64 KiB, same stream|971|1,048|
|8 threads x 4 KiB, same stream|977|1,096|
|2 threads x 4 KiB, other process and connection|982|1,073|
* Without positioned reads, the difference is run-to-run noise.
* The positioned reads also become faster with a small buffer. When the client
asks for 16 KiB responses instead of 1 MiB, positioned reads from another
connection go from 2,045 to 2,320 per second (about 14%).
Found during the measurements for HDDS-17399. The buffer has the full response
size since HDDS-13975.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]