Chu Cheng Li created HDDS-16727:
-----------------------------------
Summary: Datanode parses a block's whole metadata for every
streaming ReadBlock request
Key: HDDS-16727
URL: https://issues.apache.org/jira/browse/HDDS-16727
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
*Background*
A streaming ReadBlock request reads a range of a block. The gRPC streaming read
({{ozone.client.stream.readblock.enable}}) uses it. The Ratis data stream read
of HDDS-9904 also uses it.
For each ReadBlock request, the datanode reads the {{BlockData}} of the block
from RocksDB. The {{BlockData}} contains the chunk list of the block and the
checksums of each chunk.
*Problem*
The datanode parses all of the {{BlockData}} for each request, also when the
request reads only a few KB.
* With the default sizes (256 MB block, 4 MB chunk, 16 KB for each checksum), a
full block has 64 chunks and 16,384 checksums.
* The serialized {{BlockData}} is about 100 KB. The parse allocates about 1 MB,
because each checksum becomes a separate {{ByteString}}.
*Effect*
In a TPC-H SF100 Parquet test from Spark on one datanode, the parse used 47% to
64% of the CPU of the ReadBlock handler.
* The parse took 0.46 ms for each gRPC request and 0.67 ms for each Ratis
request. The RocksDB get took about 0.04 ms.
* A test cache of the parsed {{BlockData}} of closed containers decreased the
median time of small reads from 1.4–2.2 ms to 0.4–0.8 ms. Warm queries became
7–8% faster on both read paths.
* A cache is not safe. Reconciliation can change the blocks of a closed
container and keep the same BCSID. Block deletion and replica replacement also
change these blocks.
*Changes*
* For ReadBlock, the datanode reads the serialized {{BlockData}} and walks its
chunk list.
* It parses only the chunks in the requested range. For the other chunks, it
reads only the offset and the length.
* Nothing is cached. Thus, no data can become stale, and reads from open
containers also become faster.
* A block with an incremental chunk list (hsync), or with chunks of different
lengths, keeps the full parse. Thus, the checksum lookup does not change for
these blocks.
*Results*
A microbenchmark uses one thread and a schema V3 container with one full 256 MB
block. The old code and the new code ran in turns in one JVM on a busy laptop.
The values are medians of 15 batches.
* Block metadata for each request: 1,072 µs to 91 µs. Allocation: 1,023 KB to
131 KB.
* ReadBlock of 16 KiB: 1,493 µs to 688 µs.
* ReadBlock of 128 KiB: 1,839 µs to 624 µs.
* ReadBlock of 1 MiB: 2,159 µs to 1,308 µs.
*Tests*
* Unit tests examine the chunk walk and the cases that keep the full parse.
* A {{KeyValueHandler}} test examines the checksums of a ReadBlock in the
requested range.
Found in the TPC-H benchmark of HDDS-9904
(https://github.com/apache/ozone/pull/10743).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
