[
https://issues.apache.org/jira/browse/HDDS-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-16727:
--------------------------------
Target Version/s: 2.3.0
> Datanode parses a block's whole metadata for every streaming ReadBlock request
> ------------------------------------------------------------------------------
>
> Key: HDDS-16727
> URL: https://issues.apache.org/jira/browse/HDDS-16727
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
>
> *Background*
> A streaming ReadBlock request reads a range of a block. The gRPC streaming
> read ({{ozone.client.stream.readblock.enable}}) uses it. The Ratis data
> stream read of HDDS-9904 also uses it.
> For each ReadBlock request, the datanode reads the {{BlockData}} of the block
> from RocksDB. The {{BlockData}} contains the chunk list of the block and the
> checksums of each chunk.
> *Problem*
> The datanode parses all of the {{BlockData}} for each request, also when the
> request reads only a few KB.
> * With the default sizes (256 MB block, 4 MB chunk, 16 KB for each checksum),
> a full block has 64 chunks and 16,384 checksums.
> * The serialized {{BlockData}} is about 100 KB. The parse allocates about 1
> MB, because each checksum becomes a separate {{ByteString}}.
> *Effect*
> In a TPC-H SF100 Parquet test from Spark on one datanode, the parse used 47%
> to 64% of the CPU of the ReadBlock handler.
> * The parse took 0.46 ms for each gRPC request and 0.67 ms for each Ratis
> request. The RocksDB get took about 0.04 ms.
> * A test cache of the parsed {{BlockData}} of closed containers decreased the
> median time of small reads from 1.4–2.2 ms to 0.4–0.8 ms. Warm queries became
> 7–8% faster on both read paths.
> * A cache is not safe. Reconciliation can change the blocks of a closed
> container and keep the same BCSID. Block deletion and replica replacement
> also change these blocks.
> *Changes*
> * For ReadBlock, the datanode reads the serialized {{BlockData}} and walks
> its chunk list.
> * It parses only the chunks in the requested range. For the other chunks, it
> reads only the offset and the length.
> * Nothing is cached. Thus, no data can become stale, and reads from open
> containers also become faster.
> * A block with an incremental chunk list (hsync), or with chunks of different
> lengths, keeps the full parse. Thus, the checksum lookup does not change for
> these blocks.
> *Results*
> A microbenchmark uses one thread and a schema V3 container with one full 256
> MB block. The old code and the new code ran in turns in one JVM on a busy
> laptop. The values are medians of 15 batches.
> * Block metadata for each request: 1,072 µs to 91 µs. Allocation: 1,023 KB to
> 131 KB.
> * ReadBlock of 16 KiB: 1,493 µs to 688 µs.
> * ReadBlock of 128 KiB: 1,839 µs to 624 µs.
> * ReadBlock of 1 MiB: 2,159 µs to 1,308 µs.
> *Tests*
> * Unit tests examine the chunk walk and the cases that keep the full parse.
> * A {{KeyValueHandler}} test examines the checksums of a ReadBlock in the
> requested range.
> Found in the TPC-H benchmark of HDDS-9904
> (https://github.com/apache/ozone/pull/10743).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
