Andrey Yarovoy created HDDS-17865:
-------------------------------------
Summary: Datanode GetBlock decodes and re-encodes the whole stored
BlockData, including every chunk's checksum list
Key: HDDS-17865
URL: https://issues.apache.org/jira/browse/HDDS-17865
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Andrey Yarovoy
h3. Problem
For every GetBlock, the datanode parses the stored block metadata from RocksDB
and then serializes the same data back into the response:
{noformat}
KeyValueHandler.handleGetBlock
BlockManagerImpl.getBlock -> DatanodeStore.getBlockByID ->
blockDataTable.get(blockKey)
BlockData codec (DelegatedCodec): ContainerProtos.BlockData.parseFrom(bytes) ->
BlockData.getFromProtoBuf
BlockData.getProtoBufMessage() computeSize(chunks) check, new builder,
addAllChunks
responseData.getSerializedSize() for the bytesGetBlock metric
gRPC / domain socket serializes the response again
{noformat}
{\{getBlock}} itself needs only the block commit sequence ID from the decoded
object, for the BCSID check. The response bytes are effectively the bytes that
were stored.
The decode is expensive because each \{{ChunkInfo}} carries a \{{ChecksumData}}
with one \{{repeated bytes checksums}} entry per
\{{ozone.client.bytes.per.checksum}} (default 16 KB, CRC32). A 4 MB chunk has
256 checksums, and a full 256 MB block has about 16k \{{ByteString}} objects to
allocate on every GetBlock.
h3. Evidence
async-profiler CPU profiles of datanodes serving HBase YCSB workload E (scan).
HBase reopens store files when it switches a scanner to stream read, and every
reopen issues a fresh GetBlock per block.
||Profile||GetBlock, % of datanode CPU||Inside GetBlock||
|Ratis RATIS/THREE + short-circuit read|28.7% (75% of
\{{HddsDispatcher.dispatch}} CPU)|BlockData parse 58%
(\{{ChecksumData$Builder.mergeFrom}}, of which \{{ProtobufArrayList.add}} ->
\{{Arrays.copyOf}} 49%), response \{{getSerializedSize}} 11%, RocksDB
\{{getDirect}} 14%, \{{getShortCircuitFd}} 5%|
|EC rs-3-2|11.2%|proto decode/re-encode 9.3% of datanode CPU, RocksDB get 2.6%|
On the RegionServer, GetBlock is 11% (EC) and 19.6% (Ratis) of busy handler
time on the scan path. Part of that is the network round trip, but the
datanode-side cost is mostly this decode/re-encode. The datanode itself is not
saturated (dispatcher busy below 1% of thread time), so the gain is CPU and
datanode headroom, plus a shorter GetBlock round trip.
h3. Proposed fix
Serve the stored bytes without a full decode. The wire format stays exactly the
same.
# Read the raw value of the block key (\{{RDBTable.get(byte[])}}), skipping the
BlockData codec.
# Partially parse the raw bytes with \{{CodedInputStream}}. Read \{{blockID}}
(for the BCSID check) and \{{metadata}} (to detect
\{{INCREMENTAL_CHUNK_LIST}}), and skip every \{{chunks}} field without decoding
it.
# If the block uses an incremental chunk list (hsync, last chunk kept in
\{{lastChunkInfoTable}}), fall back to the current path, which merges the last
chunk.
# Otherwise put the raw bytes into \{{GetBlockResponseProto}} field 1
(\{{blockData}}) as an already-encoded length-delimited field, for example
through the builder's unknown-field set followed by \{{buildPartial()}}. The
serialized response is then byte-for-byte what it is today, and clients parse
it unchanged.
# Use the raw length for the \{{bytesGetBlock}} and \{{localBytesGetBlock}}
metrics instead of \{{getSerializedSize()}}.
Alternatives considered:
* *Cache decoded BlockData on the datanode.* A full block's decoded checksum
list costs hundreds of KB of heap, so a useful cache is large. Caching the raw
bytes instead is cheap, but it pays off only together with step 4.
* *\{{[lazy = true]}} on \{{ChunkInfo.checksumData}}.* This is a proto
annotation only, so it is wire compatible. Whether the shaded protobuf-java
runtime honors it for proto2 singular message fields needs checking before
relying on it.
h3. Interactions
* HDDS-16333 (open) asks GetBlock to validate \{{ChunkInfo}}. Validation on the
read path needs the decode this change removes. Validating once at PutBlock
time, or only on the fallback path, keeps both.
* HDDS-17112 removed a similar decode from PutBlock with an incremental chunk
list. This is the read-side counterpart.
* The client still parses the full response once per GetBlock. That cost is out
of scope here.
h3. Compatibility
No proto change, no RocksDB layout change, wire-identical responses. Blocks
with an incremental chunk list keep the current code path.
h3. Testing
* Unit test: for blocks with 1, 64 and 128 chunks (256 checksums each), the new
response bytes equal the bytes of the current \{{getProtoBufMessage()}}
response. Also cover a BCSID mismatch, a missing block, and the
incremental-chunk-list fallback.
* Microbenchmark of \{{handleGetBlock}} with a 64-chunk block, before and
after. The profile predicts about 70% less GetBlock CPU.
* Re-profile the datanode under YCSB E. On the Ratis cluster, GetBlock should
drop from about 29% to below 10% of datanode CPU.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]