[ 
https://issues.apache.org/jira/browse/HDDS-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy reassigned HDDS-17865:
-------------------------------------

    Assignee: Andrey Yarovoy

> Datanode GetBlock decodes and re-encodes the whole stored BlockData, 
> including every chunk's checksum list
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17865
>                 URL: https://issues.apache.org/jira/browse/HDDS-17865
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Datanode
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> h3. Problem
> For every GetBlock, the datanode parses the stored block metadata from 
> RocksDB and then serializes the same data back into the response:
> {noformat}
> KeyValueHandler.handleGetBlock
> BlockManagerImpl.getBlock -> DatanodeStore.getBlockByID -> 
> blockDataTable.get(blockKey)
> BlockData codec (DelegatedCodec): ContainerProtos.BlockData.parseFrom(bytes) 
> -> BlockData.getFromProtoBuf
> BlockData.getProtoBufMessage() computeSize(chunks) check, new builder, 
> addAllChunks
> responseData.getSerializedSize() for the bytesGetBlock metric
> gRPC / domain socket serializes the response again
> {noformat}
> {\{getBlock}} itself needs only the block commit sequence ID from the decoded 
> object, for the BCSID check. The response bytes are effectively the bytes 
> that were stored.
> The decode is expensive because each \{{ChunkInfo}} carries a 
> \{{ChecksumData}} with one \{{repeated bytes checksums}} entry per 
> \{{ozone.client.bytes.per.checksum}} (default 16 KB, CRC32). A 4 MB chunk has 
> 256 checksums, and a full 256 MB block has about 16k \{{ByteString}} objects 
> to allocate on every GetBlock.
> h3. Evidence
> async-profiler CPU profiles of datanodes serving HBase YCSB workload E 
> (scan). HBase reopens store files when it switches a scanner to stream read, 
> and every reopen issues a fresh GetBlock per block.
> ||Profile||GetBlock, % of datanode CPU||Inside GetBlock||
> |Ratis RATIS/THREE + short-circuit read|28.7% (75% of 
> \{{HddsDispatcher.dispatch}} CPU)|BlockData parse 58% 
> (\{{ChecksumData$Builder.mergeFrom}}, of which \{{ProtobufArrayList.add}} -> 
> \{{Arrays.copyOf}} 49%), response \{{getSerializedSize}} 11%, RocksDB 
> \{{getDirect}} 14%, \{{getShortCircuitFd}} 5%|
> |EC rs-3-2|11.2%|proto decode/re-encode 9.3% of datanode CPU, RocksDB get 
> 2.6%|
> On the RegionServer, GetBlock is 11% (EC) and 19.6% (Ratis) of busy handler 
> time on the scan path. Part of that is the network round trip, but the 
> datanode-side cost is mostly this decode/re-encode. The datanode itself is 
> not saturated (dispatcher busy below 1% of thread time), so the gain is CPU 
> and datanode headroom, plus a shorter GetBlock round trip.
> h3. Proposed fix
> Serve the stored bytes without a full decode. The wire format stays exactly 
> the same.
> # Read the raw value of the block key (\{{RDBTable.get(byte[])}}), skipping 
> the BlockData codec.
> # Partially parse the raw bytes with \{{CodedInputStream}}. Read \{{blockID}} 
> (for the BCSID check) and \{{metadata}} (to detect 
> \{{INCREMENTAL_CHUNK_LIST}}), and skip every \{{chunks}} field without 
> decoding it.
> # If the block uses an incremental chunk list (hsync, last chunk kept in 
> \{{lastChunkInfoTable}}), fall back to the current path, which merges the 
> last chunk.
> # Otherwise put the raw bytes into \{{GetBlockResponseProto}} field 1 
> (\{{blockData}}) as an already-encoded length-delimited field, for example 
> through the builder's unknown-field set followed by \{{buildPartial()}}. The 
> serialized response is then byte-for-byte what it is today, and clients parse 
> it unchanged.
> # Use the raw length for the \{{bytesGetBlock}} and \{{localBytesGetBlock}} 
> metrics instead of \{{getSerializedSize()}}.
> Alternatives considered:
> * *Cache decoded BlockData on the datanode.* A full block's decoded checksum 
> list costs hundreds of KB of heap, so a useful cache is large. Caching the 
> raw bytes instead is cheap, but it pays off only together with step 4.
> * *\{{[lazy = true]}} on \{{ChunkInfo.checksumData}}.* This is a proto 
> annotation only, so it is wire compatible. Whether the shaded protobuf-java 
> runtime honors it for proto2 singular message fields needs checking before 
> relying on it.
> h3. Interactions
> * HDDS-16333 (open) asks GetBlock to validate \{{ChunkInfo}}. Validation on 
> the read path needs the decode this change removes. Validating once at 
> PutBlock time, or only on the fallback path, keeps both.
> * HDDS-17112 removed a similar decode from PutBlock with an incremental chunk 
> list. This is the read-side counterpart.
> * The client still parses the full response once per GetBlock. That cost is 
> out of scope here.
> h3. Compatibility
> No proto change, no RocksDB layout change, wire-identical responses. Blocks 
> with an incremental chunk list keep the current code path.
> h3. Testing
> * Unit test: for blocks with 1, 64 and 128 chunks (256 checksums each), the 
> new response bytes equal the bytes of the current \{{getProtoBufMessage()}} 
> response. Also cover a BCSID mismatch, a missing block, and the 
> incremental-chunk-list fallback.
> * Microbenchmark of \{{handleGetBlock}} with a 64-chunk block, before and 
> after. The profile predicts about 70% less GetBlock CPU.
> * Re-profile the datanode under YCSB E. On the Ratis cluster, GetBlock should 
> drop from about 29% to below 10% of datanode CPU.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to