[
https://issues.apache.org/jira/browse/HDDS-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-15356:
------------------------------
Status: Patch Available (was: Open)
> Make multi-buffer chunk checksum allocation-free
> ------------------------------------------------
>
> Key: HDDS-15356
> URL: https://issues.apache.org/jira/browse/HDDS-15356
> Project: Apache Ozone
> Issue Type: Improvement
> Components: common
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> A **multi-buffer** `ChunkBuffer` is one whose data is stored as more than
> one underlying `ByteBuffer`
> instead of a single contiguous block. In production this shape occurs when
> the data was assembled from
> a list of Netty `ByteBuf`s (Ratis state-machine-data read via
> `ChunkedNioFile`), when the verifier
> received a `List<ByteString>` spanning more than one checksum window
> (client read-verify), or when an
> operator opted into `IncrementalChunkBuffer` via
> `ozone.client.bytebuffer.increment > 0`.
> `Checksum.computeChecksum(ChunkBuffer)` previously delegated to
> `ChunkBuffer.iterate(bytesPerChecksum)`, which allocates a fresh
> `byte[bytesPerChecksum]` (1 MB by
> default) and memcpys into it whenever a checksum window straddles two of
> those underlying ByteBuffers.
> This change rewrites both the no-cache and cache compute paths to walk
> `data.asByteBufferList()`
> directly, slicing each window via a non-copying `ByteBuffer.duplicate()`
> helper (`BufferUtils.slice`)
> and feeding slices to a new `Checksum.StreamingChecksum` strategy that
> wraps `ChecksumByteBuffer`
> (CRC32, CRC32C) or `MessageDigest` (SHA-256, MD5). The `update(ByteBuffer)`
> contracts of both define
> incremental update as byte-equivalent to a single update over the
> concatenation, so output bytes are
> bit-identical.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
