[
https://issues.apache.org/jira/browse/HBASE-30392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-30392:
---------------------------
Description:
When an HFile block is cached on write, the cache buffer contains the block
header and on-disk data, but HFileBlock's onDiskDataSizeWithHeader metadata
also includes trailing checksum bytes that are not present in the cache buffer.
This occurs in HFileBlock.Writer#getBlockForCaching().
cloneOnDiskBufferWithHeader() copies only the header and on-disk data, while
the metadata is set to onDiskBlockBytesWithHeader.size() +
onDiskChecksum.length. The resulting size mismatch can make cache validation or
later block unpacking use an incorrect length, causing cache mismatches or
read/decompression failures.
The fix is to set onDiskDataSizeWithHeader to
onDiskBlockBytesWithHeader.size(), while leaving onDiskSizeWithoutHeader
unchanged because the physical HFile block size still includes checksum bytes.
was:
h3. What happens
When an HFile block is cached on write, the cache buffer contains only the
block header and on-disk data. The HFileBlock metadata field
onDiskDataSizeWithHeader is nevertheless populated with a value that also
includes trailing checksum bytes.
h3. When it happens
This occurs when cache-on-write creates a block through
HFileBlock.Writer#getBlockForCaching, including compressed cache and index or
inline blocks.
h3. Impact
The metadata and buffer length disagree. Subsequent cache block validation or
unpacking can use an incorrect data length, causing cache equality failures or
read or decompression errors.
h3. Root cause
HFileBlock.Writer#getBlockForCaching uses onDiskBlockBytesWithHeader.size() +
onDiskChecksum.length even though cloneOnDiskBufferWithHeader() copies only the
header and data, and the cache context has no checksum bytes. The invariant for
onDiskDataSizeWithHeader is the header plus data length, excluding checksums.
h3. Proposed fix
Set onDiskDataSizeWithHeader to onDiskBlockBytesWithHeader.size(), while
preserving onDiskSizeWithoutHeader, which continues to include checksum bytes
for the physical file size.
h3. Reproduction / Testing
Add test evidence here.
> Exclude checksum bytes from cache-on-write HFile block data size
> ----------------------------------------------------------------
>
> Key: HBASE-30392
> URL: https://issues.apache.org/jira/browse/HBASE-30392
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
> Affects Versions: 2.4.11
> Reporter: huginn
> Priority: Major
> Labels: pull-request-available
>
> When an HFile block is cached on write, the cache buffer contains the block
> header and on-disk data, but HFileBlock's onDiskDataSizeWithHeader metadata
> also includes trailing checksum bytes that are not present in the cache
> buffer.
> This occurs in HFileBlock.Writer#getBlockForCaching().
> cloneOnDiskBufferWithHeader() copies only the header and on-disk data, while
> the metadata is set to onDiskBlockBytesWithHeader.size() +
> onDiskChecksum.length. The resulting size mismatch can make cache validation
> or later block unpacking use an incorrect length, causing cache mismatches or
> read/decompression failures.
> The fix is to set onDiskDataSizeWithHeader to
> onDiskBlockBytesWithHeader.size(), while leaving onDiskSizeWithoutHeader
> unchanged because the physical HFile block size still includes checksum bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
