[
https://issues.apache.org/jira/browse/HBASE-30390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-30390:
---------------------------
Description:
When HFile blocks are written with buffered data block encoding, data block
encoders retain the previous cell for delta encoding. When a block is shipped,
EncodingState.beforeShipped() materializes a complete copy of the previous
cell, including its value bytes. This is unnecessary for FAST_DIFF, which needs
the previous key metadata and value only for comparison, while the encoded
value bytes are already present in the writer buffer.
The complete copy adds value-array allocations and byte copies to the HFile
write path, increasing allocation and garbage-collection overhead. This can
occur when FAST_DIFF or DIFF encoding is used and writer-backed cells are
shipped or reused.
The fix is to keep a key-only materialization for the previous cell, track the
previous value length separately, and make FAST_DIFF compare the previous value
from the encoded writer buffer after shipping. The writer buffer range is
passed to the encoding state, and regression coverage is added for shipped
cells, reused buffers, and failed encoding paths.
Reproduction and validation:
* Use buffered data block encoding with FAST_DIFF and ship or reuse
writer-backed cells.
* Verify that previous-value comparison after shipping reads from the writer
buffer without copying the complete cell value.
* Run TestFastDiffEncodingState, which covers shipped cells, reused buffers,
and failed encoding paths.
was:
## What happens
Data block encoders retain previous cells while writing HFile blocks. When a
block is shipped, the current implementation materializes a complete copy of
the previous cell, including its value bytes, even though FAST_DIFF only needs
the previous value for comparison.
## When it happens
When HFile blocks are written with buffered data block encoding, especially
FAST_DIFF or DIFF, and writer-backed cells are shipped or reused.
## Impact
This creates unnecessary value-array allocations and byte copies on the write
path, increasing allocation and garbage-collection overhead for encoded HFile
writes.
## Root cause
On master, EncodingState.beforeShipped() copies the previous cell with
KeyValueUtil.copyToNewKeyValue(). FastDiffDeltaEncoder then reads the copied
cell for previous-value length and content comparison. The previous key
metadata must remain stable, but copying the complete value is unnecessary
because the encoded value bytes are already present in the writer buffer.
Upstream code pointers:
-
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodingState.java:43-49
-
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:217-269
-
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:822-827
## Proposed fix
Keep a key-only materialization for general encoder state, track the previous
value length separately, and make FAST_DIFF compare against the value bytes
already present in the encoded writer buffer after shipping. Pass the writer
buffer range to the encoding state and add focused regression coverage for
shipped cells, reused buffers, and encoding failures.
## Reproduction
A focused regression test will be included with the proposed change.
> Avoid copying previous cell values during data block encoding
> -------------------------------------------------------------
>
> Key: HBASE-30390
> URL: https://issues.apache.org/jira/browse/HBASE-30390
> Project: HBase
> Issue Type: Improvement
> Components: HFile
> Affects Versions: 2.4.11
> Reporter: huginn
> Priority: Major
> Labels: pull-request-available
>
> When HFile blocks are written with buffered data block encoding, data block
> encoders retain the previous cell for delta encoding. When a block is
> shipped, EncodingState.beforeShipped() materializes a complete copy of the
> previous cell, including its value bytes. This is unnecessary for FAST_DIFF,
> which needs the previous key metadata and value only for comparison, while
> the encoded value bytes are already present in the writer buffer.
> The complete copy adds value-array allocations and byte copies to the HFile
> write path, increasing allocation and garbage-collection overhead. This can
> occur when FAST_DIFF or DIFF encoding is used and writer-backed cells are
> shipped or reused.
> The fix is to keep a key-only materialization for the previous cell, track
> the previous value length separately, and make FAST_DIFF compare the previous
> value from the encoded writer buffer after shipping. The writer buffer range
> is passed to the encoding state, and regression coverage is added for shipped
> cells, reused buffers, and failed encoding paths.
> Reproduction and validation:
> * Use buffered data block encoding with FAST_DIFF and ship or reuse
> writer-backed cells.
> * Verify that previous-value comparison after shipping reads from the writer
> buffer without copying the complete cell value.
> * Run TestFastDiffEncodingState, which covers shipped cells, reused buffers,
> and failed encoding paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
