huginn created HBASE-30390:
------------------------------
Summary: Avoid copying previous cell values during data block
encoding
Key: HBASE-30390
URL: https://issues.apache.org/jira/browse/HBASE-30390
Project: HBase
Issue Type: Improvement
Components: HFile
Affects Versions: 2.4.11
Reporter: huginn
## What happens
Data block encoders retain previous cells while writing HFile blocks. When a
block is shipped, the current implementation materializes a complete copy of
the previous cell, including its value bytes, even though FAST_DIFF only needs
the previous value for comparison.
## When it happens
When HFile blocks are written with buffered data block encoding, especially
FAST_DIFF or DIFF, and writer-backed cells are shipped or reused.
## Impact
This creates unnecessary value-array allocations and byte copies on the write
path, increasing allocation and garbage-collection overhead for encoded HFile
writes.
## Root cause
On master, EncodingState.beforeShipped() copies the previous cell with
KeyValueUtil.copyToNewKeyValue(). FastDiffDeltaEncoder then reads the copied
cell for previous-value length and content comparison. The previous key
metadata must remain stable, but copying the complete value is unnecessary
because the encoded value bytes are already present in the writer buffer.
Upstream code pointers:
-
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/EncodingState.java:43-49
-
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java:217-269
-
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:822-827
## Proposed fix
Keep a key-only materialization for general encoder state, track the previous
value length separately, and make FAST_DIFF compare against the value bytes
already present in the encoded writer buffer after shipping. Pass the writer
buffer range to the encoding state and add focused regression coverage for
shipped cells, reused buffers, and encoding failures.
## Reproduction
A focused regression test will be included with the proposed change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
