[
https://issues.apache.org/jira/browse/HBASE-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Liu updated HBASE-30246:
-----------------------------
Description:
During MOB compaction, {{DefaultMobStoreCompactor}} resolves each MOB reference
cell via:
{code:java}
mobCell = mobStore.resolve(c, cacheMobBlocksOnCompaction, false).getCell();
{code}
{{HMobStore#resolve(...)}} returns a {{Closeable}} {{MobCell}} that owns the
{{StoreFileScanner}} opened against the MOB file (and the off-heap ByteBuffers
backing
the cell). The code calls {{getCell()}} and drops the {{MobCell}}, so the
scanner/buffers
are never released — once per reference cell, for the whole compaction. On the
off-heap
read path this pressures the {{ByteBuffAllocator}} pool and can lead to buffer
exhaustion
during large MOB compactions. The bug is long-standing (it predates
HBASE-30177, which only
changed the cacheBlocks argument) and was spotted during the HBASE-30177 review.
h3. Fix
Close the {{MobCell}} via try-with-resources right after resolving. The cell
must be copied
to the heap *before* close, because closing releases/recycles the buffers
backing it, while
HFile writers/encoders retain references to appended cells (lastCell,
firstCellInBlock,
encoder prevCell) until {{beforeShipped()}}. Closing without copying would turn
the leak into
silent HFile corruption.
{code:java}
protected ExtendedCell resolveMobCell(ExtendedCell reference) throws
IOException {
try (MobCell mobCell = mobStore.resolve(reference,
cacheMobBlocksOnCompaction, false)) {
return KeyValueUtil.copyToNewKeyValue(mobCell.getCell());
}
}
{code}
h3. Testing
* New
{{TestDefaultMobStoreCompactor#testResolveMobCellClosesMobCellAndReturnsIndependentCopy}}
asserts {{close()}} is called and an independent heap copy is returned.
* {{TestHMobStore}} updated to close resolved {{MobCell}}s.
was:
During MOB compaction, \{{DefaultMobStoreCompactor}} resolves each MOB
reference cell via:
{code:java}
mobCell = mobStore.resolve(c, cacheMobBlocksOnCompaction, false).getCell();
{code}
{\{HMobStore#resolve(...)}} returns a \{{Closeable}} \{{MobCell}} that owns the
{\{StoreFileScanner}} opened against the MOB file (and the off-heap ByteBuffers
backing
the cell). The code calls \{{getCell()}} and drops the \{{MobCell}}, so the
scanner/buffers
are never released — once per reference cell, for the whole compaction. On the
off-heap
read path this pressures the \{{ByteBuffAllocator}} pool and can lead to buffer
exhaustion
during large MOB compactions. The bug is long-standing (it predates
HBASE-30177, which only
changed the cacheBlocks argument) and was spotted during the HBASE-30177 review.
h3. Fix
Close the \{{MobCell}} via try-with-resources right after resolving. The cell
must be copied
to the heap *before* close, because closing releases/recycles the buffers
backing it, while
HFile writers/encoders retain references to appended cells (lastCell,
firstCellInBlock,
encoder prevCell) until \{{beforeShipped()}}. Closing without copying would
turn the leak into
silent HFile corruption.
{code:java}
protected ExtendedCell resolveMobCell(ExtendedCell reference) throws
IOException {
try (MobCell mobCell = mobStore.resolve(reference, cacheMobBlocksOnCompaction,
false)) {
return KeyValueUtil.copyToNewKeyValue(mobCell.getCell());
}
}
{code}
h3. Testing
* New
\{{TestDefaultMobStoreCompactor#testResolveMobCellClosesMobCellAndReturnsIndependentCopy}}
asserts \{{close()}} is called and an independent heap copy is returned.
* \{{TestHMobStore}} updated to close resolved \{{MobCell}}s.
> MOB compaction does not close MobCell after resolving reference cells
> ---------------------------------------------------------------------
>
> Key: HBASE-30246
> URL: https://issues.apache.org/jira/browse/HBASE-30246
> Project: HBase
> Issue Type: Bug
> Components: Compaction, mob
> Reporter: Xiao Liu
> Assignee: Xiao Liu
> Priority: Major
> Fix For: 2.7.0, 3.0.0-beta-2, 2.5.16, 2.6.7
>
>
> During MOB compaction, {{DefaultMobStoreCompactor}} resolves each MOB
> reference cell via:
> {code:java}
> mobCell = mobStore.resolve(c, cacheMobBlocksOnCompaction, false).getCell();
> {code}
> {{HMobStore#resolve(...)}} returns a {{Closeable}} {{MobCell}} that owns the
> {{StoreFileScanner}} opened against the MOB file (and the off-heap
> ByteBuffers backing
> the cell). The code calls {{getCell()}} and drops the {{MobCell}}, so the
> scanner/buffers
> are never released — once per reference cell, for the whole compaction. On
> the off-heap
> read path this pressures the {{ByteBuffAllocator}} pool and can lead to
> buffer exhaustion
> during large MOB compactions. The bug is long-standing (it predates
> HBASE-30177, which only
> changed the cacheBlocks argument) and was spotted during the HBASE-30177
> review.
> h3. Fix
> Close the {{MobCell}} via try-with-resources right after resolving. The cell
> must be copied
> to the heap *before* close, because closing releases/recycles the buffers
> backing it, while
> HFile writers/encoders retain references to appended cells (lastCell,
> firstCellInBlock,
> encoder prevCell) until {{beforeShipped()}}. Closing without copying would
> turn the leak into
> silent HFile corruption.
> {code:java}
> protected ExtendedCell resolveMobCell(ExtendedCell reference) throws
> IOException {
> try (MobCell mobCell = mobStore.resolve(reference,
> cacheMobBlocksOnCompaction, false)) {
> return KeyValueUtil.copyToNewKeyValue(mobCell.getCell());
> }
> }
> {code}
> h3. Testing
> * New
> {{TestDefaultMobStoreCompactor#testResolveMobCellClosesMobCellAndReturnsIndependentCopy}}
> asserts {{close()}} is called and an independent heap copy is returned.
> * {{TestHMobStore}} updated to close resolved {{MobCell}}s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
