[
https://issues.apache.org/jira/browse/HBASE-28065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17762370#comment-17762370
]
Nick Dimiduk commented on HBASE-28065:
--------------------------------------
{noformat}
2023-08-28T13:59:25,123 [regionserver/hostname:60020-shortCompactions-0] ERROR
org.apache.hadoop.hbase.regionserver.CompactSplit: Compaction failed
region=region,1688226949649.e656bdf67a97606e533632c12be7d346.,
storeName=e656bdf67a97606e533632c12be7d346/0, priority=10,
startTime=1693231165117
java.lang.IllegalArgumentException: size to allocate should >=0
at
org.apache.hadoop.hbase.io.ByteBuffAllocator.allocate(ByteBuffAllocator.java:300)
~[hbase-common-2.5-hubspot-20230418.173647-18.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.allocate(HFileBlock.java:1623)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1694)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1503)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1331)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1252)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.readAndUpdateNewBlock(HFileReaderImpl.java:956)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$HFileScannerImpl.seekTo(HFileReaderImpl.java:945)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:312)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:215)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:412)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:323)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:289)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.createScanner(Compactor.java:538)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor$1.createScanner(Compactor.java:265)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:354)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:122)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1145)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2287)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:667)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:716)
~[hbase-server-2.5-hubspot-20230418.183923-40.jar:2.5-hubspot-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
{noformat}
> Corrupt HFile data is mishandled in several cases
> -------------------------------------------------
>
> Key: HBASE-28065
> URL: https://issues.apache.org/jira/browse/HBASE-28065
> Project: HBase
> Issue Type: Bug
> Components: HFile
> Affects Versions: 2.5.2
> Reporter: Nick Dimiduk
> Priority: Major
>
> While riding over a spat of HDFS data corruption issues, we've observed
> several places in the read path that do not fall back to HDFS checksum
> appropriately. These failures manifest during client reads and during
> compactions. Sometimes failure is detected by the fallback
> {{verifyOnDiskSizeMatchesHeader}}, sometimes we attempt to allocate a buffer
> with a negative size, and sometimes we read through to a failure from block
> decompression.
> After code study, I think that all three cases arise from using a block
> header that was read without checksum validation.
> Will post up the stack traces in the comments. Now sure if we'll want a
> single patch or multiple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
