[
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508820#comment-17508820
]
Xiaolin Ha commented on HBASE-26862:
------------------------------------
Though the exception stack trace shows that it happens in the rollback of a
failed split process, I think it can be happened to branch-2+, too.
Because the CME is caused by concurrent modify of
KeyValueHeap#scannersForDelayedClose(an ArrayList), which will be iterated in
KeyValueHeap#close() and modified in KeyValueHeap#next().
{code:java}
public void close() {
for (KeyValueScanner scanner : this.scannersForDelayedClose) {
scanner.close();
}
..... {code}
{code:java}
@Override
public Cell next() throws IOException {
if(this.current == null) {
return null;
}
Cell kvReturn = this.current.next();
Cell kvNext = this.current.peek();
if (kvNext == null) {
this.scannersForDelayedClose.add(this.current);
.... {code}
Actually if there are concurrent region close and scanning(to the end), this
problem may occur.
> Fix CME when closing region
> ---------------------------
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Affects Versions: 2.4.11
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs
> when closing a scanning region, who is adding scanners to the delayed close
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO [regionserver/hostname/ip:16020-splits-0]
> regionserver.SplitRequest: Running rollback/cleanup of failed split of
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
> java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
> at
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
> at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
