[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-05-07 Thread Andrew Kyle Purtell (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-26862:

Fix Version/s: (was: 2.5.0)
   (was: 2.6.0)
   (was: 3.0.0-alpha-3)
   (was: 2.4.13)

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-05-04 Thread Xiaolin Ha (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-26862:
---
Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

This error is introduced by branch-1, and I find there are some differences 
between the upper layer scanner close codes of branch-1 and branch-2+. There is 
no need for branch-2+ to fix this issue. Just close it now.

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-04-29 Thread Andrew Kyle Purtell (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated HBASE-26862:

Fix Version/s: 2.4.13
   (was: 2.4.12)

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-03-20 Thread Xiaolin Ha (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-26862:
---
Component/s: regionserver

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver, Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-03-20 Thread Xiaolin Ha (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-26862:
---
Priority: Minor  (was: Major)

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Updated] (HBASE-26862) Fix CME when closing region

2022-03-20 Thread Xiaolin Ha (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaolin Ha updated HBASE-26862:
---
Status: Patch Available  (was: Open)

> Fix CME when closing region
> ---
>
> Key: HBASE-26862
> URL: https://issues.apache.org/jira/browse/HBASE-26862
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 2.4.11
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.12
>
>
> KeyValueHeap.scannersForDelayedClose is not thread safe, CME might occurs 
> when closing a scanning region, who is adding scanners to the  delayed close 
> list.
> {code:java}
> 2022-03-18 17:24:17,435 INFO  [regionserver/hostname/ip:16020-splits-0] 
> regionserver.SplitRequest: Running rollback/cleanup of failed split of 
> hmart-waimaiups_poi_bitmap_theme_pool,,1645769651132.56363df04da85752aa67669989de19fd.;
>  java.util.ConcurrentModificationException
> java.io.IOException: java.util.ConcurrentModificationException
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1767)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1593)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:394)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:276)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:567)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.doSplitting(SplitRequest.java:82)
>         at 
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:156)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>         at java.util.ArrayList$Itr.next(ArrayList.java:851)
>         at 
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.close(KeyValueHeap.java:221)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:548)
>         at 
> org.apache.hadoop.hbase.regionserver.StoreScanner.close(StoreScanner.java:520)
>         at 
> org.apache.hadoop.hbase.regionserver.HStore.closeChangedReadersObservers(HStore.java:1296)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:1001)
>         at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:131)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1745)
>         at 
> org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1741)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         ... 3 more {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)