[
https://issues.apache.org/jira/browse/HBASE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310330#comment-14310330
]
Enis Soztutar commented on HBASE-12891:
---------------------------------------
With addendum TestHBaseFsck fails now. tableInfos is not thread safe. I think
we should revert the patch until we fix it.
{code}
java.io.IOException: java.util.ConcurrentModificationException
at
org.apache.hadoop.hbase.util.HBaseFsck.checkAndFixConsistency(HBaseFsck.java:1726)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:593)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:612)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:68)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testSplitDaughtersNotInMeta(TestHBaseFsck.java:1729)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionConsistency(HBaseFsck.java:2046)
at org.apache.hadoop.hbase.util.HBaseFsck.access$2(HBaseFsck.java:1968)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegionConsistency.call(HBaseFsck.java:2388)
at
org.apache.hadoop.hbase.util.HBaseFsck$WorkItemRegionConsistency.call(HBaseFsck.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
> Parallel execution for Hbck checkRegionConsistency
> --------------------------------------------------
>
> Key: HBASE-12891
> URL: https://issues.apache.org/jira/browse/HBASE-12891
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 2.0.0, 0.98.10, 1.1.0
> Reporter: churro morales
> Assignee: churro morales
> Fix For: 2.0.0, 1.1.0, 0.98.11
>
> Attachments: HBASE-12891-v1.patch, HBASE-12891.98.patch,
> HBASE-12891.patch, HBASE-12891.patch, hbase-12891-addendum1.patch
>
>
> We have a lot of regions on our cluster ~500k and noticed that hbck took
> quite some time in checkAndFixConsistency(). [~davelatham] patched our
> cluster to do this check in parallel to speed things up. I'll attach the
> patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
