[ https://issues.apache.org/jira/browse/HBASE-27384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608018#comment-17608018 ]
Aman Poonia edited comment on HBASE-27384 at 9/22/22 1:03 AM: -------------------------------------------------------------- The issue here is we use LinkedHashSet which would throws exception when we modify the Set after a iterator is created. [https://github.com/apache/hbase/blob/branch-2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerWorkQueue.java#L184] {code:java} // code placeholder public E take() throws InterruptedException { E x; takeLock.lockInterruptibly(); try { while (delegate.isEmpty()) { notEmpty.await(); } final Iterator<E> iter = delegate.iterator(); x = iter.next(); iter.remove(); if (!delegate.isEmpty()) { notEmpty.signal(); } } finally { takeLock.unlock(); } return x; } {code} [LinkedHasSet javadoc|https://docs.oracle.com/javase/7/docs/api/java/util/LinkedHashSet.html] As we can see in above code while we are reading the set, we don't take putLock and only use takeLock which leaves the Set open for modification. was (Author: mnpoonia): The issue here is we use LinkedHashSet which would throws exception when we modify the Set after a iterator is created. [https://github.com/apache/hbase/blob/branch-2.4/hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/RegionNormalizerWorkQueue.java#L184] {code:java} // code placeholder public E take() throws InterruptedException { E x; takeLock.lockInterruptibly(); try { while (delegate.isEmpty()) { notEmpty.await(); } final Iterator<E> iter = delegate.iterator(); x = iter.next(); iter.remove(); if (!delegate.isEmpty()) { notEmpty.signal(); } } finally { takeLock.unlock(); } return x; } {code} As we can see in above code while we are reading the set, we don't take putLock and only use takeLock which leaves the Set open for modification. > Concurrent modification in RegionNormalizerWorkQueue > ---------------------------------------------------- > > Key: HBASE-27384 > URL: https://issues.apache.org/jira/browse/HBASE-27384 > Project: HBase > Issue Type: Bug > Components: Normalizer > Affects Versions: 2.4.14 > Reporter: Aman Poonia > Assignee: Aman Poonia > Priority: Minor > > {*}Error: > java.util.ConcurrentModificationException{*}{{{}java.util.concurrent.ExecutionException: > java.util.ConcurrentModificationException at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at > org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.testTake(TestRegionNormalizerWorkQueue.java:211) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at > org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at > org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at > org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) > at > org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.lang.Thread.run(Thread.java:750) Caused by: > java.util.ConcurrentModificationException at > java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) > at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742) at > org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorkQueue.take(RegionNormalizerWorkQueue.java:192) > at > org.apache.hadoop.hbase.master.normalizer.TestRegionNormalizerWorkQueue.lambda$testTake$3(TestRegionNormalizerWorkQueue.java:192) > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) > at > java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632) > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at > java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175){}}} -- This message was sent by Atlassian Jira (v8.20.10#820010)