[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException when 50 regions present
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13279413#comment-13279413
]
Hudson commented on HBASE-6018:
---
Integrated in HBase-0.94-security #27 (See
[https://builds.apache.org/job/HBase-0.94-security/27/])
HBASE-6018 hbck fails with a RejectedExecutionException when 50 regions
present (Revision 1339877)
Result = SUCCESS
jmhsieh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbck fails with a RejectedExecutionException when 50 regions present
-
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277599#comment-13277599
]
Jonathan Hsieh commented on HBASE-6018:
---
I will commit if this passes the hadoopqabot.
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277643#comment-13277643
]
Hadoop QA commented on HBASE-6018:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12527787/hbase-6018-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 32 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1907//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1907//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1907//console
This message is automatically generated.
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277686#comment-13277686
]
Anoop Sam John commented on HBASE-6018:
---
I feel this change should be fine enough to handle our requirement.
We make the core threads count = max threads count and use an unbounded queue.
hbasefsck.numthreads - By this count we mean ideally that these many
parallel operations can run.
As we make the core threads count as this value rather than 1, the submitted
tasks will create new threads or allocat to available threads. Only when more
than these many requests, those will get queued.
Also we make executor.allowCoreThreadTimeOut(true). So there is absolutely no
problem in increasing the core threads count from 1..
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277920#comment-13277920
]
Lars Hofhansl commented on HBASE-6018:
--
The change is fine. I was just pointing out the executor weirdness.
From your quoted text...
bq. If corePoolSize or more threads are running, the Executor always prefers
queuing a request rather than adding a new thread
Since LinkedBlockingQueue is unbounded the executor will *never* grow the
number of threads past the number of core threads specified (re-read the doc if
you don't believe me :) ). This is certainly not what you want although in the
case of HBCK it does not matter, as you're not enqueuing 10's of tasks.
At least the matching change should be to set core threads and max threads to
the same value (in order to avoid later confusion).
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277922#comment-13277922
]
Lars Hofhansl commented on HBASE-6018:
--
NM me. The patch already does this. +1
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException when 50 regions present
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278363#comment-13278363
]
Hudson commented on HBASE-6018:
---
Integrated in HBase-TRUNK #2895 (See
[https://builds.apache.org/job/HBase-TRUNK/2895/])
HBASE-6018 hbck fails with a RejectedExecutionException when 50 regions
present (Revision 1339876)
Result = SUCCESS
jmhsieh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbck fails with a RejectedExecutionException when 50 regions present
-
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException when 50 regions present
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278377#comment-13278377
]
Hudson commented on HBASE-6018:
---
Integrated in HBase-0.94 #197 (See
[https://builds.apache.org/job/HBase-0.94/197/])
HBASE-6018 hbck fails with a RejectedExecutionException when 50 regions
present (Revision 1339877)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbck fails with a RejectedExecutionException when 50 regions present
-
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException when 50 regions present
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278420#comment-13278420
]
Hudson commented on HBASE-6018:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #8 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/8/])
HBASE-6018 hbck fails with a RejectedExecutionException when 50 regions
present (Revision 1339876)
Result = FAILURE
jmhsieh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbck fails with a RejectedExecutionException when 50 regions present
-
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException when 50 regions present
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278523#comment-13278523
]
Hudson commented on HBASE-6018:
---
Integrated in HBase-0.92-security #107 (See
[https://builds.apache.org/job/HBase-0.92-security/107/])
HBASE-6018 hbck fails with a RejectedExecutionException when 50 regions
present (Revision 1339879)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbck fails with a RejectedExecutionException when 50 regions present
-
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: hbase-6018-v2.patch, hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276787#comment-13276787
]
Jonathan Hsieh commented on HBASE-6018:
---
This line seems related to attempt to enqueue a work item into a
SynchronousQueue introduced in HBASE-4859. I don't understand why a
SynchronousQueue is used (it has no capacity!)
Problem goes away after this change:
{code}
diff --git a/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
index 83aa316..8a050fd 100644
--- a/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
+++ b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
@@ -33,7 +33,8 @@ import java.util.SortedSet;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentSkipListMap;
-import java.util.concurrent.SynchronousQueue;
+//import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
@@ -217,9 +218,9 @@ public class HBaseFsck {
this.conf = conf;
int numThreads = conf.getInt(hbasefsck.numthreads, MAX_NUM_THREADS);
-executor = new ThreadPoolExecutor(1, numThreads,
+executor = new ThreadPoolExecutor(numThreads, numThreads,
THREADS_KEEP_ALIVE_SECONDS, TimeUnit.SECONDS,
-new SynchronousQueueRunnable());
+new LinkedBlockingQueueRunnable());
executor.allowCoreThreadTimeOut(true);
}
{code}
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276813#comment-13276813
]
Jonathan Hsieh commented on HBASE-6018:
---
When there are MAX_NUM_THREADS regions, the SynchronousQueue version works,
when it there are MAX_NUM_THREADS regions it fails with the
RejectedExectionException.
A workaround is to add hbase-site.xml hbasefsck.numthreads property set to a
value larger than the number of regions in your hbase instance. (you can
purposely set it low to trigger the problem).
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276844#comment-13276844
]
Hadoop QA commented on HBASE-6018:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12527633/hbase-6018.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 31 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1888//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1888//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1888//console
This message is automatically generated.
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277146#comment-13277146
]
Zhihong Yu commented on HBASE-6018:
---
Patch looks good.
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277237#comment-13277237
]
Lars Hofhansl commented on HBASE-6018:
--
ThreadPoolExecutor is pretty stupid (if you ask me). Unless the Queue fills up
it will never allocate more than threads than indicated by the number of core
threads.
The LinkedBlockingQueue has no limit, so it'll only ever use
conf.getInt(hbasefsck.numthreads) number of threads.
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6018) hbck fails with a RejectedExecutionException
[
https://issues.apache.org/jira/browse/HBASE-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277581#comment-13277581
]
Jonathan Hsieh commented on HBASE-6018:
---
I'm convinced that SynchronousQueue was just wrong and the failure was
expected behavior. I'm a little confused by your statement Lars; queues are
just used if there are more tasks than max concurrent enough threads in the
pool.
See java doc here
http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/ThreadPoolExecutor.html
{quote}
Queuing
Any BlockingQueue may be used to transfer and hold submitted tasks. The use of
this queue interacts with pool sizing:
If fewer than corePoolSize threads are running, the Executor always prefers
adding a new thread rather than queuing.
If corePoolSize or more threads are running, the Executor always prefers
queuing a request rather than adding a new thread.
If a request cannot be queued, a new thread is created unless this would exceed
maximumPoolSize, in which case, the task will be rejected.
{quote}
hbck fails with a RejectedExecutionException
Key: HBASE-6018
URL: https://issues.apache.org/jira/browse/HBASE-6018
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-6018.patch
On a long running job 0.94.0rc3 cluster, we get to a point where hbck
consistently encounters this error and fails:
{code}
Exception in thread main java.util.concurrent.RejectedExecutionException
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.hadoop.hbase.util.HBaseFsck.loadHdfsRegionInfos(HBaseFsck.java:633)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:354)
at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:382)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3120)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
