[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261330#comment-14261330 ] Dima Spivak commented on HBASE-12740: - :) I'm convinced. You're awesome, [~jurmous]. Just two minor nits: - Since poolSize is a constant, we should probably have it be a static final int and call it POOL_SIZE to make it stick out. - When constructing the scheduled thread pool executor, we should pass in POOL_SIZE instead of the hardcoded 7 (if I'm not mistaken). Other than that, great work. +1 Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261442#comment-14261442
]
Hadoop QA commented on HBASE-12740:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689556/HBASE-12740-v4.patch
against master branch at commit c6002ccd3873f3950df50bef7f4ac4c5760d595f.
ATTACHMENT ID: 12689556
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12265//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740.patch,
PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261473#comment-14261473 ] stack commented on HBASE-12740: --- Pardon me. I applied v3 with [~dimaspivak] items addressed. Was then trying to cherry-pick to branch-1 and beyond but was taking a little while (TestHBaseFsck runs are a bit of beast). I reverted my master work since [~jurmous] showed up w/ new version. Will go forward with v4. Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261499#comment-14261499 ] Hudson commented on HBASE-12740: SUCCESS: Integrated in HBase-TRUNK #5978 (See [https://builds.apache.org/job/HBase-TRUNK/5978/]) HBASE-12740 Improve performance of TestHBaseFsck (stack: rev 042743fff076a66e25c93fce97ab3e75adb8190d) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261568#comment-14261568
]
Hadoop QA commented on HBASE-12740:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689560/HBASE-12740-v4.patch
against master branch at commit 734324e80f44be4d40b1ee4612a838b581d504b2.
ATTACHMENT ID: 12689560
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12267//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch,
HBASE-12740.patch, PROFILE_after_patch.png,
PROFILE_before_patch_test_fails.png
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261570#comment-14261570 ] Hudson commented on HBASE-12740: SUCCESS: Integrated in HBase-TRUNK #5979 (See [https://builds.apache.org/job/HBase-TRUNK/5979/]) Revert HBASE-12740 Improve performance of TestHBaseFsck (stack: rev 734324e80f44be4d40b1ee4612a838b581d504b2) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261674#comment-14261674 ] Hudson commented on HBASE-12740: SUCCESS: Integrated in HBase-1.0 #621 (See [https://builds.apache.org/job/HBase-1.0/621/]) HBASE-12740 Improve performance of TestHBaseFsck (stack: rev d940e8f6c48ffafb4406da572c05ee6353d4c28b) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261713#comment-14261713 ] Hudson commented on HBASE-12740: FAILURE: Integrated in HBase-TRUNK #5980 (See [https://builds.apache.org/job/HBase-TRUNK/5980/]) HBASE-12740 Improve performance of TestHBaseFsck (stack: rev baa2d05374efc0f5b95b8f1f779462038a9f96b2) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261738#comment-14261738 ] Hudson commented on HBASE-12740: SUCCESS: Integrated in HBase-1.1 #38 (See [https://builds.apache.org/job/HBase-1.1/38/]) HBASE-12740 Improve performance of TestHBaseFsck (stack: rev ad7d7f59775556157215a710456d8401056165c6) * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740-v4.patch, HBASE-12740-v4.patch, HBASE-12740.patch, PROFILE_after_patch.png, PROFILE_before_patch_test_fails.png TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260480#comment-14260480 ] Dima Spivak commented on HBASE-12740: - Hey [~jurmous], Sorry for the delay in getting back to this; busy time with the holidays. I finally had a chance to apply your patch to master (thanks again for rebasing for me) and, having run it a half dozen times, I can't seem to reproduce the speed-up you're seeing on TestHBaseFsck. Running Maven 3.0.4 and Java 1.7u67, I get averaged runtimes of about 277 s before the patch is applied and 287 s afterwards. Am I missing something? Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740-v3.patch, HBASE-12740.patch TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257020#comment-14257020
]
Hadoop QA commented on HBASE-12740:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688843/HBASE-12740-v1.patch
against master branch at commit 0d5f967c7ff4809ca782f68a9e48896dd04b16f6.
ATTACHMENT ID: 12688843
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12205//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740.patch
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257129#comment-14257129
]
Hadoop QA commented on HBASE-12740:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688855/HBASE-12740-v2.patch
against master branch at commit 0d5f967c7ff4809ca782f68a9e48896dd04b16f6.
ATTACHMENT ID: 12688855
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12207//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740.patch
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257360#comment-14257360 ] Dima Spivak commented on HBASE-12740: - Jurriaan, Would you mind posting another patch of this rebased to the latest commit of master? I can't apply it cleanly since it collides with your changes in HBASE-12736. I'll take one final look around once hadoop-qa runs on that, but this looks fantastic. Well done. Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch, HBASE-12740.patch TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14257761#comment-14257761
]
Hadoop QA commented on HBASE-12740:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688938/HBASE-12740-v3.patch
against master branch at commit 2b090d6832c944ea47f81e1b6db518a74c1149e0.
ATTACHMENT ID: 12688938
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12215//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740-v1.patch, HBASE-12740-v2.patch,
HBASE-12740-v3.patch, HBASE-12740.patch
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14255986#comment-14255986 ] Dima Spivak commented on HBASE-12740: - Thanks so much for taking this on, Jurriaan (especially taking a good step in moving to the new non H- objects). Looks lovely, with some comments: - Since I see a lot of 7s and factors of 7, might it be worth defining a constant at the top and then just using throughout so that if this needs tuning in the future, it can be changed in one place? - HBaseTestingUtility has its own deleteTable method. Might it be worth plugging those in here instead of having this alternative implementation around? - What was the rationale for having setupTableWithRegionReplica return an HTable instance (instead of a plain Table)? Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740.patch TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[
https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256041#comment-14256041
]
Hadoop QA commented on HBASE-12740:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688662/HBASE-12740.patch
against master branch at commit 4d53fe5dcd7b3bd8ca59e2a0683697e2cddaecfc.
ATTACHMENT ID: 12688662
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the master's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12186//console
This message is automatically generated.
Improve performance of TestHBaseFsck
Key: HBASE-12740
URL: https://issues.apache.org/jira/browse/HBASE-12740
Project: HBase
Issue Type: Bug
Components: util
Reporter: Jurriaan Mous
Assignee: Jurriaan Mous
Attachments: HBASE-12740.patch
TestHBaseFsck performs poor on my machine. It crashes because the threads
reach the 2000 thread limit on my machine. Looking at the code a lot of
optimization is possible and some API calls are used wrong. A lot of Admin
instances are created and never closed, lots of Tables are not closed,
ThreadPoolExecutors are not shut down and an unlimited thread pool which does
not recycle threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12740) Improve performance of TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256328#comment-14256328 ] stack commented on HBASE-12740: --- The releaseaudit warning is not this patch and has since been fixed. Unapproved licenses: src/main/docbkx/hbase-default.xml Patch looks great caveat [~dimaspivak] feedback Improve performance of TestHBaseFsck Key: HBASE-12740 URL: https://issues.apache.org/jira/browse/HBASE-12740 Project: HBase Issue Type: Bug Components: util Reporter: Jurriaan Mous Assignee: Jurriaan Mous Attachments: HBASE-12740.patch TestHBaseFsck performs poor on my machine. It crashes because the threads reach the 2000 thread limit on my machine. Looking at the code a lot of optimization is possible and some API calls are used wrong. A lot of Admin instances are created and never closed, lots of Tables are not closed, ThreadPoolExecutors are not shut down and an unlimited thread pool which does not recycle threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
