[jira] [Updated] (HBASE-4518) TestServerCustomProtocol is flaky
[
https://issues.apache.org/jira/browse/HBASE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-4518:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Hadoop QA test failures were unrelated, so patch committed to 0.92 branch and
trunk.
TestServerCustomProtocol is flaky
-
Key: HBASE-4518
URL: https://issues.apache.org/jira/browse/HBASE-4518
Project: HBase
Issue Type: Test
Components: coprocessors, test
Affects Versions: 0.92.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Fix For: 0.92.0
Attachments: HBASE-4518.patch,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol-output.txt,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.txt
TestServerCustomProtocol has been showing some intermittent failures in
Jenkins due to what looks like region transitions.
Here is the most recent failure:
{noformat}
Results :
Failed tests:
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
Results should contain region
test,bbb,1317332645939.aea9154349b9e0dc207e2e9476702763. for row 'bbb'
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142866#comment-13142866 ] gaojinchao commented on HBASE-4577: --- Yes,I think so. But I am diging. Because I am not familiar with MR and this issue is not very important to release 0.92 version, Please give me some time. Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB - Key: HBASE-4577 URL: https://issues.apache.org/jira/browse/HBASE-4577 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: gaojinchao Priority: Minor Fix For: 0.92.0 Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch Minor issue while looking at the RS metrics: bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008 I guess there's a truncation somewhere when it's adding the numbers up. FWIW there's no compression on that table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4719) HBase script assumes pre-Hadoop 0.21 layout of jar files
[
https://issues.apache.org/jira/browse/HBASE-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142871#comment-13142871
]
Hudson commented on HBASE-4719:
---
Integrated in HBase-TRUNK #2405 (See
[https://builds.apache.org/job/HBase-TRUNK/2405/])
HBASE-4719 HBase script assumes pre-Hadoop 0.21 layout of jar files
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/bin/hbase
HBase script assumes pre-Hadoop 0.21 layout of jar files
Key: HBASE-4719
URL: https://issues.apache.org/jira/browse/HBASE-4719
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Fix For: 0.92.0
Attachments: HBASE-4719.patch.txt
The following in the bin/hbase:
{noformat}
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar`)
{noformat}
assumes a pre-21 Hadoop layout. It'll be better to dynamically account for
either hadoop-core* or hadoop-common*, hadoop-hdfs*, hadoop-mapreduce*
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4736) [book] troubleshooting.xml - xrefs to arch chapter
[ https://issues.apache.org/jira/browse/HBASE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142872#comment-13142872 ] Hudson commented on HBASE-4736: --- Integrated in HBase-TRUNK #2405 (See [https://builds.apache.org/job/HBase-TRUNK/2405/]) HBASE-4736 troubleshooting.xml, xrefs to Arch chapter dmeil : Files : * /hbase/trunk/src/docbkx/troubleshooting.xml [book] troubleshooting.xml - xrefs to arch chapter -- Key: HBASE-4736 URL: https://issues.apache.org/jira/browse/HBASE-4736 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Trivial Attachments: troubleshooting_HBASE_4736.xml.patch troubleshooting.xml * added xrefs from Client, NameNode, and RegionServer sections in Troubleshooting to their respective Architecture chapter sections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4480) Testing script to simplify local testing
[ https://issues.apache.org/jira/browse/HBASE-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142873#comment-13142873 ] Hudson commented on HBASE-4480: --- Integrated in HBase-TRUNK #2405 (See [https://builds.apache.org/job/HBase-TRUNK/2405/]) HBASE-4480 Testing script to simplify local testing HBASE-4480 Testing script to simplify local testing stack : Files : * /hbase/trunk/src/test/bin * /hbase/trunk/src/test/bin/test-util.sh stack : Files : * /hbase/trunk/CHANGES.txt Testing script to simplify local testing Key: HBASE-4480 URL: https://issues.apache.org/jira/browse/HBASE-4480 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jesse Yates Priority: Minor Labels: test Fix For: 0.94.0 Attachments: HBASE-4480.patch, HBASE-4480_v2.patch, HBASE-4480_v3.patch, HBASE-4480_v4.patch, runtest-no-npe-check.sh, runtest.sh, runtest2.sh As mentioned by http://search-hadoop.com/m/r2Ab624ES3e and http://search-hadoop.com/m/cZjDH1ykGIA it would be nice if we could have a script that would handle more of the finer points of running/checking our test suite. This script should: (1) Allow people to determine which tests are hanging/taking a long time to run (2) Allow rerunning of particular tests to make sure it wasn't an artifact of running the whole suite that caused the failure (3) Allow people to specify to run just unit tests or also integration tests (essentially wrapping calls to 'maven test' and 'maven verify'). This script should just be a convenience script - running tests directly from maven should not be impacted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.
[ https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4718: -- Attachment: hbase-4718.v3.includes-hbase-3316.patch Updated patch generated using --no-prefix, uses enum from HBASE-4716, and also includes HBASE-3316. Backport HBASE-4552 to 0.90 branch. --- Key: HBASE-4718 URL: https://issues.apache.org/jira/browse/HBASE-4718 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 4718-v2.90, hbase-4718.0.90.patch, hbase-4718.v3.includes-hbase-3316.patch In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about whether and how to backport HBASE-4552 to the 0.90 branch. This is a potentially compatibility breaking so several approaches hav ebeen suggested. 1) provide patch but do not integrate 2) integrate patch that extends and deprecates old api without removing old api. It has been argued that clients are supposed to use LoadIncrementalHFiles api and not at the internal HRegionServer RPC api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.
[ https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142915#comment-13142915 ] Hadoop QA commented on HBASE-4718: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502112/hbase-4718.v3.includes-hbase-3316.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/153//console This message is automatically generated. Backport HBASE-4552 to 0.90 branch. --- Key: HBASE-4718 URL: https://issues.apache.org/jira/browse/HBASE-4718 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 4718-v2.90, hbase-4718.0.90.patch, hbase-4718.v3.includes-hbase-3316.patch In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about whether and how to backport HBASE-4552 to the 0.90 branch. This is a potentially compatibility breaking so several approaches hav ebeen suggested. 1) provide patch but do not integrate 2) integrate patch that extends and deprecates old api without removing old api. It has been argued that clients are supposed to use LoadIncrementalHFiles api and not at the internal HRegionServer RPC api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142914#comment-13142914 ] Lars Hofhansl commented on HBASE-4583: -- I have a new patch that does the above. I wrote a multithreaded stress test and I found the following: 1. I get incorrect behavior if I remove duplicates after the rowlock is released. (doing that with the rowlock held solved the problem). 2. Even though the end result is correct, if I have the threads do GET operations I see that they do not always get all columns for a row. More interestingly I find sometimes that even KVs with a different timestamps and different memstoreTS show up in the same Get. (note that no flushing happens during the tests, so this has nothing to do with store files not carrying a memstoreTS). I thought at the least the Memstore was consistent (i.e. scanners only see columns when rwcc is rolled forward, but then they would see all columns affected by that writepoint). Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142929#comment-13142929 ] Hadoop QA commented on HBASE-4298: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502103/4298-trunk-v3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.TestGlobalMemStoreSize org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/152//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/152//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/152//console This message is automatically generated. Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0, 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4719) HBase script assumes pre-Hadoop 0.21 layout of jar files
[
https://issues.apache.org/jira/browse/HBASE-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13142937#comment-13142937
]
Hudson commented on HBASE-4719:
---
Integrated in HBase-0.92 #103 (See
[https://builds.apache.org/job/HBase-0.92/103/])
HBASE-4719 HBase script assumes pre-Hadoop 0.21 layout of jar files
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
* /hbase/branches/0.92/bin/hbase
HBase script assumes pre-Hadoop 0.21 layout of jar files
Key: HBASE-4719
URL: https://issues.apache.org/jira/browse/HBASE-4719
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
Fix For: 0.92.0
Attachments: HBASE-4719.patch.txt
The following in the bin/hbase:
{noformat}
HADOOPCPPATH=$(append_path ${HADOOPCPPATH} `ls
${HADOOP_HOME}/hadoop-core*.jar`)
{noformat}
assumes a pre-21 Hadoop layout. It'll be better to dynamically account for
either hadoop-core* or hadoop-common*, hadoop-hdfs*, hadoop-mapreduce*
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4518) TestServerCustomProtocol is flaky
[
https://issues.apache.org/jira/browse/HBASE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143028#comment-13143028
]
Hudson commented on HBASE-4518:
---
Integrated in HBase-TRUNK #2406 (See
[https://builds.apache.org/job/HBase-TRUNK/2406/])
HBASE-4518 TestServerCustomProtocol fails intermittently
garyh :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
TestServerCustomProtocol is flaky
-
Key: HBASE-4518
URL: https://issues.apache.org/jira/browse/HBASE-4518
Project: HBase
Issue Type: Test
Components: coprocessors, test
Affects Versions: 0.92.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Fix For: 0.92.0
Attachments: HBASE-4518.patch,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol-output.txt,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.txt
TestServerCustomProtocol has been showing some intermittent failures in
Jenkins due to what looks like region transitions.
Here is the most recent failure:
{noformat}
Results :
Failed tests:
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
Results should contain region
test,bbb,1317332645939.aea9154349b9e0dc207e2e9476702763. for row 'bbb'
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143049#comment-13143049 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/ --- (Updated 2011-11-03 11:42:29.986508) Review request for hbase. Changes --- Make test cases cost less time. Summary --- Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. This addresses bug HBase-4120. https://issues.apache.org/jira/browse/HBase-4120 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestActionPriority.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestPriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestTablePriority.java PRE-CREATION Diff: https://reviews.apache.org/r/1421/diff Testing --- Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client. Thanks, Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4518) TestServerCustomProtocol is flaky
[
https://issues.apache.org/jira/browse/HBASE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143054#comment-13143054
]
Hudson commented on HBASE-4518:
---
Integrated in HBase-0.92 #104 (See
[https://builds.apache.org/job/HBase-0.92/104/])
HBASE-4518 TestServerCustomProtocol fails intermittently
garyh :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java
TestServerCustomProtocol is flaky
-
Key: HBASE-4518
URL: https://issues.apache.org/jira/browse/HBASE-4518
Project: HBase
Issue Type: Test
Components: coprocessors, test
Affects Versions: 0.92.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Fix For: 0.92.0
Attachments: HBASE-4518.patch,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol-output.txt,
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.txt
TestServerCustomProtocol has been showing some intermittent failures in
Jenkins due to what looks like region transitions.
Here is the most recent failure:
{noformat}
Results :
Failed tests:
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol):
Results should contain region
test,bbb,1317332645939.aea9154349b9e0dc207e2e9476702763. for row 'bbb'
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Jia updated HBASE-4120: --- Attachment: TablePriority_v8_for_trunk.patch table priority patch for trunk. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch, TablePriority_v8_for_trunk.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4735) [book] book.xml - schema design, added code example in keysize section
[ https://issues.apache.org/jira/browse/HBASE-4735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143076#comment-13143076 ] Doug Meil commented on HBASE-4735: -- Thanks Stack, I'll make that change in another patch. [book] book.xml - schema design, added code example in keysize section -- Key: HBASE-4735 URL: https://issues.apache.org/jira/browse/HBASE-4735 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: book_HBASE_4735.xml.patch book.xml * schema design chapter, keysize section - added code example in keysize section for long and hashes showing the differences between storing as bytes and string. This has come up on the dist-list. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143172#comment-13143172 ] gaojinchao commented on HBASE-4577: --- I didn't find any exception log. It seems the test case has a bug. Processing steps: 1. Creating table with 5 regions 2. Producing data base on 5 regions 3. Changing table to 15 regions 4. Loading data to new table Some cases, the data of 1 region may become the data of 11 regions. but the parameter of hbase.bulkload.retries.number is set to 10,we only try 10 time, some data can't load to the region. If I am wrong, Please correct me! Thanks I think we should change table to 14 rather than 15 regions in this case. Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB - Key: HBASE-4577 URL: https://issues.apache.org/jira/browse/HBASE-4577 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: gaojinchao Priority: Minor Fix For: 0.92.0 Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch Minor issue while looking at the RS metrics: bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008 I guess there's a truncation somewhere when it's adding the numbers up. FWIW there's no compression on that table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4577) Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB
[ https://issues.apache.org/jira/browse/HBASE-4577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143174#comment-13143174 ] gaojinchao commented on HBASE-4577: --- look this logs: 2011-11-02 04:23:42,761 ERROR [main] mapreduce.LoadIncrementalHFiles(214): Retry attempted 10 times without completing, bailing out 2011-11-02 04:23:42,761 ERROR [main] mapreduce.LoadIncrementalHFiles(240): - Bulk load aborted with some files not yet loaded: - hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,27.bottom hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-A/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,27.top hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,28.bottom hdfs://localhost:45622/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/4660a8cd-aaa5-466a-8ff1-5b824cd4553e/testLocalMRIncrementalLoad/info-B/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/TestTable,28.top Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB - Key: HBASE-4577 URL: https://issues.apache.org/jira/browse/HBASE-4577 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: gaojinchao Priority: Minor Fix For: 0.92.0 Attachments: HBASE-4577_trial_Trunk.patch, HBASE-4577_trunk.patch Minor issue while looking at the RS metrics: bq. numberOfStorefiles=8, storefileUncompressedSizeMB=2418, storefileSizeMB=2420, compressionRatio=1.0008 I guess there's a truncation somewhere when it's adding the numbers up. FWIW there's no compression on that table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143190#comment-13143190 ] Hadoop QA commented on HBASE-4120: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502140/TablePriority_v8_for_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -145 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 68 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/154//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/154//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/154//console This message is automatically generated. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch, TablePriority_v8_for_trunk.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Attachment: 4553-v12.txt Rebase against trunk. The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.
[ https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143230#comment-13143230 ] Ted Yu commented on HBASE-4718: --- Since enum was removed by HBASE-4716 addendum, I think patch v2 is closer to the current TRUNK. Backport HBASE-4552 to 0.90 branch. --- Key: HBASE-4718 URL: https://issues.apache.org/jira/browse/HBASE-4718 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 4718-v2.90, hbase-4718.0.90.patch, hbase-4718.v3.includes-hbase-3316.patch In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about whether and how to backport HBASE-4552 to the 0.90 branch. This is a potentially compatibility breaking so several approaches hav ebeen suggested. 1) provide patch but do not integrate 2) integrate patch that extends and deprecates old api without removing old api. It has been argued that clients are supposed to use LoadIncrementalHFiles api and not at the internal HRegionServer RPC api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker 8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM 3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner 1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver 3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache 7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager 2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException 1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner 1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler 2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager 10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler 3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568 org.apache.hadoop.hbase.regionserver.TestRpcMetrics 1 2.028 org.apache.hadoop.hbase.regionserver.TestSeekOptimizations 6 3.031 org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol 6 20.087 org.apache.hadoop.hbase.regionserver.TestSplitLogWorker 5 2.062 org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary 1 16.88
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143250#comment-13143250 ] Todd Lipcon commented on HBASE-4737: The SUREFIRE-329 patch actually lets you annotate by method, not just by class. So if there are some classes that have a couple small mixed with a large, they don't necessarily have to move to a new class Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143255#comment-13143255 ] nkeywal commented on HBASE-4737: I didn't know. But I believe there could be side effects on non necessary before after Class. For example, in io.hfile.TestHFileBlock# testBlockHeapSize you don't need to start a cluster, so I expect it's necessary to move it outside of the class if we want to put it in the small 'no cluster' category? On Thu, Nov 3, 2011 at 4:47 PM, Todd Lipcon (Commented) (JIRA) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Patch Available (was: Open) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Attachment: 2003_4737_pom.patch The patch on pom.xml creates 3 profiles: - nonParallelTests, the default, same behaviour as today: fork always no // - parallelTests: 3 threads within a jvm, forked once - singleJVMTests: fork once, no //. Not used today, but can be useful for tests To test on a separate bix to see if there is no side effect with the default box. At the beginning everything was going on the console without clear reason. Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143271#comment-13143271 ] Hadoop QA commented on HBASE-4737: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502170/2003_4737_pom.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestOpenedRegionHandler org.apache.hadoop.hbase.filter.TestColumnRangeFilter org.apache.hadoop.hbase.io.TestHalfStoreFileReader org.apache.hadoop.hbase.client.TestMultipleTimestamps org.apache.hadoop.hbase.mapred.TestTableInputFormat org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.master.TestCatalogJanitor org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.rest.TestStatusResource org.apache.hadoop.hbase.filter.TestFilterList org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap org.apache.hadoop.hbase.master.TestHMasterRPCException org.apache.hadoop.hbase.rest.TestMultiRowResource org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery org.apache.hadoop.hbase.mapreduce.TestTableMapReduce org.apache.hadoop.hbase.util.TestHBaseFsckComparator org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface org.apache.hadoop.hbase.util.TestMergeTable org.apache.hadoop.hbase.util.TestIncrementingEnvironmentEdge org.apache.hadoop.hbase.io.TestHbaseObjectWritable org.apache.hadoop.hbase.executor.TestExecutorService org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable org.apache.hadoop.hbase.client.TestScannerTimeout org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler org.apache.hadoop.hbase.regionserver.TestParallelPut org.apache.hadoop.hbase.regionserver.TestResettingCounters org.apache.hadoop.hbase.zookeeper.TestZKTable org.apache.hadoop.hbase.client.TestMultiParallel org.apache.hadoop.hbase.master.TestDefaultLoadBalancer org.apache.hadoop.hbase.client.TestHTablePool org.apache.hadoop.hbase.zookeeper.TestHQuorumPeer org.apache.hadoop.hbase.regionserver.TestAtomicOperation org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove org.apache.hadoop.hbase.rest.TestScannerResource org.apache.hadoop.hbase.rest.model.TestStorageClusterVersionModel org.apache.hadoop.hbase.regionserver.TestStoreFileBlockCacheSummary org.apache.hadoop.hbase.util.TestHBaseFsck org.apache.hadoop.hbase.rest.model.TestScannerModel org.apache.hadoop.hbase.regionserver.TestScanWithBloomError org.apache.hadoop.hbase.regionserver.TestColumnSeeking org.apache.hadoop.hbase.TestRegionRebalancing org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.TestFullLogReconstruction org.apache.hadoop.hbase.rest.model.TestTableRegionModel org.apache.hadoop.hbase.io.hfile.TestHFile org.apache.hadoop.hbase.zookeeper.TestZooKeeperNodeTracker org.apache.hadoop.hbase.regionserver.TestStoreFile org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort org.apache.hadoop.hbase.master.TestMasterTransitions org.apache.hadoop.hbase.zookeeper.TestZooKeeperMainServerArg org.apache.hadoop.hbase.util.TestMergeTool
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Open (was: Patch Available) Too many open files everywhere. Let's try again Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Patch Available (was: Open) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143288#comment-13143288 ] stack commented on HBASE-4737: -- I think you have to reupload the patch N. Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143290#comment-13143290 ] stack commented on HBASE-4737: -- Weird that its 'too many files' because ulimit printed out at start of test says: asf001.sp2.ygridcore.net core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 2048 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 32768 Running in Jenkins mode Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Attachment: 2003_4737_pom.patch Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Open (was: Patch Available) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Patch Available (was: Open) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143298#comment-13143298 ] stack commented on HBASE-4712: -- Why the 15 and 50 second limits. Are they fuzzy limits? If hard, why these numbers? Otherwise, excellent. Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143312#comment-13143312 ] Hadoop QA commented on HBASE-4553: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502166/4553-v12.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 48 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/155//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/155//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/155//console This message is automatically generated. The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143315#comment-13143315 ] Ted Yu commented on HBASE-4120: --- In Performance_of_Table_priority.pdf attached to this JIRA, you can see performance charts illustrating the benefits of table prioritization. From Liu Jia: After we finished the HBase_isolation_and_allocation_user_guide.pdf there were about twelve servers in the cluster, and from the screen shots in the HBase_isolation_and_allocation_user_guide.pdf you can see there are at least 700 regions on region server. Prior to that we had two engineers working on this functionality for about 1.5 month. Most of time, there are at least three projects running on this cluster. Applications can share the cluster and run concurrently. Some of the aplications like TaoBao's data cube has about 1TB data per month and another application (user behavior tracking) has about 800GB data per day but we only keep two days' data. The number of write requests is about 3 per second,with record size of about 70B. These two aplications had run on this cluster separately for at least one month. isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch, TablePriority_v8_for_trunk.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143316#comment-13143316 ] Hadoop QA commented on HBASE-4737: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502176/2003_4737_pom.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/157//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/157//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/157//console This message is automatically generated. Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143320#comment-13143320 ] nkeywal commented on HBASE-4712: For 15s, it's fuzzy to help parallelization. But there are only a few tests that don't use a cluster and last more than 15s, I could give them a try. org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter3 22.694 org.apache.hadoop.hbase.regionserver.TestMemStore 21 28.672 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6 19.568 org.apache.hadoop.hbase.util.TestIdLock 1 15.198 Same for 50, and as well the longer it last, the more chance you have to get a flaky test :-) The tests over 50s are: org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster 1 84.463 org.apache.hadoop.hbase.client.TestAdmin33 369.833 org.apache.hadoop.hbase.client.TestFromClientSide 49 154.41 org.apache.hadoop.hbase.client.TestMultipleTimestamps 8 66.497 org.apache.hadoop.hbase.client.TestShell1 69.65 org.apache.hadoop.hbase.mapred.TestTableMapReduce 1 63.846 org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat 8 152.609 org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery 3 74.628 org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan 11 516.619 org.apache.hadoop.hbase.mapreduce.TestTableMapReduce1 92.144 org.apache.hadoop.hbase.master.TestDistributedLogSplitting 4 112.606 org.apache.hadoop.hbase.master.TestMasterFailover 4 76.658 org.apache.hadoop.hbase.master.TestRollingRestart 1 54.738 org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit 28 230.379 org.apache.hadoop.hbase.regionserver.wal.TestHLog 9 59.889 org.apache.hadoop.hbase.regionserver.wal.TestLogRolling 3 310.537 org.apache.hadoop.hbase.replication.TestMasterReplication 2 75.346 org.apache.hadoop.hbase.replication.TestReplication 7 163.158 org.apache.hadoop.hbase.TestFullLogReconstruction 1 57.742 org.apache.hadoop.hbase.TestHBaseTestingUtility 7 73.654 org.apache.hadoop.hbase.TestRegionRebalancing 1 58.25 org.apache.hadoop.hbase.TestZooKeeper 7 86.267 org.apache.hadoop.hbase.util.TestMergeTool 1 257.223 Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Attachment: 4553-v12.txt The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Open (was: Patch Available) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4738) [89-fb] HBaseAdmin has ambiguous varargs invocations
[89-fb] HBaseAdmin has ambiguous varargs invocations Key: HBASE-4738 URL: https://issues.apache.org/jira/browse/HBASE-4738 Project: HBase Issue Type: Bug Components: client Environment: JDK 1.6.0_26 on OSX Reporter: Christopher Gist Priority: Minor Ambiguous invocations of varargs methods with non-varargs arguments relied on the compiler to implicitly cast the arguments to Object[]. Some compilers apparently do not make this implicit cast, but instead wrap the arguments in another Object[] causing them to be interpreted incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Patch Available (was: Open) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143323#comment-13143323 ] stack commented on HBASE-4712: -- Grand. Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143324#comment-13143324 ] stack commented on HBASE-4737: -- What happened? It ran tests for 15 minutes then gave up? Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143331#comment-13143331
]
stack commented on HBASE-4298:
--
I ran TestLogRolling locally and can't get it to fail. It failed with 'Cannot
lock storage
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/target/test-data/7f164eb7-a297-40fb-89ff-6bd9a0e331db/dfscluster_e7b1f239-5bca-46c6-abfb-f8237c09a7ce/dfs/data/data3.
The directory is already locked.'
TestDistributedLogSplitting fails with too many open files but ulimit for files
says 32k?
{code}
asf011.sp2.ygridcore.net
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2048
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
32768
Running in Jenkins mode
{code}
This is good to go I'd say.
Support to drain RS nodes through ZK
Key: HBASE-4298
URL: https://issues.apache.org/jira/browse/HBASE-4298
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.90.4
Environment: all
Reporter: Aravind Gottipati
Priority: Critical
Labels: patch
Fix For: 0.92.0, 0.90.5
Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch,
drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch
HDFS currently has a way to exclude certain datanodes and prevent them from
getting new blocks. HDFS goes one step further and even drains these nodes
for you. This enhancement is a step in that direction.
The idea is that we mark nodes in zookeeper as draining nodes. This means
that they don't get any more new regions. These draining nodes look exactly
the same as the corresponding nodes in /rs, except they live under /draining.
Eventually, support for draining them can be added. I am submitting two
patches for review - one for the 0.90 branch and one for trunk (in git).
Here are the two patches
0.90 -
https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
trunk -
https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
I have tested both these patches and they work as advertised.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143337#comment-13143337 ] nkeywal commented on HBASE-4737: surefire conflict? Is there another build on the same machine? Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143349#comment-13143349 ] Lars Hofhansl commented on HBASE-4583: -- The problem is that if the changes are not visible immediately there might be old scanners still scanning the old KVs in the memstore and hence these cannot be removed. Increment/Append/ICV currently work because all changes are visible to *all* scanners, because only then can we safely remove the duplicates. I now think that this: bq. Another option would be to remove the previous KVs after you roll rwcc forward and release the row lock, before dropping the region-level lock Actually does not work. In order to make this work we would need to know the lowest readpoint currently used by any other (read) transaction and only remove duplicate KVs *before* that readpoint. Since we do not have that information and readers also do not take out any locks, I don't see any way of making this work. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4718) Backport HBASE-4552 to 0.90 branch.
[
https://issues.apache.org/jira/browse/HBASE-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143352#comment-13143352
]
Ted Yu commented on HBASE-4718:
---
I got the following when running test suite with patch v2:
{code}
testLogRollOnPipelineRestart(org.apache.hadoop.hbase.regionserver.wal.TestLogRolling)
Time elapsed: 713.958 sec ERROR!
java.io.FileNotFoundException: File does not exist:
hdfs://localhost:54304/user/zhihyu/.logs/10.246.204.38,54320,1320336018579/10.246.204.38%3A54320.1320336018858
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:466)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:676)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1424)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1419)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:57)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:158)
at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:629)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnPipelineRestart(TestLogRolling.java:489)
{code}
In the test output, I saw:
{code}
2011-11-03 09:14:06,358 ERROR [Shutdown of
DFS[DFSClient[clientName=DFSClient_hb_rs_10.246.204.38,54321,1320336018597_1320336018852,
ugi=zhihyu.hfs.1,supergroup]]] hdfs.DFSClient$LeaseChecker(1074): Exception
closing file
/user/zhihyu/.logs/10.246.204.38,54321,1320336018597/10.246.204.38%3A54321.1320336374373
: java.io.IOException: Error Recovery for block blk_-8007982726961331998_1076
failed because recovery from primary datanode 127.0.0.1:54952 failed 6 times.
Pipeline was 127.0.0.1:54952. Aborting...
java.io.IOException: Error Recovery for block blk_-8007982726961331998_1076
failed because recovery from primary datanode 127.0.0.1:54952 failed 6 times.
Pipeline was 127.0.0.1:54952. Aborting...
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2741)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:2172)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2371)
{code}
I ran the test by itself and got same result.
Backport HBASE-4552 to 0.90 branch.
---
Key: HBASE-4718
URL: https://issues.apache.org/jira/browse/HBASE-4718
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 4718-v2.90, hbase-4718.0.90.patch,
hbase-4718.v3.includes-hbase-3316.patch
In discussion of HBASE-4552 / HBASE-4677 there has been some discussion about
whether and how to backport HBASE-4552 to the 0.90 branch. This is a
potentially compatibility breaking so several approaches hav ebeen suggested.
1) provide patch but do not integrate
2) integrate patch that extends and deprecates old api without removing old
api. It has been argued that clients are supposed to use
LoadIncrementalHFiles api and not at the internal HRegionServer RPC api.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4739) Master dying while going to close a region can leave it in transition forever
Master dying while going to close a region can leave it in transition forever
-
Key: HBASE-4739
URL: https://issues.apache.org/jira/browse/HBASE-4739
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Minor
Fix For: 0.92.0, 0.94.0, 0.90.5
I saw this in the aftermath of HBASE-4729 on a 0.92 refreshed yesterday, when
the master died it had just created the RIT znode for a region but didn't tell
the RS to close it yet.
When the master restarted it saw the znode and started printing this:
{quote}
2011-11-03 00:02:49,130 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Regions in transition timed out:
TestTable,0007560564,1320253568406.f76899564cabe7e9857c3aeb526ec9dc.
state=CLOSING, ts=1320253605285, server=sv4r11s38,62003,1320195046948
2011-11-03 00:02:49,130 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been CLOSING for too long, this should eventually complete or the
server will expire, doing nothing
{quote}
It's never going to happen, and it's blocking balancing.
I'm marking this as minor since I believe this situation is pretty rare unless
you hit other bugs while trying out stuff to root bugs out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4738) [89-fb] HBaseAdmin has ambiguous varargs invocations
[ https://issues.apache.org/jira/browse/HBASE-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-4738: --- Attachment: D231.1.patch cgist requested code review of HBASE-4738 [jira] [89-fb] HBaseAdmin has ambiguous varargs invocations. Reviewers: Karthik, JIRA Fixed HBaseAdmin ambiguous varargs invocations HBaseAdmin had ambiguous uses of varargs parameters which relied on the compiler to implicitly cast arguments to Object[] for correct operation. Certain java compilers (1.6.0_26 on OSX) were found to instead wrap the argument inside a new Object[], breaking HBaseAdmin and the HBase shell. This change avoids the ambiguous invocations by explicitly casting the arguments to Object[]. Ambiguous invocations of varargs methods with non-varargs arguments relied on the compiler to implicitly cast the arguments to Object[]. Some compilers apparently do not make this implicit cast, but instead wrap the arguments in another Object[] causing them to be interpreted incorrectly. TEST PLAN compiled HBaseAdmin with JDK 1.6.0_24 and 1.6.0_26 and tested the HBase shell with commands which take a variable number of arguments. REVISION DETAIL https://reviews.facebook.net/D231 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/441/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. [89-fb] HBaseAdmin has ambiguous varargs invocations Key: HBASE-4738 URL: https://issues.apache.org/jira/browse/HBASE-4738 Project: HBase Issue Type: Bug Components: client Environment: JDK 1.6.0_26 on OSX Reporter: Christopher Gist Priority: Minor Attachments: D231.1.patch Ambiguous invocations of varargs methods with non-varargs arguments relied on the compiler to implicitly cast the arguments to Object[]. Some compilers apparently do not make this implicit cast, but instead wrap the arguments in another Object[] causing them to be interpreted incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table
[ https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4669: -- Release Note: A new config parameter, hbase.master.enabletable.roundrobin, was introduced in this JIRA. The default value is false and old behavior is kept. When set true, regions of the table being enabled are assigned round-robin across the cluster. Add an option of using round-robin assignment for enabling table Key: HBASE-4669 URL: https://issues.apache.org/jira/browse/HBASE-4669 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4, 0.94.0 Reporter: Jieshan Bean Assignee: Jieshan Bean Priority: Minor Fix For: 0.94.0 Attachments: HBASE-4669-90-V2.patch, HBASE-4669-90.patch, HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch Under some scenarios, we use the function of disable/enable HTable. But currently, enable HTable uses the random-assignment. We hope all the regions show a better distribution, no matter how many regions and how many regionservers. So I suggest to add an option of using round-robin assignment on enable-table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Open (was: Patch Available) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Attachment: 2003_4737_pom.patch Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525 org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner 6
[jira] [Updated] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4737: --- Status: Patch Available (was: Open) one more time Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter 1 19.045 org.apache.hadoop.hbase.io.hfile.slab.TestSingleSizeCache 5 24.294 org.apache.hadoop.hbase.io.hfile.slab.TestSlabCache 7 19.818 org.apache.hadoop.hbase.io.hfile.TestHFileBlock 7 25.226 org.apache.hadoop.hbase.io.hfile.TestLruBlockCache7 0.343 org.apache.hadoop.hbase.mapreduce.TestImportTsv 8 40.391 org.apache.hadoop.hbase.master.TestActiveMasterManager2 0.724 org.apache.hadoop.hbase.master.TestHMasterRPCException1 1.17 org.apache.hadoop.hbase.master.TestLogsCleaner1 2.953 org.apache.hadoop.hbase.master.TestMaster 1 18.918 org.apache.hadoop.hbase.master.TestOpenedRegionHandler2 20.57 org.apache.hadoop.hbase.master.TestSplitLogManager10 13.979 org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion 3 21.675 org.apache.hadoop.hbase.regionserver.handler.TestOpenRegionHandler3 0.887 org.apache.hadoop.hbase.regionserver.TestBlocksRead 4 1.42 org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter 3 22.694 org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed 3 29.764 org.apache.hadoop.hbase.regionserver.TestHRegion 57 28.552 org.apache.hadoop.hbase.regionserver.TestMasterAddressManager 1 0.525
[jira] [Updated] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravind Gottipati updated HBASE-4298: - Attachment: trunk_with_test.txt Patch against trunk with Stack's test case. Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0, 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4740) [bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error.
[bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error. Key: HBASE-4740 URL: https://issues.apache.org/jira/browse/HBASE-4740 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Critical Running TestHFileOutputFormat more frequently seems to show that it has become flaky. It is difficult to tell if this is because of a unrecoverable failure or a recoverable failure. To make this visiable from test and for users, we need to make a change to bulkload call's interface on HRegionServer. The change should make successful rpcs return true, recoverable failures return false, and unrecoverable failure throw an IOException. This an RPC change so would be really good to get this api right before the final 0.92 goes out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4740) [bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error.
[ https://issues.apache.org/jira/browse/HBASE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4740: -- Description: Running TestHFileOutputFormat more frequently seems to show that it has become flaky. It is difficult to tell if this is because of a unrecoverable failure or a recoverable failure. To make this visiable from test and for users, we need to make a change to bulkload call's interface on HRegionServer. The change should make successful rpcs return true, recoverable failures return false, and unrecoverable failure throw an IOException. This is an RPC change, so it would be really good to get this api right before the final 0.92 goes out. (was: Running TestHFileOutputFormat more frequently seems to show that it has become flaky. It is difficult to tell if this is because of a unrecoverable failure or a recoverable failure. To make this visiable from test and for users, we need to make a change to bulkload call's interface on HRegionServer. The change should make successful rpcs return true, recoverable failures return false, and unrecoverable failure throw an IOException. This an RPC change so would be really good to get this api right before the final 0.92 goes out.) [bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error. Key: HBASE-4740 URL: https://issues.apache.org/jira/browse/HBASE-4740 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Critical Running TestHFileOutputFormat more frequently seems to show that it has become flaky. It is difficult to tell if this is because of a unrecoverable failure or a recoverable failure. To make this visiable from test and for users, we need to make a change to bulkload call's interface on HRegionServer. The change should make successful rpcs return true, recoverable failures return false, and unrecoverable failure throw an IOException. This is an RPC change, so it would be really good to get this api right before the final 0.92 goes out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4740) [bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error.
[ https://issues.apache.org/jira/browse/HBASE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4740: -- Fix Version/s: 0.92.0 [bulk load] the HBASE-4552 API can't tell if errors on region server is recoverable or unrecoverable error. Key: HBASE-4740 URL: https://issues.apache.org/jira/browse/HBASE-4740 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Critical Fix For: 0.92.0 Running TestHFileOutputFormat more frequently seems to show that it has become flaky. It is difficult to tell if this is because of a unrecoverable failure or a recoverable failure. To make this visiable from test and for users, we need to make a change to bulkload call's interface on HRegionServer. The change should make successful rpcs return true, recoverable failures return false, and unrecoverable failure throw an IOException. This is an RPC change, so it would be really good to get this api right before the final 0.92 goes out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Attachment: 3939-v6.txt Fix failed compile by adding VERSION to the test rpc Interface in TestDelayedRpc Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Status: Open (was: Patch Available) Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4737) Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM
[ https://issues.apache.org/jira/browse/HBASE-4737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143397#comment-13143397 ] Hadoop QA commented on HBASE-4737: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502183/2003_4737_pom.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestMasterReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/159//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/159//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/159//console This message is automatically generated. Split the tests in small/medium/large; allow small tests to be ran in parallel within a single JVM -- Key: HBASE-4737 URL: https://issues.apache.org/jira/browse/HBASE-4737 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 2003_4737_pom.patch, 2003_4737_pom.patch, 2003_4737_pom.patch 1) Split the tests in 3 categories - small: no cluster, less than 15s, can be run in parallel with other tests in a JVM - medium: 45s, no flaky, useful to detect bugs immediatly - large: remaining 2) Allow to run a subset: developpers should need to run only small and medium before submitting a patch - will need a surefire patch, see http://jira.codehaus.org/browse/SUREFIRE-329 Small is the default. All other tests will have to be marked Medium or Large with a JUnit category. Proposed split: Small:122 classes, 479 methods, ~3 minutes when no //) Medium: 78 classes, 373 methods, ~23 minutes Large: 34 classes, 221 methods, ~60 minutes I will have to extract the methods that are today in large or medium but could be in small (typically io.hfile.TestHFileBlock#testBlockHeapSize), it will be done in a second step (and another JIRA). MEDIUM LIST (name; number of methods, time) org.apache.hadoop.hbase.avro.TestAvroServer 3 31.468 org.apache.hadoop.hbase.catalog.TestCatalogTracker8 4.174 org.apache.hadoop.hbase.catalog.TestMetaReaderEditorNoCluster 1 3.888 org.apache.hadoop.hbase.catalog.TestMetaReaderEditor 5 30.157 org.apache.hadoop.hbase.client.replication.TestReplicationAdmin 1 0.762 org.apache.hadoop.hbase.client.TestHCM3 21.961 org.apache.hadoop.hbase.client.TestHTablePool 18 26.274 org.apache.hadoop.hbase.client.TestHTableUtil 2 16.997 org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD 3 24.629 org.apache.hadoop.hbase.client.TestMetaScanner1 16.365 org.apache.hadoop.hbase.client.TestMultiParallel 10 34.077 org.apache.hadoop.hbase.client.TestTimestampsFilter 3 27.547 org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol 44 16.834 org.apache.hadoop.hbase.coprocessor.TestClassLoading 5 31.346 org.apache.hadoop.hbase.coprocessor.TestCoprocessorEndpoint 2 32.736 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithAbort 1 13.874 org.apache.hadoop.hbase.coprocessor.TestMasterCoprocessorExceptionWithRemove 1 16.923 org.apache.hadoop.hbase.coprocessor.TestMasterObserver3 29.97 org.apache.hadoop.hbase.coprocessor.TestRegionObserverBypass 2 14.976 org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface 5 33.353 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort 1 16.596 org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithRemove 1 18.183 org.apache.hadoop.hbase.coprocessor.TestWALObserver 3 19.373 org.apache.hadoop.hbase.filter.TestColumnRangeFilter
[jira] [Commented] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143399#comment-13143399 ] stack commented on HBASE-3939: -- @Gary This patch ok w/ you? Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143401#comment-13143401 ] Jean-Daniel Cryans commented on HBASE-4213: --- The latest patch doesn't apply to 0.92, would it be possible to have another version for it? Thanks. Also looking over the patch names, it would sure be nice if they followed some sort of convention. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-4741:
-
Assignee: Ted Yu
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143410#comment-13143410
]
Jean-Daniel Cryans commented on HBASE-4741:
---
Instead of throwing an IOE a TableNotFoundException should be used.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3939) Some crossports of Hadoop IPC fixes
[
https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143415#comment-13143415
]
stack commented on HBASE-3939:
--
We can't get to the new rpc message because 0.90.x clients don't get that far;
they fail reading zookeeper master entry:
{code}
hbase(main):002:0 debug
Debug mode is ON
hbase(main):003:0 list
TABLE
ERROR: java.lang.IllegalArgumentException: Not a host:port pair:
�12...@h-24-30.sfo.stumble.neth-24-30.sfo.stumble.net,60637,1320344107987
Backtrace: org/apache/hadoop/hbase/HServerAddress.java:60:in `init'
org/apache/hadoop/hbase/MasterAddressTracker.java:63:in
`getMasterAddress'
org/apache/hadoop/hbase/client/HConnectionManager.java:354:in
`getMaster'
org/apache/hadoop/hbase/client/HBaseAdmin.java:94:in `init'
{code}
ServerName is in the way of our getting to rpc.
I tried making an HBaseAdmin instance but that don't work because on
construction it tries to talk to Master (which is dumb):
{code}
hbase(main):008:0 a = HBaseAdmin.new(c)
NativeException: java.lang.IllegalArgumentException: Not a host:port pair:
�12...@h-24-30.sfo.stumble.neth-24-30.sfo.stumble.net,60637,1320344107987
{code}
Can't create an HTable.
Reverting the ServerName change is not an option.
Some crossports of Hadoop IPC fixes
---
Key: HBASE-3939
URL: https://issues.apache.org/jira/browse/HBASE-3939
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.92.0
Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt,
3939-v6.txt, 3939.txt
A few fixes from Hadoop IPC that we should probably cross-port into our copy:
- HADOOP-7227: remove the protocol version check at call time
- HADOOP-7146: fix a socket leak in server
- HADOOP-7121: fix behavior when response serialization throws an exception
- HADOOP-7346: send back nicer error response when client is using an out of
date IPC version
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4741:
--
Attachment: 4741-v2.txt
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4298: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch and trunk. Thanks for the patch Aravind. This is a new feature that has been running for months now at my place of employ. Our ops like it (and dev'd it). Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0, 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reopened HBASE-4298: -- Reopening. It was applied to 0.92 but we might want to apply to 0.90... moving it over there. Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0, 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143431#comment-13143431
]
Jean-Daniel Cryans commented on HBASE-4741:
---
Looks better, actually you could use checkTableModifiable, see how
TableEventHandler uses it too. Also if you keep the same log message please fix
the typo in HTableDescritor.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143440#comment-13143440
]
Ted Yu commented on HBASE-4741:
---
HBA uses this.connection.getMaster() to get to master.
However there is no getter for MasterServices in HConnection interface.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143468#comment-13143468 ] Hadoop QA commented on HBASE-3939: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502190/3939-v6.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/161//console This message is automatically generated. Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Attachment: 4213-0.92.txt Patch 4213-0.92.txt is rebased for 0.92 branch. HMaster.waitForCurrentSchemaChange() has been removed. TestInstantSchemaChangeFailover and TestInstantSchemaChange passed. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-0.92.txt, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143479#comment-13143479 ] Hadoop QA commented on HBASE-4298: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502184/trunk_with_test.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 46 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/160//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/160//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/160//console This message is automatically generated. Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4726) RS should close region if it fails to mark it as 'OPENED'.
[ https://issues.apache.org/jira/browse/HBASE-4726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143492#comment-13143492 ] Madhuwanti Vaidya commented on HBASE-4726: -- This is the scenario we saw: 1. Region r is opened on RS1 and ZNode is updated with RS2ZK_REGION_OPENED. 2. But for some reason update to META takes a long time. (Not sure why yet) 3. hbck reports r is still unassigned - since update to META has not happened yet. 4. hbck -fix is run. 5. Now master tries to assign r to RS2. The following two happen simultaneously: 6a. Master now processes the old RS2ZK_REGION_OPENED update from RS1. This updates META and clears ZK. 6a. RS2 onlines the r - but when it tries to set state to RS2ZK_REGION_OPENED, it gets a ZK NoNodeException error. RS should close region if it fails to mark it as 'OPENED'. -- Key: HBASE-4726 URL: https://issues.apache.org/jira/browse/HBASE-4726 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.89.20100924 Reporter: Madhuwanti Vaidya Assignee: Madhuwanti Vaidya Priority: Minor Currently if a RS fails to mark a region as 'OPENED' it only logs an error. It will leave the region open - this has caused duplicate region assignments in one of our production clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4741:
--
Attachment: 4741-v3.txt
Added check for more operations.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741-v3.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4742) Split dead server's log in parallel
Split dead server's log in parallel --- Key: HBASE-4742 URL: https://issues.apache.org/jira/browse/HBASE-4742 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang When one region server goes down, the master will shutdown the region server and split its log. However, splitting log is a blocking call and it would take some time. If more than one region server go down, the master will split its log one by one, which is not efficient. Since we have the distributed log split, we could split these logs from the dead servers in parallel. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4743) [book] more information about scan timeouts
[book] more information about scan timeouts Key: HBASE-4743 URL: https://issues.apache.org/jira/browse/HBASE-4743 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * fixed typo in schema design section (missing a comma in the CF cardinality section) performance.xml * added more information about scan timeouts and per-row processing costs troubleshooting * added link from scantimeout entry to the performance section on the caching section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143538#comment-13143538 ] Hudson commented on HBASE-4298: --- Integrated in HBase-TRUNK #2407 (See [https://builds.apache.org/job/HBase-TRUNK/2407/]) HBASE-4298 Support to drain RS nodes through ZK stack : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/DrainingServerTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4743) [book] more information about scan timeouts
[ https://issues.apache.org/jira/browse/HBASE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4743: - Attachment: docbkx_HBASE_4743.patch [book] more information about scan timeouts Key: HBASE-4743 URL: https://issues.apache.org/jira/browse/HBASE-4743 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4743.patch book.xml * fixed typo in schema design section (missing a comma in the CF cardinality section) performance.xml * added more information about scan timeouts and per-row processing costs troubleshooting * added link from scantimeout entry to the performance section on the caching section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4743) [book] more information about scan timeouts
[ https://issues.apache.org/jira/browse/HBASE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4743: - Status: Patch Available (was: Open) [book] more information about scan timeouts Key: HBASE-4743 URL: https://issues.apache.org/jira/browse/HBASE-4743 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4743.patch book.xml * fixed typo in schema design section (missing a comma in the CF cardinality section) performance.xml * added more information about scan timeouts and per-row processing costs troubleshooting * added link from scantimeout entry to the performance section on the caching section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4743) [book] more information about scan timeouts
[ https://issues.apache.org/jira/browse/HBASE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-4743: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] more information about scan timeouts Key: HBASE-4743 URL: https://issues.apache.org/jira/browse/HBASE-4743 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_HBASE_4743.patch book.xml * fixed typo in schema design section (missing a comma in the CF cardinality section) performance.xml * added more information about scan timeouts and per-row processing costs troubleshooting * added link from scantimeout entry to the performance section on the caching section -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143546#comment-13143546 ] Hadoop QA commented on HBASE-4213: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502207/4213-0.92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -161 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 52 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.thrift2.TestThriftHBaseServiceHandler org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.client.TestInstantSchemaChangeFailover org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/162//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/162//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/162//console This message is automatically generated. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: 4213-0.92.txt, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143556#comment-13143556
]
Hadoop QA commented on HBASE-4741:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12502212/4741-v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 javadoc. The javadoc tool appears to have generated -164 warning
messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 46 new Findbugs (version
1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.master.TestMasterFailover
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/163//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/163//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/163//console
This message is automatically generated.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741-v3.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Open (was: Patch Available) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Patch Available (was: Open) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Attachment: 4553-v12.txt The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4738) [89-fb] HBaseAdmin has ambiguous varargs invocations
[ https://issues.apache.org/jira/browse/HBASE-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143569#comment-13143569 ] Phabricator commented on HBASE-4738: Karthik has accepted the revision HBASE-4738 [jira] [89-fb] HBaseAdmin has ambiguous varargs invocations. REVISION DETAIL https://reviews.facebook.net/D231 [89-fb] HBaseAdmin has ambiguous varargs invocations Key: HBASE-4738 URL: https://issues.apache.org/jira/browse/HBASE-4738 Project: HBase Issue Type: Bug Components: client Environment: JDK 1.6.0_26 on OSX Reporter: Christopher Gist Priority: Minor Attachments: D231.1.patch Ambiguous invocations of varargs methods with non-varargs arguments relied on the compiler to implicitly cast the arguments to Object[]. Some compilers apparently do not make this implicit cast, but instead wrap the arguments in another Object[] causing them to be interpreted incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4738) [89-fb] HBaseAdmin has ambiguous varargs invocations
[ https://issues.apache.org/jira/browse/HBASE-4738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143568#comment-13143568 ] Phabricator commented on HBASE-4738: Karthik has added CCs to the revision HBASE-4738 [jira] [89-fb] HBaseAdmin has ambiguous varargs invocations. Added CCs: stack Reviewed internally also. REVISION DETAIL https://reviews.facebook.net/D231 [89-fb] HBaseAdmin has ambiguous varargs invocations Key: HBASE-4738 URL: https://issues.apache.org/jira/browse/HBASE-4738 Project: HBase Issue Type: Bug Components: client Environment: JDK 1.6.0_26 on OSX Reporter: Christopher Gist Priority: Minor Attachments: D231.1.patch Ambiguous invocations of varargs methods with non-varargs arguments relied on the compiler to implicitly cast the arguments to Object[]. Some compilers apparently do not make this implicit cast, but instead wrap the arguments in another Object[] causing them to be interpreted incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143572#comment-13143572 ] Hadoop QA commented on HBASE-4553: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502227/4553-v12.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -164 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 48 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/164//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/164//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/164//console This message is automatically generated. The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Attachment: 3939-v7.txt Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939-v7.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Status: Patch Available (was: Open) Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939-v7.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Status: Open (was: Patch Available) Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Priority: Critical Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt, 3939-v6.txt, 3939-v7.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4741) Online schema change doesn't return errors
[
https://issues.apache.org/jira/browse/HBASE-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143579#comment-13143579
]
Jean-Daniel Cryans commented on HBASE-4741:
---
bq. HBA uses this.connection.getMaster() to get to master.
Ha! I was somehow reading that you were doing the changes in HMaster, my bad.
In this case checkTableExistence isn't needed because getTableDescriptor does
what's needed (eg it will throw the TableNotFoundException).
Reading the code I see stuff like:
{code}
public void modifyColumn(final byte [] tableName, HColumnDescriptor
descriptor)
throws IOException {
try {
getMaster().modifyColumn(tableName, descriptor);
} catch (RemoteException re) {
// Convert RE exceptions in here; client shouldn't have to deal with them,
// at least w/ the type of exceptions that come out of this method:
// TableNotFoundException, etc.
throw RemoteExceptionHandler.decodeRemoteException(re);
}
}
{code}
So it seems that it used to be that the checks were done in the master. I would
prefer to see the checks done over there since other clients (like asynchbase)
would also need to implement the same checks.
Back to the patch, it's also missing addColumn.
Online schema change doesn't return errors
--
Key: HBASE-4741
URL: https://issues.apache.org/jira/browse/HBASE-4741
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Jean-Daniel Cryans
Assignee: Ted Yu
Priority: Critical
Fix For: 0.92.0
Attachments: 4741-v2.txt, 4741-v3.txt, 4741.txt
Still after the fun I had over in HBASE-4729, I tried to finish altering my
table (remove a family) since only half of it was changed so I did this:
{quote}
hbase(main):002:0 alter 'TestTable', NAME = 'allo', METHOD = 'delete'
Updating all regions with the new schema...
244/244 regions updated.
Done.
0 row(s) in 1.2480 seconds
{quote}
Nice it all looks good, but over in the master log:
{quote}
org.apache.hadoop.hbase.InvalidFamilyOperationException: Family 'allo' does
not exist so cannot be deleted
at
org.apache.hadoop.hbase.master.handler.TableDeleteFamilyHandler.handleTableOperation(TableDeleteFamilyHandler.java:56)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:86)
at
org.apache.hadoop.hbase.master.HMaster.deleteColumn(HMaster.java:1011)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:348)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1242)
{quote}
Maybe we should do checks before launching the async task.
Marking critical as this is a regression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143585#comment-13143585 ] Hudson commented on HBASE-4298: --- Integrated in HBase-0.92 #105 (See [https://builds.apache.org/job/HBase-0.92/105/]) HBASE-4298 Support to drain RS nodes through ZK stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/DrainingServerTracker.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestDrainingServer.java Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.90.5 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3939) Some crossports of Hadoop IPC fixes
[
https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143591#comment-13143591
]
Gary Helmling commented on HBASE-3939:
--
Would be easier to review if the latest patch was in review board.
Here are comments from what I see:
In {{HBaseServer}}:
# In {{setupResponse()}} we take a {{Status}} instance (which is good), but it
doesn't get passed back through to the client. If we're modifying the wire
protocol to pass RPC version, I think we should also take advantage of the
change to serialize {{Status}} back to the client. This allows the client to
differentiate between fatal errors (authentication error, or bad rpc version)
which should kill the connection, and request-specific errors. By the way,
adding {{Status}} here allows me to remove it from the HBASE-2742 patch, which
is great!
In {{Invocation}}:
# In the constructor:
{code}
rpcVersion = WritableRpcEngine.writableRpcVersion;
{code}
This binds {{Invocation}} unnecessarily to {{WritableRpcEngine}}. We've
diverged a little from Hadoop RPC by making {{Invocation}} a top-level class,
allowing it to be shared between RPC engine implementations, so this would
undermine that. Since {{rpcVersion}} only seems to relate to {{Invocation}}
serialization, why not just define {{RPC_VERSION}} as a static final constant
on {{Invocation}}? Or alternately, couldn't we just make {{Invocation}}
implement {{VersionedWritable}} and let that handle the check for us?
Other than that, this patch looks fine to me. I've applied it together with
the HBASE-2742 patch and run through a few of the tests, and so far seems to
work fine.
Some crossports of Hadoop IPC fixes
---
Key: HBASE-3939
URL: https://issues.apache.org/jira/browse/HBASE-3939
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.92.0
Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939-v5.txt,
3939-v6.txt, 3939-v7.txt, 3939.txt
A few fixes from Hadoop IPC that we should probably cross-port into our copy:
- HADOOP-7227: remove the protocol version check at call time
- HADOOP-7146: fix a socket leak in server
- HADOOP-7121: fix behavior when response serialization throws an exception
- HADOOP-7346: send back nicer error response when client is using an out of
date IPC version
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4744) Remove @Ignore for testLogRollAfterSplitStart
Remove @Ignore for testLogRollAfterSplitStart - Key: HBASE-4744 URL: https://issues.apache.org/jira/browse/HBASE-4744 Project: HBase Issue Type: Test Affects Versions: 0.94.0 Reporter: Nicolas Spiegelberg Priority: Critical We fixed a data loss bug in HBASE-2312 by adding non-recursive creates to HDFS. Although a number of HDFS versions have this fix, the official HDFS 0.20.205 branch currently doesn't, so we needed to mark the test as ignored. Please revisit before the RC of 0.94, which should have 0.20.205.1 or later the necessary HDFS patches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4674) splitLog silently fails
[ https://issues.apache.org/jira/browse/HBASE-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg resolved HBASE-4674. Resolution: Duplicate Fix Version/s: 0.92.0 Fixed by HBASE-2312 splitLog silently fails --- Key: HBASE-4674 URL: https://issues.apache.org/jira/browse/HBASE-4674 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Environment: splitLog() can fail silently and region can open w/o its edits getting replayed. Reporter: Prakash Khemani Assignee: Prakash Khemani Priority: Blocker Fix For: 0.92.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Patch Available (was: Open) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4553) The update of .tableinfo is not atomic; we remove then rename
[ https://issues.apache.org/jira/browse/HBASE-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4553: - Status: Open (was: Patch Available) The update of .tableinfo is not atomic; we remove then rename - Key: HBASE-4553 URL: https://issues.apache.org/jira/browse/HBASE-4553 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 3446-v8.txt, 4553-v10.txt, 4553-v11.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v12.txt, 4553-v5.txt, 4553-v9.txt, HBase-4553-TestAvroServer.patch This comes of HBASE-4547. The rename in 0.20 hdfs fails if file exists already. In 0.20+ its better but still 'some' issues if existing reader when file is renamed. This issue is about fixing this (though we depend on fix first being in hdfs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
