date:20130206


[ 
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572871#comment-13572871
 ] 

Himanshu Vashishtha commented on HBASE-7723:


looking at it.

 Remove NN URI from ZK splitlogs.
 

 Key: HBASE-7723
 URL: https://issues.apache.org/jira/browse/HBASE-7723
 Project: HBase
  Issue Type: Bug
  Components: hadoop2, master
Affects Versions: 0.92.0
Reporter: Kevin Odell
Assignee: Himanshu Vashishtha
 Fix For: 0.96.0, 0.94.5

 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, 
 HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, 
 HBASE-7723-94-v4.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch


 When moving to HDFS HA or removing HA we end up changing the NN namespace.  
 This can cause the HMaster not to start up fully due to trying to split 
 phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: 
 Wrong FS: error messages.  The HLogs in question might not even be on HDFS 
 anymore.  You have to go in a manually clear out the ZK splitlogs directory 
 to get HBase to properly boot up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572890#comment-13572890
 ] 

Ted Yu commented on HBASE-7778:
---

After reverting the changes (locally) to JVMClusterUtil.java from HBASE-7475, I 
got the following tests pass:

  904  mt 
-Dtest=TestReplicationDisabledinactivePeer,TestRestartCluster,TestCatalogTrackerOnCluster
  905  mt -Dtest=TestUpgradeFromHFileV1ToEncoding

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572895#comment-13572895
]

stack commented on HBASE-7723:
--

So, when HA NN, the filesystem scheme and host/port are different? You have an
example please Himanshu? I'd be surprised if splitlogs is only place we don't
have the assumption that different scheme or host or port is not a different
filesystem altogether. Good on you.

Remove NN URI from ZK splitlogs.

Key: HBASE-7723
URL: https://issues.apache.org/jira/browse/HBASE-7723
Project: HBase
Issue Type: Bug
Components: hadoop2, master
Affects Versions: 0.92.0
Reporter: Kevin Odell
Assignee: Himanshu Vashishtha
Fix For: 0.96.0, 0.94.5

Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt,
HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch,
HBASE-7723-94-v4.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch

When moving to HDFS HA or removing HA we end up changing the NN namespace.
This can cause the HMaster not to start up fully due to trying to split
phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException:
Wrong FS: error messages. The HLogs in question might not even be on HDFS
anymore. You have to go in a manually clear out the ZK splitlogs directory
to get HBase to properly boot up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7768) zkcluster in local mode not seeing configurations in hbase-{site|default}.xml

2013-02-06 Thread Enis Soztutar (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-7768:
-

   Resolution: Fixed
Fix Version/s: (was: 0.94.6)
   (was: 0.92.3)
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed this to trunk. Thanks. 

 zkcluster in local mode not seeing configurations in hbase-{site|default}.xml
 -

 Key: HBASE-7768
 URL: https://issues.apache.org/jira/browse/HBASE-7768
 Project: HBase
  Issue Type: Bug
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7768_2.patch, HBASE-7768.patch


 in case of local mode we are creating mini zk cluster with default constructor
 {code}
   if (LocalHBaseCluster.isLocal(conf)) {
 final MiniZooKeeperCluster zooKeeperCluster =
   new MiniZooKeeperCluster();
 {code}
 {code}
   public MiniZooKeeperCluster() {
 this(new Configuration());
   }
 {code}
 which is not reading any configurations in hbase-site|default.xml
 I think we can pass configuration object to MiniZooKeeperCluster

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7678) make storefile management pluggable, together with compaction


 [ 
https://issues.apache.org/jira/browse/HBASE-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7678:


Attachment: (was: HBASE-7678-with-7604-and-7773.patch)

 make storefile management pluggable, together with compaction
 -

 Key: HBASE-7678
 URL: https://issues.apache.org/jira/browse/HBASE-7678
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HBASE-7678-v0.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7287) HBase inconsistencies when merge failing on Files have same sequenceid

2013-02-06 Thread Jean-Marc Spaggiari (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Marc Spaggiari updated HBASE-7287:
---

Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

Since HBASE-1212 will remove this part of the code, there is no more need to 
have this fixed.

 HBase inconsistencies when merge failing on Files have same sequenceid
 

 Key: HBASE-7287
 URL: https://issues.apache.org/jira/browse/HBASE-7287
 Project: HBase
  Issue Type: Bug
Reporter: Jean-Marc Spaggiari
Assignee: Jean-Marc Spaggiari
 Attachments: HBASE-7287.patch, HBASE-7287-v2.patch, 
 HBASE-7287-v3-trunk.patch, HBASE-7287-v4-trunk.patch

   Original Estimate: 1h
  Remaining Estimate: 1h

 When regions merge is failing because the files have the same sequenceID, the 
 expected region is still created even if it's not used, which leaves the 
 system with inconsistencies. The new region creation should be moved after 
 the sequenceID test to avoid this issue, until we find a way to merge regions 
 with the same sequenceID.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-7597) testRegionShouldNotBeDeployed seems to be flaky

2013-02-06 Thread Jean-Marc Spaggiari (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Marc Spaggiari resolved HBASE-7597.


Resolution: Cannot Reproduce

I haven't seen this test fail for a while now, so this can be closed.

 testRegionShouldNotBeDeployed seems to be flaky
 ---

 Key: HBASE-7597
 URL: https://issues.apache.org/jira/browse/HBASE-7597
 Project: HBase
  Issue Type: Bug
Reporter: Jean-Marc Spaggiari

 I ran the entire test suite many times and always failed on, at least, 
 testRegionShouldNotBeDeployed.
 Results below. I will attached more result when current tests are done.
 Failed tests:
 testDeleteExpiredStoreFiles(org.apache.hadoop.hbase.regionserver.TestStore):
 expected:2 but was:4
   
 testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker):
 Waiting timed out after [1 000] msec
   testRegionShouldNotBeDeployed(org.apache.hadoop.hbase.util.TestHBaseFsck):
 expected:[SHOULD_NOT_BE_DEPLOYED] but was:[]
   
 testPermissionsWatcher(org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572902#comment-13572902
 ] 

Ted Yu commented on HBASE-7778:
---

The following one line change allows the above tests to pass:
{code}
if (!noWait){
  // Thread.currentThread().interrupt();
}
{code}

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7678) make storefile management pluggable, together with compaction


 [ 
https://issues.apache.org/jira/browse/HBASE-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7678:


Attachment: HBASE-7678--and-7603.patch
HBASE-7678-v1.patch

javadoc warnings were actually in existing code. Rebased, fixed. Let me create 
an /r/

 make storefile management pluggable, together with compaction
 -

 Key: HBASE-7678
 URL: https://issues.apache.org/jira/browse/HBASE-7678
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HBASE-7678--and-7603.patch, HBASE-7678-v0.patch, 
 HBASE-7678-v1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7741) Don't use bulk assigner if assigning just several regions


[ 
https://issues.apache.org/jira/browse/HBASE-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572904#comment-13572904
 ] 

stack commented on HBASE-7741:
--

[~jxiang] Fancy.  Patch looks good to me.  One question, you add the flag 
waitTillAllAssigned... I see it being checked.  I don't see it being set when 
bulk assigning is done.  Am I missing something? (Is there a danger we'll wait 
for ever on bulk assign?)

 Don't use bulk assigner if assigning just several regions
 -

 Key: HBASE-7741
 URL: https://issues.apache.org/jira/browse/HBASE-7741
 Project: HBase
  Issue Type: Improvement
  Components: Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-7741.patch, trunk-7741_v2.patch, 
 trunk-7741_v3_1.patch, trunk-7741_v3.patch


 If just assign one region, bulk assigner may be slower.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7678) make storefile management pluggable, together with compaction


[ 
https://issues.apache.org/jira/browse/HBASE-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572909#comment-13572909
 ] 

Jimmy Xiang commented on HBASE-7678:


This is a big change. Do we have a design doc? It will be great if you can 
outline the purpose of the change, some design choices, the relationship among 
those entities such as StoreEngine, StoreFileManager, Compactor, 
CompactionPolicy, HStore, etc., how this change will be used for stripe/level 
compaction, and so on. Good stuff.

 make storefile management pluggable, together with compaction
 -

 Key: HBASE-7678
 URL: https://issues.apache.org/jira/browse/HBASE-7678
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HBASE-7678--and-7603.patch, HBASE-7678-v0.patch, 
 HBASE-7678-v1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7597) testRegionShouldNotBeDeployed seems to be flaky


[ 
https://issues.apache.org/jira/browse/HBASE-7597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572910#comment-13572910
 ] 

stack commented on HBASE-7597:
--

[~jmspaggi] Thanks for he housekeeping.

 testRegionShouldNotBeDeployed seems to be flaky
 ---

 Key: HBASE-7597
 URL: https://issues.apache.org/jira/browse/HBASE-7597
 Project: HBase
  Issue Type: Bug
Reporter: Jean-Marc Spaggiari

 I ran the entire test suite many times and always failed on, at least, 
 testRegionShouldNotBeDeployed.
 Results below. I will attached more result when current tests are done.
 Failed tests:
 testDeleteExpiredStoreFiles(org.apache.hadoop.hbase.regionserver.TestStore):
 expected:2 but was:4
   
 testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker):
 Waiting timed out after [1 000] msec
   testRegionShouldNotBeDeployed(org.apache.hadoop.hbase.util.TestHBaseFsck):
 expected:[SHOULD_NOT_BE_DEPLOYED] but was:[]
   
 testPermissionsWatcher(org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572912#comment-13572912
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

Yup, I've essentially did that as well -- I concerned that if this will 
re-break HBASE-7475.  [~nkeywal]  -- any comments on this?

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572915#comment-13572915
]

Himanshu Vashishtha commented on HBASE-7723:

Yes, with NN HA:
{code}
namehbase.rootdir/name
valuehdfs://cloudone/hbase/92/value
{code}

with non-HA
{code}
namehbase.rootdir/name
valuehdfs://cloudone:1234/hbase/94/value
{code}

When we change the filesystem, we need to restart the cluster after making
changes in the hbase-site.xml file.
But, we shouldn't wipe the zk data as part of the upgrade. If the shutdown is
not proper, we may have some splitlog znodes, which currently stores the
absolute path. When new configuration comes in, the old path becomes obsolete.
Therefore, this jira tends to change the read behavior for SplitLog entities
for 94 (we may have old znodes written by old master and we need to support
rolling upgrade); and change write+read behavior to avoid writing the NN URI
for trunk.

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572918#comment-13572918
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

I'm more concerned about these tests being racy and flakey due to the noWait 
variable in the JVMClusterUtil#shutdown method.  

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7475) TestUpgradeFromHFileV1ToEncoding.testUpgrade hangs


 [ 
https://issues.apache.org/jira/browse/HBASE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-7475:
--

Attachment: 7475.addendum

Proposed addendum.

There is no need to interrupt current thread at the end of cluster shutdown.

 TestUpgradeFromHFileV1ToEncoding.testUpgrade hangs
 --

 Key: HBASE-7475
 URL: https://issues.apache.org/jira/browse/HBASE-7475
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 7475.addendum, 7475.v1.patch


 I'm having a look. Here is the stack I got locally:
 {noformat}
 pool-1-thread-1 prio=10 tid=0x7f27c8406000 nid=0xf908 in Object.wait() 
 [0x7f27cec5b000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xe88b19b0 (a 
 org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
 at java.lang.Thread.join(Thread.java:1186)
 - locked 0xe88b19b0 (a 
 org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
 at java.lang.Thread.join(Thread.java:1239)
 at 
 org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245)
 at 
 org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:501)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:856)
 at 
 org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
 {noformat}
 {noformat}
 RegionServer:0;localhost,35592,1357148534219-splits-1357148554209 daemon 
 prio=10 tid=0x40ed1000 nid=0x1178 waiting on condition 
 [0x7f27b3d3c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1164)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:966)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:919)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:246)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:187)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:198)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:224)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor.offlineParentInMeta(MetaEditor.java:229)
 at 
 org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:341)
 at 
 org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:471)
 at 
 org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:68)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 {noformat}
 RegionServer:0;localhost,35592,1357148534219 prio=10 tid=0x7f27c0929000 
 nid=0x5a7 waiting on condition [0x7f27be5e3000]
java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xe88b1978 (a 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at 
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
 at 
 java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1253)
 at 
 org.apache.hadoop.hbase.regionserver.CompactSplitThread.waitFor(CompactSplitThread.java:252)
 at 
 org.apache.hadoop.hbase.regionserver.CompactSplitThread.join(CompactSplitThread.java:261)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:948)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135)
 {noformat}

--
This

[jira] [Commented] (HBASE-7678) make storefile management pluggable, together with compaction


[ 
https://issues.apache.org/jira/browse/HBASE-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572928#comment-13572928
 ] 

Sergey Shelukhin commented on HBASE-7678:
-

preliminary /r/ created at https://reviews.apache.org/r/9340/
I will be busy with something else most of the day today, probably; will create 
a writeup tomorrow.
There's some pre-writeup in the txt attached to HBASE-7519.

 make storefile management pluggable, together with compaction
 -

 Key: HBASE-7678
 URL: https://issues.apache.org/jira/browse/HBASE-7678
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HBASE-7678--and-7603.patch, HBASE-7678-v0.patch, 
 HBASE-7678-v1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572929#comment-13572929
]

Himanshu Vashishtha commented on HBASE-7723:

Lars, will your patch work in case the change is like the above example? looks
like toUri() get rid of scheme and host, but what if we change the directory
level too? Is it worth considering?

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7741) Don't use bulk assigner if assigning just several regions


[ 
https://issues.apache.org/jira/browse/HBASE-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572942#comment-13572942
 ] 

Jimmy Xiang commented on HBASE-7741:


It is read from configuration and passed to bulk assigner from AM.  It won't 
wait for ever if waitTillAllAssigned is set to true.  It will time out in some 
time if some regions stuck in transition. Since nobody depends on the regions 
to be assigned, I think it is better to let the bulk assigner run and not to 
wait for it. This will speed up the starts up a little. If there are several 
region servers crash at the same time, this can free up the SSH thread a little 
sooner too. 

 Don't use bulk assigner if assigning just several regions
 -

 Key: HBASE-7741
 URL: https://issues.apache.org/jira/browse/HBASE-7741
 Project: HBase
  Issue Type: Improvement
  Components: Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: trunk-7741.patch, trunk-7741_v2.patch, 
 trunk-7741_v3_1.patch, trunk-7741_v3.patch


 If just assign one region, bulk assigner may be slower.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.

2013-02-06 Thread Elliott Clark (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572946#comment-13572946
 ] 

Elliott Clark commented on HBASE-7763:
--

bq.Although maybe this is not a big deal to change, maybe someone else can 
comment.
Bulk loaded files break this contract pretty badly.  (So maybe size based 
sorting should be used only for stores containing bulk loaded files)  But it 
does seem like something that would need to be discussed first.

bq.Any particular reason to run two iterations of selection?
To have behave something more like a recursive en queued compaction.



 Compactions not sorting based on size anymore.
 --

 Key: HBASE-7763
 URL: https://issues.apache.org/jira/browse/HBASE-7763
 Project: HBase
  Issue Type: Bug
  Components: Compaction
Affects Versions: 0.96.0, 0.94.4
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Critical
 Fix For: 0.96.0, 0.94.6

 Attachments: HBASE-7763-trunk-TESTING.patch, 
 HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch


 Currently compaction selection is not sorting based on size.  This causes 
 selection to choose larger files to re-write than are needed when bulk loads 
 are involved.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7475) TestUpgradeFromHFileV1ToEncoding.testUpgrade hangs


[ 
https://issues.apache.org/jira/browse/HBASE-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572947#comment-13572947
 ] 

Jonathan Hsieh commented on HBASE-7475:
---

Its been a month so it seems that an addendum is not the right way to go here. 
I've linked this issue to HBASE-7778 -- let's just keep the discussion on these 
newly found problems over there.

 TestUpgradeFromHFileV1ToEncoding.testUpgrade hangs
 --

 Key: HBASE-7475
 URL: https://issues.apache.org/jira/browse/HBASE-7475
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.96.0
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
 Fix For: 0.96.0

 Attachments: 7475.addendum, 7475.v1.patch


 I'm having a look. Here is the stack I got locally:
 {noformat}
 pool-1-thread-1 prio=10 tid=0x7f27c8406000 nid=0xf908 in Object.wait() 
 [0x7f27cec5b000]
java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on 0xe88b19b0 (a 
 org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
 at java.lang.Thread.join(Thread.java:1186)
 - locked 0xe88b19b0 (a 
 org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
 at java.lang.Thread.join(Thread.java:1239)
 at 
 org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:245)
 at 
 org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:501)
 at 
 org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:856)
 at 
 org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
 {noformat}
 {noformat}
 RegionServer:0;localhost,35592,1357148534219-splits-1357148554209 daemon 
 prio=10 tid=0x40ed1000 nid=0x1178 waiting on condition 
 [0x7f27b3d3c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1164)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:966)
 at 
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:919)
 at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:246)
 at org.apache.hadoop.hbase.client.HTable.init(HTable.java:187)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.getHTable(MetaReader.java:198)
 at 
 org.apache.hadoop.hbase.catalog.MetaReader.getMetaHTable(MetaReader.java:224)
 at 
 org.apache.hadoop.hbase.catalog.MetaEditor.offlineParentInMeta(MetaEditor.java:229)
 at 
 org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:341)
 at 
 org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:471)
 at 
 org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:68)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:662)
 {noformat}
 {noformat}
 RegionServer:0;localhost,35592,1357148534219 prio=10 tid=0x7f27c0929000 
 nid=0x5a7 waiting on condition [0x7f27be5e3000]
java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xe88b1978 (a 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at 
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
 at 
 java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1253)
 at 
 org.apache.hadoop.hbase.regionserver.CompactSplitThread.waitFor(CompactSplitThread.java:252)
 at 
 org.apache.hadoop.hbase.regionserver.CompactSplitThread.join(CompactSplitThread.java:261)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:948)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151)
 at 
 org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103)

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572956#comment-13572956
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

It looks like there are two threads that are waiting/joining on the minicluster 
threads.  It seems there is a race -- on one path, it seems that 
LocalHBaseCluster#join() gets and likely clears the interrupt flag by catching 
an interrupted exception near the call to Threads#threaddumpingIsAlive.  The 
other path, the interrupt state set in the link ted commented out triggers the 
exception on the following Thread#sleep call.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called


[ 
https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572966#comment-13572966
 ] 

Hudson commented on HBASE-7772:
---

Integrated in HBase-0.94 #830 (See 
[https://builds.apache.org/job/HBase-0.94/830/])
HBASE-7772 Cluster ID is not initialized correctly in RPC client (Revision 
1443181)

 Result = FAILURE
garyh : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java


 clusterId is not set in conf properly if only 
 TableMapReduceUtil.initCredentials() is called
 

 Key: HBASE-7772
 URL: https://issues.apache.org/jira/browse/HBASE-7772
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
 Fix For: 0.96.0, 0.94.5

 Attachments: HBASE-7772_94.patch, HBASE-7772.patch, 
 HBASE-7772-v2_0.94.patch, HBASE-7772-v2.patch


 clusterId is not set in the job's conf correctly if only 
 TableMapReduceUtil.initCredentials() is called and thus fails to include the 
 token when using an hbase client in any of the job's tasks. This is a 
 regression.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding


[ 
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572967#comment-13572967
 ] 

Ted Yu commented on HBASE-4676:
---

A diff of the findbugs warnings shows the following:
{code}
tyu$ diff 4348-newPatchFindbugsWarningshbase-server.xml 
4354-newPatchFindbugsWarningshbase-server.xml 
...
1196a1197,1205
   BugInstance type=REC_CATCH_EXCEPTION priority=2 abbrev=REC 
 category=STYLE
 Class classname=org.apache.hadoop.hbase.thrift2.ThriftServer
   SourceLine classname=org.apache.hadoop.hbase.thrift2.ThriftServer 
 start=72 end=290 sourcefile=ThriftServer.java 
 sourcepath=org/apache/hadoop/hbase/thrift2/ThriftServer.java/
 /Class
 Method classname=org.apache.hadoop.hbase.thrift2.ThriftServer 
 name=main signature=([Ljava/lang/String;)V isStatic=true
   SourceLine classname=org.apache.hadoop.hbase.thrift2.ThriftServer 
 start=202 end=290 startBytecode=0 endBytecode=1246 
 sourcefile=ThriftServer.java 
 sourcepath=org/apache/hadoop/hbase/thrift2/ThriftServer.java/
 /Method
 SourceLine classname=org.apache.hadoop.hbase.thrift2.ThriftServer 
 start=284 end=284 startBytecode=479 endBytecode=479 
 sourcefile=ThriftServer.java 
 sourcepath=org/apache/hadoop/hbase/thrift2/ThriftServer.java/
   /BugInstance
{code}
The class was not touched by this JIRA. Looks like HBASE-7757 touched that 
class.

 Prefix Compression - Trie data block encoding
 -

 Key: HBASE-4676
 URL: https://issues.apache.org/jira/browse/HBASE-4676
 Project: HBase
  Issue Type: New Feature
  Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
 Attachments: 4676-prefix-tree-trunk-v16.patch, 
 HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v10.patch, 
 HBASE-4676-prefix-tree-trunk-v11.patch, 
 HBASE-4676-prefix-tree-trunk-v12.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v14.patch, 
 HBASE-4676-prefix-tree-trunk-v15.patch, 
 HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, 
 HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, 
 HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, 
 HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, 
 PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by 
 blockSize.png


 The HBase data block format has room for 2 significant improvements for 
 applications that have high block cache hit ratios.  
 First, there is no prefix compression, and the current KeyValue format is 
 somewhat metadata heavy, so there can be tremendous memory bloat for many 
 common data layouts, specifically those with long keys and short values.
 Second, there is no random access to KeyValues inside data blocks.  This 
 means that every time you double the datablock size, average seek time (or 
 average cpu consumption) goes up by a factor of 2.  The standard 64KB block 
 size is ~10x slower for random seeks than a 4KB block size, but block sizes 
 as small as 4KB cause problems elsewhere.  Using block sizes of 256KB or 1MB 
 or more may be more efficient from a disk access and block-cache perspective 
 in many big-data applications, but doing so is infeasible from a random seek 
 perspective.
 The PrefixTrie block encoding format attempts to solve both of these 
 problems.  Some features:
 * trie format for row key encoding completely eliminates duplicate row keys 
 and encodes similar row keys into a standard trie structure which also saves 
 a lot of space
 * the column family is currently stored once at the beginning of each block.  
 this could easily be modified to allow multiple family names per block
 * all qualifiers in the block are stored in their own trie format which 
 caters nicely to wide rows.  duplicate qualifers between rows are eliminated. 
  the size of this trie determines the width of the block's qualifier 
 fixed-width-int
 * the minimum timestamp is stored at the beginning of the block, and deltas 
 are calculated from that.  the maximum delta determines the width of the 
 block's timestamp fixed-width-int
 The block is structured with metadata at the beginning, then a section for 
 the row trie, then the column trie, then the timestamp deltas, and then then 
 all the values.  Most work is done in the row trie, where every leaf node 
 (corresponding to a row) contains a list of offsets/references corresponding 
 to the cells in that row.  Each cell is fixed-width to enable binary 
 searching and is represented by [1 byte

[jira] [Updated] (HBASE-7561) Display the total number of regions for a given table on the master webUI

2013-02-06 Thread Michael Weng (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Weng updated HBASE-7561:


Attachment: HBASE-7561-trunk.txt

Attached patch for trunk.

I am having some trouble bring up hbase on my local machine. No verification 
for this patch at this point.

 Display the total number of regions for a given table on the master webUI
 -

 Key: HBASE-7561
 URL: https://issues.apache.org/jira/browse/HBASE-7561
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 0.94.4
Reporter: Ming Ma
Assignee: Michael Weng
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7561-trunk.txt, HBASE-7561.txt, screenshot-1.jpg


 This is to make it easy to find out the summary of the # of regions for each 
 table on one web page. Currently you need to click on each table URL to find 
 out the # of region of that table. We find this useful to support a shared 
 cluster with different clients.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572973#comment-13572973
]

Ted Yu commented on HBASE-4676:
---

Will integrate later in the evening if there is no further review comment.

Prefix Compression - Trie data block encoding
-

Key: HBASE-4676
URL: https://issues.apache.org/jira/browse/HBASE-4676
Project: HBase
Issue Type: New Feature
Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
Attachments: 4676-prefix-tree-trunk-v16.patch,
HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch,
HBASE-4676-prefix-tree-trunk-v10.patch,
HBASE-4676-prefix-tree-trunk-v11.patch,
HBASE-4676-prefix-tree-trunk-v12.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v14.patch,
HBASE-4676-prefix-tree-trunk-v15.patch,
HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch,
HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch,
HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch,
HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar,
PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
blockSize.png

The HBase data block format has room for 2 significant improvements for
applications that have high block cache hit ratios.
First, there is no prefix compression, and the current KeyValue format is
somewhat metadata heavy, so there can be tremendous memory bloat for many
common data layouts, specifically those with long keys and short values.
Second, there is no random access to KeyValues inside data blocks. This
means that every time you double the datablock size, average seek time (or
average cpu consumption) goes up by a factor of 2. The standard 64KB block
size is ~10x slower for random seeks than a 4KB block size, but block sizes
as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB
or more may be more efficient from a disk access and block-cache perspective
in many big-data applications, but doing so is infeasible from a random seek
perspective.
The PrefixTrie block encoding format attempts to solve both of these
problems. Some features:
* trie format for row key encoding completely eliminates duplicate row keys
and encodes similar row keys into a standard trie structure which also saves
a lot of space
* the column family is currently stored once at the beginning of each block.
this could easily be modified to allow multiple family names per block
* all qualifiers in the block are stored in their own trie format which
caters nicely to wide rows. duplicate qualifers between rows are eliminated.
the size of this trie determines the width of the block's qualifier
fixed-width-int
* the minimum timestamp is stored at the beginning of the block, and deltas
are calculated from that. the maximum delta determines the width of the
block's timestamp fixed-width-int
The block is structured with metadata at the beginning, then a section for
the row trie, then the column trie, then the timestamp deltas, and then then
all the values. Most work is done in the row trie, where every leaf node
(corresponding to a row) contains a list of offsets/references corresponding
to the cells in that row. Each cell is fixed-width to enable binary
searching and is represented by [1 byte operationType, X bytes qualifier
offset, X bytes timestamp delta offset].
If all operation types are the same for a block, there will be zero per-cell
overhead. Same for timestamps. Same for qualifiers when i get a chance.
So, the compression aspect is very strong, but makes a few small sacrifices
on VarInt size to enable faster binary searches in trie fan-out nodes.
A more compressed but slower version might build on this by also applying
further (suffix, etc) compression on the trie nodes at the cost of slower
write speed. Even further compression could be obtained by using all VInts
instead of FInts with a sacrifice on random seek speed (though not huge).
One current drawback is the current write speed. While programmed with good
constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not
programmed with the same level of optimization as the read path. Work will
need to be done to optimize the data structures used for encoding and could
probably show a 10x increase. It will still be slower than delta encoding,
but with a much higher decode speed. I have not yet created a

[jira] [Updated] (HBASE-7561) Display the total number of regions for a given table on the master webUI


 [ 
https://issues.apache.org/jira/browse/HBASE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7561:


Attachment: HBASE-7561-v0.patch

Here's the port to trunk.  I haven't checked it on cluster yet as my machine is 
busy, will do shortly.

 Display the total number of regions for a given table on the master webUI
 -

 Key: HBASE-7561
 URL: https://issues.apache.org/jira/browse/HBASE-7561
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 0.94.4
Reporter: Ming Ma
Assignee: Michael Weng
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7561-trunk.txt, HBASE-7561.txt, 
 HBASE-7561-v0.patch, screenshot-1.jpg


 This is to make it easy to find out the summary of the # of regions for each 
 table on one web page. Currently you need to click on each table URL to find 
 out the # of region of that table. We find this useful to support a shared 
 cluster with different clients.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7678) make storefile management pluggable, together with compaction


[ 
https://issues.apache.org/jira/browse/HBASE-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572978#comment-13572978
 ] 

Hadoop QA commented on HBASE-7678:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12568304/HBASE-7678--and-7603.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 24 new 
or modified tests.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 1 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4359//console

This message is automatically generated.

 make storefile management pluggable, together with compaction
 -

 Key: HBASE-7678
 URL: https://issues.apache.org/jira/browse/HBASE-7678
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
 Attachments: HBASE-7678--and-7603.patch, HBASE-7678-v0.patch, 
 HBASE-7678-v1.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7561) Display the total number of regions for a given table on the master webUI


[ 
https://issues.apache.org/jira/browse/HBASE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572981#comment-13572981
 ] 

Sergey Shelukhin commented on HBASE-7561:
-

Woops, disregard my patch

 Display the total number of regions for a given table on the master webUI
 -

 Key: HBASE-7561
 URL: https://issues.apache.org/jira/browse/HBASE-7561
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 0.94.4
Reporter: Ming Ma
Assignee: Michael Weng
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7561-trunk.txt, HBASE-7561.txt, 
 HBASE-7561-v0.patch, screenshot-1.jpg


 This is to make it easy to find out the summary of the # of regions for each 
 table on one web page. Currently you need to click on each table URL to find 
 out the # of region of that table. We find this useful to support a shared 
 cluster with different clients.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.

[
https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572984#comment-13572984
]

stack commented on HBASE-7763:
--

bq. Seq nums are just breaking a tie on ordering for key values that have the
same timestamp. We've always said that it's non-deterministic which kv you will
get back if you have two writes at the same timestamp.

Yeah, it'd break the tie in favor of the mostly recently written which would be
less surprising than if it were the reverse. I suppose too, we would need to
always select a contiguous set of files -- contiguous in the order in which
they were written -- if we wanted to be sure to return the last written (if we
could select any set, we might not include the file that had the key w/ most
recent update). I believe it used to work this way (thats why the note in the
refguide that Sergey quotes).

This is great stuff you fellas are doing. It can make all the difference in
the world... any improvement found herein.

Compactions not sorting based on size anymore.
--

Key: HBASE-7763
URL: https://issues.apache.org/jira/browse/HBASE-7763
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.96.0, 0.94.4
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Critical
Fix For: 0.96.0, 0.94.6

Attachments: HBASE-7763-trunk-TESTING.patch,
HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch

Currently compaction selection is not sorting based on size. This causes
selection to choose larger files to re-write than are needed when bulk loads
are involved.

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572986#comment-13572986
]

stack commented on HBASE-4676:
--

+1 on integrating.

[~mcorgan] Any chance of adding a release note up here in JIRA?

Prefix Compression - Trie data block encoding
-

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

2013-02-06 Thread Matt Corgan (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572987#comment-13572987
]

Matt Corgan commented on HBASE-4676:

yep - will do. i'll create a few follow on issues as well for a benchmark,
tool for testing on cluster data, pooling decoders, etc

Prefix Compression - Trie data block encoding
-

[jira] [Updated] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper

2013-02-06 Thread Jean-Daniel Cryans (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jean-Daniel Cryans updated HBASE-3171:
--

Attachment: HBASE-3171-v3.patch

Posting a much needed rebase because of HBASE-7213 and a bunch of others. I
just ran TestHBaseFsck and it worked.

Drop ROOT and instead store META location(s) directly in ZooKeeper
--

Key: HBASE-3171
URL: https://issues.apache.org/jira/browse/HBASE-3171
Project: HBase
Issue Type: Improvement
Components: Client, master, regionserver, Zookeeper
Reporter: Jonathan Gray
Attachments: HBASE-3171.patch, HBASE-3171-v2.patch,
HBASE-3171-v3.patch

Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and
reading the META location, we should just store the META location directly in
ZooKeeper.
The purpose of the root region from the bigtable paper was to support
multiple meta regions. Currently, we explicitly only support a single meta
region, so the translation from our current code of a single root location to
a single meta location will be very simple. Long-term, it seems reasonable
that we could store several meta region locations in ZK. There's been some
discussion in HBASE-1755 about actually moving META into ZK, but I think this
jira is a good step towards taking some of the complexity out of how we have
to deal with catalog tables everywhere.
As-is, a new client already requires ZK to get the root location, so this
would not change those requirements in any way.
The primary motivation for this is to simplify things like CatalogTracker.
The way we can handle root in that class is really simple but the tracking of
meta is difficulty and a bit hacky. This hack on tracking of the meta
location is what caused one of the bugs over in HBASE-3159.

[jira] [Updated] (HBASE-4676) Prefix Compression - Trie data block encoding


 [ 
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-4676:
--

Attachment: 4676-prefix-tree-trunk-v17.patch

Patch v17 addresses the following findbugs warning:
{code}
tyu$ diff 4348-newPatchFindbugsWarningshbase-common.xml 
4354-newPatchFindbugsWarningshbase-common.xml 
3c3
 BugCollection version=2.0.1 sequence=0 timestamp=94668480 
analysisTimestamp=1360116568099 release=
---
 BugCollection version=2.0.1 sequence=0 timestamp=94668480 
 analysisTimestamp=1360175312469 release=
172a173,184
   BugInstance type=NP_NULL_ON_SOME_PATH_MIGHT_BE_INFEASIBLE priority=2 
 abbrev=NP category=STYLE
 Class classname=org.apache.hadoop.hbase.util.Bytes
   SourceLine classname=org.apache.hadoop.hbase.util.Bytes start=57 
 end=1761 sourcefile=Bytes.java 
 sourcepath=org/apache/hadoop/hbase/util/Bytes.java/
 /Class
 Method classname=org.apache.hadoop.hbase.util.Bytes name=equals 
 signature=(Ljava/util/List;Ljava/util/List;)Z isStatic=true
   SourceLine classname=org.apache.hadoop.hbase.util.Bytes start=1725 
 end=1742 startBytecode=0 endBytecode=252 sourcefile=Bytes.java 
 sourcepath=org/apache/hadoop/hbase/util/Bytes.java/
 /Method
 LocalVariable name=a register=0 pc=30 
 role=LOCAL_VARIABLE_VALUE_OF/
 SourceLine classname=org.apache.hadoop.hbase.util.Bytes start=1734 
 end=1734 startBytecode=31 endBytecode=31 sourcefile=Bytes.java 
 sourcepath=org/apache/hadoop/hbase/util/Bytes.java 
 role=SOURCE_LINE_DEREF/
 SourceLine classname=org.apache.hadoop.hbase.util.Bytes start=1728 
 end=1728 startBytecode=15 endBytecode=15 sourcefile=Bytes.java 
 sourcepath=org/apache/hadoop/hbase/util/Bytes.java 
 role=SOURCE_LINE_KNOWN_NULL/
 SourceLine classname=org.apache.hadoop.hbase.util.Bytes start=1731 
 end=1731 startBytecode=21 endBytecode=21 sourcefile=Bytes.java 
 sourcepath=org/apache/hadoop/hbase/util/Bytes.java 
 role=SOURCE_LINE_KNOWN_NULL/
   /BugInstance
{code}

 Prefix Compression - Trie data block encoding
 -

 Key: HBASE-4676
 URL: https://issues.apache.org/jira/browse/HBASE-4676
 Project: HBase
  Issue Type: New Feature
  Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
 Attachments: 4676-prefix-tree-trunk-v16.patch, 
 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, 
 HBASE-4676-common-and-server-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v10.patch, 
 HBASE-4676-prefix-tree-trunk-v11.patch, 
 HBASE-4676-prefix-tree-trunk-v12.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v14.patch, 
 HBASE-4676-prefix-tree-trunk-v15.patch, 
 HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, 
 HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, 
 HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, 
 HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, 
 PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by 
 blockSize.png


 The HBase data block format has room for 2 significant improvements for 
 applications that have high block cache hit ratios.  
 First, there is no prefix compression, and the current KeyValue format is 
 somewhat metadata heavy, so there can be tremendous memory bloat for many 
 common data layouts, specifically those with long keys and short values.
 Second, there is no random access to KeyValues inside data blocks.  This 
 means that every time you double the datablock size, average seek time (or 
 average cpu consumption) goes up by a factor of 2.  The standard 64KB block 
 size is ~10x slower for random seeks than a 4KB block size, but block sizes 
 as small as 4KB cause problems elsewhere.  Using block sizes of 256KB or 1MB 
 or more may be more efficient from a disk access and block-cache perspective 
 in many big-data applications, but doing so is infeasible from a random seek 
 perspective.
 The PrefixTrie block encoding format attempts to solve both of these 
 problems.  Some features:
 * trie format for row key encoding completely eliminates duplicate row keys 
 and encodes similar row keys into a standard trie structure which also saves 
 a lot of space
 * the column family is currently stored once at the beginning of each block.  
 this could easily be modified to allow multiple family names per block
 * all qualifiers in the block are stored in their own trie format which 
 caters nicely to wide rows.  duplicate qualifers between rows are eliminated. 
  the size of this trie determines the width of the block's qualifier

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572999#comment-13572999
 ] 

Ted Yu commented on HBASE-7778:
---

I ran the addendum through trunk test suite and tests passed.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh

 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


 [ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7778:
--

Attachment: hbase-7778.patch

Let's give the robot a run on this -- this actually fixes a trunk problem.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573003#comment-13573003
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

While I care about the test passing, I care more about understanding why it 
passes and making it easier for future folks to understand why.  In this case, 
I think it was a bug -- the last test shouldn't be removed but instead the 
boolean was inverted.  I've renamed the noWait variable and I think it makes it 
much more obvious that that was the problem.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


 [ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7778:
--

Assignee: Jonathan Hsieh
  Status: Patch Available  (was: Open)

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7233) Serializing KeyValues


 [ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-7233:
-

Attachment: 7233v10.txt

Actually include new files.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead

2013-02-06 Thread Gustavo Anatoly (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573007#comment-13573007
 ] 

Gustavo Anatoly commented on HBASE-7462:


Hi, Nicolas.

I tried mock a cluster, but my knowledge is limited yet I had some exceptions, 
so studying the TestAssigmentManager class I found this code:

{code}
  @BeforeClass
  public static void beforeClass() throws Exception {
HTU.startMiniZKCluster();
  }

  @AfterClass
  public static void afterClass() throws IOException {
HTU.shutdownMiniZKCluster();
  }
{code}

Can I use this stuff? Or I should mock ZooKeeper?

Thanks.

 TestDrainingServer is an integration test. It should be a unit test instead
 ---

 Key: HBASE-7462
 URL: https://issues.apache.org/jira/browse/HBASE-7462
 Project: HBase
  Issue Type: Wish
  Components: test
Affects Versions: 0.96.0
Reporter: nkeywal
Priority: Trivial
  Labels: noob

 TestDrainingServer tests the function that allows to say that a regionserver 
 should not get new regions.
 As it is written today, it's an integration test: it starts  stops a cluster.
 The test would be more efficient if it would just check that the 
 AssignmentManager does not use the drained region server; whatever the 
 circumstances (bulk assign or not for example).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573008#comment-13573008
 ] 

Ted Yu commented on HBASE-7778:
---

Is the following change needed ?
{code}
-  if (activeMaster != null) activeMaster.master.shutdown();
+  if (activeMaster != null)
+activeMaster.master.shutdown();
+
{code}

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573012#comment-13573012
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

I prefer it because you can more easily step through it with a debugger.  I 
probably should have put  {} around the shutdown call.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7290) Online snapshots


[ 
https://issues.apache.org/jira/browse/HBASE-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573014#comment-13573014
 ] 

Jonathan Hsieh commented on HBASE-7290:
---

The Ted's comment above was copied to and discussed in HBASE-7778.

 Online snapshots 
 -

 Key: HBASE-7290
 URL: https://issues.apache.org/jira/browse/HBASE-7290
 Project: HBase
  Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh

 HBASE-6055 will be closed when the offline snapshots pieces get merged with 
 trunk.  This umbrella issue has all the online snapshot specific patches.  
 This will get merged once one of the implementations makes it into trunk.  
 Other flavors of online snapshots can then be done as normal patches instead 
 of on a development branch.  (was: HBASE-6055 will be closed when the online 
 snapshots pieces get merged with trunk.  This umbrella issue has all the 
 online snapshot specific patches.  This will get merged once one of the 
 implementations makes it into trunk.  Other flavors of online snapshots can 
 then be done as normal patches instead of on a development branch.)
 (not a fan of the quick edit descirption jira feature)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573039#comment-13573039
]

Hadoop QA commented on HBASE-4676:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12568325/4676-prefix-tree-trunk-v17.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 105
new or modified tests.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100

{color:red}-1 core tests{color}. The patch failed these unit tests:

org.apache.hadoop.hbase.coprocessor.example.TestBulkDeleteProtocol

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4360//console

This message is automatically generated.

Prefix Compression - Trie data block encoding
-

Key: HBASE-4676
URL: https://issues.apache.org/jira/browse/HBASE-4676
Project: HBase
Issue Type: New Feature
Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
Attachments: 4676-prefix-tree-trunk-v16.patch,
4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch,
HBASE-4676-common-and-server-v8.patch,
HBASE-4676-prefix-tree-trunk-v10.patch,
HBASE-4676-prefix-tree-trunk-v11.patch,
HBASE-4676-prefix-tree-trunk-v12.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v14.patch,
HBASE-4676-prefix-tree-trunk-v15.patch,
HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch,
HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch,
HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch,
HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar,
PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
blockSize.png

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573044#comment-13573044
]

Ted Yu commented on HBASE-4676:
---

Integrated to trunk (patch v17).

Thanks for the patch, Matt.

Thanks for the review, Stack.

Prefix Compression - Trie data block encoding
-

Key: HBASE-4676
URL: https://issues.apache.org/jira/browse/HBASE-4676
Project: HBase
Issue Type: New Feature
Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
Attachments: 4676-prefix-tree-trunk-v16.patch,
4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch,
HBASE-4676-common-and-server-v8.patch,
HBASE-4676-prefix-tree-trunk-v10.patch,
HBASE-4676-prefix-tree-trunk-v11.patch,
HBASE-4676-prefix-tree-trunk-v12.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v14.patch,
HBASE-4676-prefix-tree-trunk-v15.patch,
HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch,
HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch,
HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch,
HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar,
PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
blockSize.png

[jira] [Updated] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files


 [ 
https://issues.apache.org/jira/browse/HBASE-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7783:
--

Description: Apparently, I checked in a whole bunch of binary .class files 
into hbase-server/bin directory.  I'm just going to remove them.  (was: 
Apaarently, I checked in a whole bunch of binary .class files into 
hbase-server/bin directory.  I'm just going to remove them.)

 [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files 
 -

 Key: HBASE-7783
 URL: https://issues.apache.org/jira/browse/HBASE-7783
 Project: HBase
  Issue Type: Sub-task
  Components: snapshots
Affects Versions: hbase-7290
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: hbase-7290


 Apparently, I checked in a whole bunch of binary .class files into 
 hbase-server/bin directory.  I'm just going to remove them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files

Jonathan Hsieh created HBASE-7783:
-

 Summary: [snapshot 130201 merge] Remove accidentally committed 
hbase-server/bin files 
 Key: HBASE-7783
 URL: https://issues.apache.org/jira/browse/HBASE-7783
 Project: HBase
  Issue Type: Sub-task
  Components: snapshots
Affects Versions: hbase-7290
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: hbase-7290


Apaarently, I checked in a whole bunch of binary .class files into 
hbase-server/bin directory.  I'm just going to remove them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573049#comment-13573049
 ] 

Hadoop QA commented on HBASE-7778:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12568329/hbase-7778.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4361//console

This message is automatically generated.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 ---

[jira] [Updated] (HBASE-7561) Display the total number of regions for a given table on the master webUI


 [ 
https://issues.apache.org/jira/browse/HBASE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7561:


Attachment: HBASE-7561-v0.patch

 Display the total number of regions for a given table on the master webUI
 -

 Key: HBASE-7561
 URL: https://issues.apache.org/jira/browse/HBASE-7561
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 0.94.4
Reporter: Ming Ma
Assignee: Michael Weng
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7561-trunk.txt, HBASE-7561.txt, 
 HBASE-7561-v0.patch, HBASE-7561-v0.patch, screenshot-1.jpg


 This is to make it easy to find out the summary of the # of regions for each 
 table on one web page. Currently you need to click on each table URL to find 
 out the # of region of that table. We find this useful to support a shared 
 cluster with different clients.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


[ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573053#comment-13573053
 ] 

Jonathan Hsieh commented on HBASE-7778:
---

This modifies JVMClusterUtil which is used by all minicluster unit tests and 
they all passed.

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7561) Display the total number of regions for a given table on the master webUI


[ 
https://issues.apache.org/jira/browse/HBASE-7561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573054#comment-13573054
 ] 

Sergey Shelukhin commented on HBASE-7561:
-

I had some leftover cluster from something so I tried my trunk patch, attached 
the fix which seems to be exactly equivalent of Michael's now. It works on 
front page, although doesn't display this in the 2nd User Tables view (in large 
tab)

 Display the total number of regions for a given table on the master webUI
 -

 Key: HBASE-7561
 URL: https://issues.apache.org/jira/browse/HBASE-7561
 Project: HBase
  Issue Type: Improvement
  Components: UI
Affects Versions: 0.94.4
Reporter: Ming Ma
Assignee: Michael Weng
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7561-trunk.txt, HBASE-7561.txt, 
 HBASE-7561-v0.patch, HBASE-7561-v0.patch, screenshot-1.jpg


 This is to make it easy to find out the summary of the # of regions for each 
 table on one web page. Currently you need to click on each table URL to find 
 out the # of region of that table. We find this useful to support a shared 
 cluster with different clients.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573055#comment-13573055
]

Lars Hofhansl commented on HBASE-7723:
--

No. In that case it will not work, because the path was changed from
/hbase/92 to /hbase/94. It only fixes up the HDFS part of things.

I.e. you came from Hadoop 1.0 with root: hdfs://my.name.node.com/hbase and
upgrade to Hadoop 2 where the root would be hdfs://my_cluster_name/hbase.

So in your scenario somebody renamed the directory in HDFS and then changed the
root dir in HBase? Is that a common case? If so, let's go with your approach
(after you remove the use of ZNODE_PATH_SEPARATOR :)) as it would cover this
scenario as well.

Remove NN URI from ZK splitlogs.

[jira] [Updated] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.


 [ 
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-7778:
--

Affects Version/s: hbase-7290
   0.96.0
Fix Version/s: hbase-7290
   0.96.0

 [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due 
 to interrupt flag.
 ---

 Key: HBASE-7778
 URL: https://issues.apache.org/jira/browse/HBASE-7778
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 0.96.0, hbase-7290
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 0.96.0, hbase-7290

 Attachments: hbase-7778.patch


 Something in the merge has set the interrupted flag on the main test threads 
 of TestReplicationDisabledinactivePeer, TestRestartCluster, and 
 TestCatalogTrackerOnCluster.  
 These unacceptable hacks make the tests run and pass: 
 {code}
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
  b/hbase-server/src/test/java/or
 index f3e57d6..a8d2ef7 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
 @@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
  // Shutdown hbase.
  UTIL.shutdownMiniHBaseCluster();
  // Give the various ZKWatchers some time to settle their affairs.
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(1000);
  
  // Mess with the root location in the running zk.  Set it to be nonsense.
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
  b/hbase-server/src/test/java/org/apache/h
 index 15225e1..9f7f526 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
 @@ -108,6 +108,7 @@ public class TestRestartCluster {
  UTIL.shutdownMiniHBaseCluster();
  
  LOG.info(\n\nSleeping a bit);
 +Thread.interrupted(); // HACK clear interrupt state.
  Thread.sleep(2000);
  
  LOG.info(\n\nStarting cluster the second time);
 diff --git 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
  b/hbase-server/src/t
 index b089fbe..8162f4b 100644
 --- 
 a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 +++ 
 b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
 @@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends 
 TestReplicationBase {
  // enabling and shutdown the peer
  admin.enablePeer(2);
  utility2.shutdownMiniHBaseCluster();
 +Thread.interrupted(); // HACK clear interrupted flag.
  
  byte[] rowkey = Bytes.toBytes(disable inactive peer);
  Put put = new Put(rowkey);
 {code}
 On the snapshot branch and on the trunk branch before the merge, these tests 
 passed. Need to figure out how they combination caused this behavior change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7768) zkcluster in local mode not seeing configurations in hbase-{site|default}.xml


[ 
https://issues.apache.org/jira/browse/HBASE-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573059#comment-13573059
 ] 

Hudson commented on HBASE-7768:
---

Integrated in HBase-TRUNK #3858 (See 
[https://builds.apache.org/job/HBase-TRUNK/3858/])
HBASE-7768. Zkcluster in local mode not seeing configurations in 
hbase-{site|default}.xml (Revision 1443234)

 Result = FAILURE
enis : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java


 zkcluster in local mode not seeing configurations in hbase-{site|default}.xml
 -

 Key: HBASE-7768
 URL: https://issues.apache.org/jira/browse/HBASE-7768
 Project: HBase
  Issue Type: Bug
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7768_2.patch, HBASE-7768.patch


 in case of local mode we are creating mini zk cluster with default constructor
 {code}
   if (LocalHBaseCluster.isLocal(conf)) {
 final MiniZooKeeperCluster zooKeeperCluster =
   new MiniZooKeeperCluster();
 {code}
 {code}
   public MiniZooKeeperCluster() {
 this(new Configuration());
   }
 {code}
 which is not reading any configurations in hbase-site|default.xml
 I think we can pass configuration object to MiniZooKeeperCluster

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called


[ 
https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573060#comment-13573060
 ] 

Hudson commented on HBASE-7772:
---

Integrated in HBase-TRUNK #3858 (See 
[https://builds.apache.org/job/HBase-TRUNK/3858/])
HBASE-7772 Cluster ID is not initialized correctly in RPC client (Revision 
1443221)

 Result = FAILURE
garyh : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java


 clusterId is not set in conf properly if only 
 TableMapReduceUtil.initCredentials() is called
 

 Key: HBASE-7772
 URL: https://issues.apache.org/jira/browse/HBASE-7772
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
 Fix For: 0.96.0, 0.94.5

 Attachments: HBASE-7772_94.patch, HBASE-7772.patch, 
 HBASE-7772-v2_0.94.patch, HBASE-7772-v2.patch


 clusterId is not set in the job's conf correctly if only 
 TableMapReduceUtil.initCredentials() is called and thus fails to include the 
 token when using an hbase client in any of the job's tasks. This is a 
 regression.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.

[
https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573072#comment-13573072
]

Lars Hofhansl commented on HBASE-7763:
--

So just to state the obvious: The selections depends on what metric we're
trying to optimize.
We can (1) optimize for write amplification (i.e. minimizing it) or optimize
for (2) read performance (reducing the number of scanner participating in the
merge scan).

For case #1 we'd pick larger files first, and for #2 we'd pick smaller files
first.
Making this configurable or pluggable thus makes a lot of sense.

They actual behavior in a production setting is very hard to predict, and as I
said above: The folks from Facebook did a lot of research and measuring to come
up with the current compaction selection algorithm. I'm surprised they are
quiet on this.

Compactions not sorting based on size anymore.
--

Attachments: HBASE-7763-trunk-TESTING.patch,
HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch

Currently compaction selection is not sorting based on size. This causes
selection to choose larger files to re-write than are needed when bulk loads
are involved.

[jira] [Updated] (HBASE-7776) Use ErrorReporter/Log instead of System.out in hbck


 [ 
https://issues.apache.org/jira/browse/HBASE-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-7776:
---

Attachment: trunk-7776_v1.patch

 Use ErrorReporter/Log instead of System.out in hbck
 ---

 Key: HBASE-7776
 URL: https://issues.apache.org/jira/browse/HBASE-7776
 Project: HBase
  Issue Type: Bug
  Components: hbck
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-7776_v1.patch


 There are lots places to log messages with System.out.  We should use 
 ErrorReporter or Log instead, which can be configured to catch what we want.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7776) Use ErrorReporter/Log instead of System.out in hbck


 [ 
https://issues.apache.org/jira/browse/HBASE-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-7776:
---

Status: Patch Available  (was: Open)

 Use ErrorReporter/Log instead of System.out in hbck
 ---

 Key: HBASE-7776
 URL: https://issues.apache.org/jira/browse/HBASE-7776
 Project: HBase
  Issue Type: Bug
  Components: hbck
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Attachments: trunk-7776_v1.patch


 There are lots places to log messages with System.out.  We should use 
 ErrorReporter or Log instead, which can be configured to catch what we want.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck


[ 
https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573078#comment-13573078
 ] 

Lars Hofhansl commented on HBASE-7698:
--

Should we commit as is and discuss perf improvement in another issue?


 race between RS shutdown thread and openregionhandler causes region to get 
 stuck
 

 Key: HBASE-7698
 URL: https://issues.apache.org/jira/browse/HBASE-7698
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.4
Reporter: Sergey Shelukhin
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.96.0, 0.94.5

 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, 
 HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, 
 HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch


 2013-01-22 17:59:03,237 INFO  [Shutdown of 
 org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] 
 hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing 
 fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08
 ...
 2013-01-22 17:59:03,411 DEBUG 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): 
 Closing 
 IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.:
  disabling compactions amp; flushes
 2013-01-22 17:59:03,411 DEBUG 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): 
 Updates disabled for region 
 IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.
 2013-01-22 17:59:03,415 ERROR 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): 
 Caught throwable while processing event M_RS_OPEN_REGION
 java.io.IOException: java.io.IOException: java.io.IOException: Filesystem 
 closed
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945)
   at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459)
   at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:680)
 tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never 
 called and region can get stuck.
 As an added benefit, the meta is already written by that time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck


[ 
https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573083#comment-13573083
 ] 

Jimmy Xiang commented on HBASE-7698:


It is ok with me.

 race between RS shutdown thread and openregionhandler causes region to get 
 stuck
 

 Key: HBASE-7698
 URL: https://issues.apache.org/jira/browse/HBASE-7698
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.94.4
Reporter: Sergey Shelukhin
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.96.0, 0.94.5

 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, 
 HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, 
 HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch


 2013-01-22 17:59:03,237 INFO  [Shutdown of 
 org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] 
 hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing 
 fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08
 ...
 2013-01-22 17:59:03,411 DEBUG 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): 
 Closing 
 IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.:
  disabling compactions amp; flushes
 2013-01-22 17:59:03,411 DEBUG 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): 
 Updates disabled for region 
 IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.
 2013-01-22 17:59:03,415 ERROR 
 [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): 
 Caught throwable while processing event M_RS_OPEN_REGION
 java.io.IOException: java.io.IOException: java.io.IOException: Filesystem 
 closed
   at 
 org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974)
   at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945)
   at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459)
   at 
 org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143)
   at 
 org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:680)
 tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never 
 called and region can get stuck.
 As an added benefit, the meta is already written by that time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HBASE-7784) move the code related to selection that is specific to default compaction policy, into default compaction policy (from HStore)

Sergey Shelukhin created HBASE-7784:
---

 Summary: move the code related to selection that is specific to 
default compaction policy, into default compaction policy (from HStore)
 Key: HBASE-7784
 URL: https://issues.apache.org/jira/browse/HBASE-7784
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7784) move the code related to selection that is specific to default compaction policy, into default compaction policy (from HStore)


 [ 
https://issues.apache.org/jira/browse/HBASE-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7784:


Description: There are some TODO [level compaction] in HBASE-7603 patch, 
there may also be few other similar places.  (was: There are some TODO [level 
compaction] in HBASE-7604 patch, there may also be few other similar places.)

 move the code related to selection that is specific to default compaction 
 policy, into default compaction policy (from HStore)
 --

 Key: HBASE-7784
 URL: https://issues.apache.org/jira/browse/HBASE-7784
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin

 There are some TODO [level compaction] in HBASE-7603 patch, there may also be 
 few other similar places.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HBASE-7784) move the code related to selection that is specific to default compaction policy, into default compaction policy (from HStore)


 [ 
https://issues.apache.org/jira/browse/HBASE-7784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HBASE-7784:


Description: There are some TODO [level compaction] in HBASE-7604 patch, 
there may also be few other similar places.

 move the code related to selection that is specific to default compaction 
 policy, into default compaction policy (from HStore)
 --

 Key: HBASE-7784
 URL: https://issues.apache.org/jira/browse/HBASE-7784
 Project: HBase
  Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin

 There are some TODO [level compaction] in HBASE-7604 patch, there may also be 
 few other similar places.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7776) Use ErrorReporter/Log instead of System.out in hbck

[
https://issues.apache.org/jira/browse/HBASE-7776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573111#comment-13573111
]

Hadoop QA commented on HBASE-7776:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568348/trunk-7776_v1.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4362//console

This message is automatically generated.

Use ErrorReporter/Log instead of System.out in hbck
---

Key: HBASE-7776
URL: https://issues.apache.org/jira/browse/HBASE-7776
Project: HBase
Issue Type: Bug
Components: hbck
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-7776_v1.patch

There are lots places to log messages with System.out. We should use
ErrorReporter or Log instead, which can be configured to catch what we want.

[jira] [Updated] (HBASE-7723) Remove NN URI from ZK splitlogs.


 [ 
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Himanshu Vashishtha updated HBASE-7723:
---

Attachment: HBASE-7723-94-v5.patch

is this okay Lars?
Thanks for all the reviews so far :)

 Remove NN URI from ZK splitlogs.
 

 Key: HBASE-7723
 URL: https://issues.apache.org/jira/browse/HBASE-7723
 Project: HBase
  Issue Type: Bug
  Components: hadoop2, master
Affects Versions: 0.92.0
Reporter: Kevin Odell
Assignee: Himanshu Vashishtha
 Fix For: 0.96.0, 0.94.5

 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, 
 HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, 
 HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, 
 HBASE-7723-trunk-v2.patch


 When moving to HDFS HA or removing HA we end up changing the NN namespace.  
 This can cause the HMaster not to start up fully due to trying to split 
 phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: 
 Wrong FS: error messages.  The HLogs in question might not even be on HDFS 
 anymore.  You have to go in a manually clear out the ZK splitlogs directory 
 to get HBase to properly boot up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573123#comment-13573123
]

Lars Hofhansl commented on HBASE-7723:
--

Yeah, looks good. Initially I did not like the fact that we have to count 3
levels from the back.
But with your example I realize that the root dir can be more than one level
deep so we have to count from the back.

TL;DR: +1 :)

Remove NN URI from ZK splitlogs.

[jira] [Updated] (HBASE-7403) Online Merge

2013-02-06 Thread chunhui shen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chunhui shen updated HBASE-7403:


Attachment: hbase-7403-trunkv16.patch

Addressing stack's comments on review board

 Online Merge
 

 Key: HBASE-7403
 URL: https://issues.apache.org/jira/browse/HBASE-7403
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.6

 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 
 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, 
 hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, 
 hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, 
 hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, 
 hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, 
 hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, 
 merge region.pdf


 The feature of this online merge:
 1.Online,no necessary to disable table
 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
 3.Easy to call merege request, no need to input a long region name, only 
 encoded name enough
 4.No limit when operation, you don't need to tabke care the events like 
 Server Dead, Balance, Split, Disabing/Enabing table, no need to take care 
 whether you send a wrong merge request, it has alread done for you
 5.Only little offline time for two merging regions
 Usage:
 1.Tool:  
 bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] 
 table-name region-encodedname-1 region-encodedname-2
 2.API: static void MergeManager#createMergeRequest
 We need merge in the following cases：
 1.Region hole or region overlap, can’t be fix by hbck
 2.Region become empty because of TTL and not reasonable Rowkey design
 3.Region is always empty or very small because of presplit when create table
 4.Too many empty or small regions would reduce the system performance(e.g. 
 mslab)
 Current merge tools only support offline and are not able to redo if 
 exception is thrown in the process of merging, causing a dirty data
 For online system, we need a online merge.
 This implement logic of this patch for  Online Merge is :
 For example, merge regionA and regionB into regionC
 1.Offline the two regions A and B
 2.Merge the two regions in the HDFS(Create regionC’s directory, move 
 regionA’s and regionB’s file to regionC’s directory, delete regionA’s and 
 regionB’s directory)
 3.Add the merged regionC to .META.
 4.Assign the merged regionC
 As design of this patch , once we do the merge work in the HDFS,we could redo 
 it until successful if it throws exception or abort or server restart, but 
 couldn’t be rolled back. 
 It depends on
 Use zookeeper to record the transaction journal state, make redo easier
 Use zookeeper to send/receive merge request
 Merge transaction is executed on the master
 Support calling merge request through API or shell tool
 About the merge process, please see the attachment and patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573144#comment-13573144
]

stack commented on HBASE-7723:
--

The difference is port number only? (that 0.92 vs 0.94 is a clouderia-ism? Or
you are thinking log splitting should just work even if hbase.rootdir is
relatively shifted?) HA NN has no 'port' number? Can't we equate these
'filesystems'? They are the 'same', no? Its just that one has a NN ensemble
serving its metadata?

Its probably fine making log paths relative to root dir so you can move the
root dir and it would still work (Hopefully the case where the zk data from one
filesystem is set against a different filesystem will just fail horribly).

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573150#comment-13573150
]

Lars Hofhansl commented on HBASE-7723:
--

The problem occurs every time when the hbase root is changed in any way.
In our case we moved from a single NN to an HA NN. In that case the host
portion of the URI changes.
Similarly one could imagine that the hbase root Dir is moved.

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7768) zkcluster in local mode not seeing configurations in hbase-{site|default}.xml


[ 
https://issues.apache.org/jira/browse/HBASE-7768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573169#comment-13573169
 ] 

Hudson commented on HBASE-7768:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #395 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/395/])
HBASE-7768. Zkcluster in local mode not seeing configurations in 
hbase-{site|default}.xml (Revision 1443234)

 Result = FAILURE
enis : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMasterCommandLine.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java


 zkcluster in local mode not seeing configurations in hbase-{site|default}.xml
 -

 Key: HBASE-7768
 URL: https://issues.apache.org/jira/browse/HBASE-7768
 Project: HBase
  Issue Type: Bug
Reporter: rajeshbabu
Assignee: rajeshbabu
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7768_2.patch, HBASE-7768.patch


 in case of local mode we are creating mini zk cluster with default constructor
 {code}
   if (LocalHBaseCluster.isLocal(conf)) {
 final MiniZooKeeperCluster zooKeeperCluster =
   new MiniZooKeeperCluster();
 {code}
 {code}
   public MiniZooKeeperCluster() {
 this(new Configuration());
   }
 {code}
 which is not reading any configurations in hbase-site|default.xml
 I think we can pass configuration object to MiniZooKeeperCluster

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7757) Add web UI to REST server and Thrift server


[ 
https://issues.apache.org/jira/browse/HBASE-7757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573170#comment-13573170
 ] 

Hudson commented on HBASE-7757:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #395 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/395/])
HBASE-7757 Add web UI to REST server and Thrift server (Revision 1443069)

 Result = FAILURE
jxiang : 
Files : 
* /hbase/trunk/hbase-server/pom.xml
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/rest
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/rest/index.html
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/rest/rest.jsp
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/thrift
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/thrift/index.html
* /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/thrift/thrift.jsp


 Add web UI to REST server and Thrift server
 ---

 Key: HBASE-7757
 URL: https://issues.apache.org/jira/browse/HBASE-7757
 Project: HBase
  Issue Type: Improvement
  Components: REST, Thrift, UI
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
 Fix For: 0.96.0, 0.94.5

 Attachments: 0.94_7757_v1.patch, rest-0.94.png, rest-0.96.png, 
 thrift-0.94.png, thrift-0.96.png, trunk-7757_v1.patch


 Add Hadoop HttpServer (web UI) to REST server and Thrift server. The Hadoop 
 HttpServer supports metrics/jmx/conf/logLevel/stacks, which is useful to 
 monitor REST/Thrift server.
 For REST server, use a separate listener/context to avoid path mapping 
 conflicts. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding


[ 
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573171#comment-13573171
 ] 

Hudson commented on HBASE-4676:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #395 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/395/])
HBASE-4676 Prefix Compression - Trie data block encoding (Matt Corgan) 
(Revision 1443289)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRangeTool.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/RedundantKVGenerator.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellComparator.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellOutputStream.java
* /hbase/trunk/hbase-prefix-tree
* /hbase/trunk/hbase-prefix-tree/pom.xml
* /hbase/trunk/hbase-prefix-tree/src
* /hbase/trunk/hbase-prefix-tree/src/main
* /hbase/trunk/hbase-prefix-tree/src/main/java
* /hbase/trunk/hbase-prefix-tree/src/main/java/org
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeBlockMeta.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeCodec.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeSeeker.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/ArraySearcherPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/DecoderFactory.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArraySearcher.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeCell.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column/ColumnNodeReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column/ColumnReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/row
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/row/RowNodeReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp/MvccVersionDecoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp/TimestampDecoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/EncoderFactory.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/EncoderPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/PrefixTreeEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/ThreadLocalEncoderPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column/ColumnNodeWriter.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column/ColumnSectionWriter.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other/CellTypeEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other/LongEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/row
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/row/RowNodeWriter.java
*

[jira] [Commented] (HBASE-7773) rename StoreConfiguration, remove unnecessary method


[ 
https://issues.apache.org/jira/browse/HBASE-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573172#comment-13573172
 ] 

Hudson commented on HBASE-7773:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #395 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/395/])
HBASE-7773. Rename StoreConfiguration, remove unnecessary method (Sergey 
Shelukhin) (Revision 1443165)

 Result = FAILURE
enis : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfiguration.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java


 rename StoreConfiguration, remove unnecessary method
 

 Key: HBASE-7773
 URL: https://issues.apache.org/jira/browse/HBASE-7773
 Project: HBase
  Issue Type: Improvement
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor
 Fix For: 0.96.0

 Attachments: HBASE-7773-v0.patch, HBASE-7773-v1.patch


 StoreConfiguration interface name is confusing, and one of the methods is 
 unnecessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7772) clusterId is not set in conf properly if only TableMapReduceUtil.initCredentials() is called


[ 
https://issues.apache.org/jira/browse/HBASE-7772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573173#comment-13573173
 ] 

Hudson commented on HBASE-7772:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #395 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/395/])
HBASE-7772 Cluster ID is not initialized correctly in RPC client (Revision 
1443221)

 Result = FAILURE
garyh : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java


 clusterId is not set in conf properly if only 
 TableMapReduceUtil.initCredentials() is called
 

 Key: HBASE-7772
 URL: https://issues.apache.org/jira/browse/HBASE-7772
 Project: HBase
  Issue Type: Bug
Reporter: Francis Liu
Assignee: Francis Liu
 Fix For: 0.96.0, 0.94.5

 Attachments: HBASE-7772_94.patch, HBASE-7772.patch, 
 HBASE-7772-v2_0.94.patch, HBASE-7772-v2.patch


 clusterId is not set in the job's conf correctly if only 
 TableMapReduceUtil.initCredentials() is called and thus fails to include the 
 token when using an hbase client in any of the job's tasks. This is a 
 regression.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding


[ 
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573179#comment-13573179
 ] 

Ted Yu commented on HBASE-4676:
---

I ran a few tests, marked failed by HBase-TRUNK-on-Hadoop-2.0.0 #395, locally:

 1220  mvn clean test -Dhadoop.profile=2.0 -PlocalTests 
-Dtest=TestFullLogReconstruction
 1222  mvn test -Dhadoop.profile=2.0 -PlocalTests -Dtest=TestDelayedRpc
 1223  mvn test -Dhadoop.profile=2.0 -PlocalTests 
-Dtest=TestOfflineMetaRebuildHole,TestMergeTable

They passed.

 Prefix Compression - Trie data block encoding
 -

 Key: HBASE-4676
 URL: https://issues.apache.org/jira/browse/HBASE-4676
 Project: HBase
  Issue Type: New Feature
  Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
 Attachments: 4676-prefix-tree-trunk-v16.patch, 
 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, 
 HBASE-4676-common-and-server-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v10.patch, 
 HBASE-4676-prefix-tree-trunk-v11.patch, 
 HBASE-4676-prefix-tree-trunk-v12.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v13.patch, 
 HBASE-4676-prefix-tree-trunk-v14.patch, 
 HBASE-4676-prefix-tree-trunk-v15.patch, 
 HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, 
 HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, 
 HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, 
 HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, 
 HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, 
 PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by 
 blockSize.png


 The HBase data block format has room for 2 significant improvements for 
 applications that have high block cache hit ratios.  
 First, there is no prefix compression, and the current KeyValue format is 
 somewhat metadata heavy, so there can be tremendous memory bloat for many 
 common data layouts, specifically those with long keys and short values.
 Second, there is no random access to KeyValues inside data blocks.  This 
 means that every time you double the datablock size, average seek time (or 
 average cpu consumption) goes up by a factor of 2.  The standard 64KB block 
 size is ~10x slower for random seeks than a 4KB block size, but block sizes 
 as small as 4KB cause problems elsewhere.  Using block sizes of 256KB or 1MB 
 or more may be more efficient from a disk access and block-cache perspective 
 in many big-data applications, but doing so is infeasible from a random seek 
 perspective.
 The PrefixTrie block encoding format attempts to solve both of these 
 problems.  Some features:
 * trie format for row key encoding completely eliminates duplicate row keys 
 and encodes similar row keys into a standard trie structure which also saves 
 a lot of space
 * the column family is currently stored once at the beginning of each block.  
 this could easily be modified to allow multiple family names per block
 * all qualifiers in the block are stored in their own trie format which 
 caters nicely to wide rows.  duplicate qualifers between rows are eliminated. 
  the size of this trie determines the width of the block's qualifier 
 fixed-width-int
 * the minimum timestamp is stored at the beginning of the block, and deltas 
 are calculated from that.  the maximum delta determines the width of the 
 block's timestamp fixed-width-int
 The block is structured with metadata at the beginning, then a section for 
 the row trie, then the column trie, then the timestamp deltas, and then then 
 all the values.  Most work is done in the row trie, where every leaf node 
 (corresponding to a row) contains a list of offsets/references corresponding 
 to the cells in that row.  Each cell is fixed-width to enable binary 
 searching and is represented by [1 byte operationType, X bytes qualifier 
 offset, X bytes timestamp delta offset].
 If all operation types are the same for a block, there will be zero per-cell 
 overhead.  Same for timestamps.  Same for qualifiers when i get a chance.  
 So, the compression aspect is very strong, but makes a few small sacrifices 
 on VarInt size to enable faster binary searches in trie fan-out nodes.
 A more compressed but slower version might build on this by also applying 
 further (suffix, etc) compression on the trie nodes at the cost of slower 
 write speed.  Even further compression could be obtained by using all VInts 
 instead of FInts with a sacrifice on random seek speed (though not huge).
 One current drawback is the current write speed.  While programmed with good 
 constructs like TreeMaps, ByteBuffers, binary searches,

[jira] [Commented] (HBASE-7403) Online Merge


[ 
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573180#comment-13573180
 ] 

Hadoop QA commented on HBASE-7403:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12568364/hbase-7403-trunkv16.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 9 new 
or modified tests.

{color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 
2.0 profile.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces lines longer than 
100

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//console

This message is automatically generated.

 Online Merge
 

 Key: HBASE-7403
 URL: https://issues.apache.org/jira/browse/HBASE-7403
 Project: HBase
  Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.96.0, 0.94.6

 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 
 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, 
 hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, 
 hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, 
 hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, 
 hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, 
 hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, 
 merge region.pdf


 The feature of this online merge:
 1.Online,no necessary to disable table
 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
 3.Easy to call merege request, no need to input a long region name, only 
 encoded name enough
 4.No limit when operation, you don't need to tabke care the events like 
 Server Dead, Balance, Split, Disabing/Enabing table, no need to take care 
 whether you send a wrong merge request, it has alread done for you
 5.Only little offline time for two merging regions
 Usage:
 1.Tool:  
 bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] 
 table-name region-encodedname-1 region-encodedname-2
 2.API: static void MergeManager#createMergeRequest
 We need merge in the following cases：
 1.Region hole or region overlap, can’t be fix by hbck
 2.Region become empty because of TTL and not reasonable Rowkey design
 3.Region is always empty or very small because of presplit when create table
 4.Too many empty or small regions would reduce the system performance(e.g. 
 mslab)
 Current merge tools only support offline and are not able to redo if 
 exception is thrown in the process of merging, causing a dirty data
 For online system, we need a online merge.
 This implement logic of this patch for  Online Merge is :
 For example, merge regionA and regionB into regionC
 1.Offline the two regions A and B
 2.Merge the two regions in the HDFS(Create regionC’s directory, move 
 regionA’s and regionB’s file to regionC’s directory, delete

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573189#comment-13573189
 ] 

Ted Yu commented on HBASE-7233:
---

Fro KeyValueDecoder:
{code}
+  public boolean next() {
+if (!this.hasNext) return !this.hasNext;
{code}
I think this.hasNext should be returned.

For TestBasicCellCodec:
{code}
+  public void testOne() throws IOException {
...
+  public void testThree() throws IOException {
{code}
Would testOneKeyValue(), testThreeKeyValue() be better names ?
Similar comment for TestCellMessageCodec.testOne()

For ProtobufUtil.java:
{code}
-builder.addKeyValue(toKeyValue(c));
+builder.addKeyValue(toCell(c));
{code}
It would be nice if the method name for builder can be changed to addCell().

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573204#comment-13573204
 ] 

stack commented on HBASE-7233:
--

Thanks for the comments.  First one is good.  Was sort of looking for more 
high-level commentary on whether this a good direction or not.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573210#comment-13573210
 ] 

Ted Yu commented on HBASE-7233:
---

{code}
+public class BasicCellDecoder implements CellScanner {
+public class CellMessageDecoder implements CellScanner {
+public class KeyValueDecoder implements CellScanner {
{code}
The decoders all depend on InputStream#available(). It would be nice if class 
javadoc is added explaining the context where each of them would be used.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HBASE-7765) A tool to replay HLog entries in case a log file is missed while log splitting


 [ 
https://issues.apache.org/jira/browse/HBASE-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Himanshu Vashishtha resolved HBASE-7765.


Resolution: Fixed

 A tool to replay HLog entries in case a log file is missed while log splitting
 --

 Key: HBASE-7765
 URL: https://issues.apache.org/jira/browse/HBASE-7765
 Project: HBase
  Issue Type: New Feature
  Components: wal
Affects Versions: 0.94.4
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha

 There may be a case where a HLog can escape the Log splitting. 
 It will be good to have a standalone tool that reads entries from a HLog, and 
 replays it back to HBase. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573220#comment-13573220
 ] 

stack commented on HBASE-7233:
--

Should have said more plainly that this is experimental/prototyping code for 
ideas mentioned in cited google doc and mentioned elsewhere over in rpc spec 
issue.  Patches are posted for high level does this look right feedback, not 
javadoc could be better or method naming suggestions.  Thanks.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573225#comment-13573225
]

stack commented on HBASE-7723:
--

Yes. I get that. Absolute paths will come back to bite us in various ways.

My point is that a filesystem that is running HA NN should equate to the
filesystem that is running old school NN. Up to this, a change in the root.dir
URL signified 'different filesystem' and we depend on this in loads of places
around the code (Filesystems do not match! type exceptions). Here is a case
where finding a difference in the URL hostname + port does NOT mean a different
fs.

I'm fine w/ log splitter taking relative paths thats good.

Just pointing out that this issue will rear its head again.

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573232#comment-13573232
 ] 

Ted Yu commented on HBASE-7233:
---

I read the google doc linked above. I think the patch reflects what the doc 
describes.
Meaning the direction is good.

RPC spec mentions EncodedDataBlock in several places. I am not sure the spec 
has been updated.
Will go over the spec and patch in HBASE-7533 tomorrow so that I can gain 
better understanding of these two JIRAs.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573237#comment-13573237
]

Ted Yu commented on HBASE-4676:
---

https://builds.apache.org/job/HBase-TRUNK/3859/console shows some test failures
which didn't happen in previous trunk builds.

Will investigate tomorrow morning.

If trunk builds fail consecutively, will consider reverting.

Prefix Compression - Trie data block encoding
-

Key: HBASE-4676
URL: https://issues.apache.org/jira/browse/HBASE-4676
Project: HBase
Issue Type: New Feature
Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
Attachments: 4676-prefix-tree-trunk-v16.patch,
4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch,
HBASE-4676-common-and-server-v8.patch,
HBASE-4676-prefix-tree-trunk-v10.patch,
HBASE-4676-prefix-tree-trunk-v11.patch,
HBASE-4676-prefix-tree-trunk-v12.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v14.patch,
HBASE-4676-prefix-tree-trunk-v15.patch,
HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch,
HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch,
HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch,
HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar,
PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
blockSize.png

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573241#comment-13573241
 ] 

stack commented on HBASE-7233:
--

[~ted_yu] Beware it is a work in progress

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding

[
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573246#comment-13573246
]

Ted Yu commented on HBASE-4676:
---

https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/396/console seems to
be close to normal.

HBase-TRUNK-on-Hadoop-2.0.0 #395 and trunk build #3859 were both on ubuntu1.

Prefix Compression - Trie data block encoding
-

Key: HBASE-4676
URL: https://issues.apache.org/jira/browse/HBASE-4676
Project: HBase
Issue Type: New Feature
Components: io, Performance, regionserver
Affects Versions: 0.96.0
Reporter: Matt Corgan
Assignee: Matt Corgan
Priority: Critical
Attachments: 4676-prefix-tree-trunk-v16.patch,
4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch,
HBASE-4676-common-and-server-v8.patch,
HBASE-4676-prefix-tree-trunk-v10.patch,
HBASE-4676-prefix-tree-trunk-v11.patch,
HBASE-4676-prefix-tree-trunk-v12.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v13.patch,
HBASE-4676-prefix-tree-trunk-v14.patch,
HBASE-4676-prefix-tree-trunk-v15.patch,
HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch,
HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch,
HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch,
HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch,
HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar,
PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by
blockSize.png

[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding


[ 
https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573247#comment-13573247
 ] 

Hudson commented on HBASE-4676:
---

Integrated in HBase-TRUNK #3859 (See 
[https://builds.apache.org/job/HBase-TRUNK/3859/])
HBASE-4676 Prefix Compression - Trie data block encoding (Matt Corgan) 
(Revision 1443289)

 Result = ABORTED
tedyu : 
Files : 
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoding.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteRangeTool.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/test/RedundantKVGenerator.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellComparator.java
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hbase/cell/CellOutputStream.java
* /hbase/trunk/hbase-prefix-tree
* /hbase/trunk/hbase-prefix-tree/pom.xml
* /hbase/trunk/hbase-prefix-tree/src
* /hbase/trunk/hbase-prefix-tree/src/main
* /hbase/trunk/hbase-prefix-tree/src/main/java
* /hbase/trunk/hbase-prefix-tree/src/main/java/org
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec
* /hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeBlockMeta.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeCodec.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PrefixTreeSeeker.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/ArraySearcherPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/DecoderFactory.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArrayReversibleScanner.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeArraySearcher.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/PrefixTreeCell.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column/ColumnNodeReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/column/ColumnReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/row
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/row/RowNodeReader.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp/MvccVersionDecoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/decode/timestamp/TimestampDecoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/EncoderFactory.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/EncoderPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/PrefixTreeEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/ThreadLocalEncoderPool.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column/ColumnNodeWriter.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/column/ColumnSectionWriter.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other/CellTypeEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/other/LongEncoder.java
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/row
* 
/hbase/trunk/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/encode/row/RowNodeWriter.java
*

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573249#comment-13573249
]

Lars Hofhansl commented on HBASE-7723:
--

Sorry, misunderstood your comment.

I guess two FSs could be considered the same if one is a NN ensemble containing
the old pre HA NN.
Then a NN roundtrip would be needed to verify that.

Where else are we storing absolute paths?

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7233) Serializing KeyValues

2013-02-06 Thread Matt Corgan (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573258#comment-13573258
]

Matt Corgan commented on HBASE-7233:

I like it. Nice and simple.

What are your thoughts on BasicCellEncoder vs KeyValueEncoder? Are you
introducing the new BasicCell format because it's easier for a non-java
client to decode?

KeyValueEncoder appears faster in this benchmark because the serialization is
less granular, but I think that will become irrelevant over time if we get the
Cell interface all the way up the read path. It's faster now because you know
the input cell is KeyValue so can just cast. If you don't know the input Cell
implementation, you'd have to append each KeyValue field separately.

Would probably be good to include BasicCellEncoder and KeyValue encoder just to
make sure that the necessary abstractions are in place to add future encodings.
Later on, my guess is one of the delta-style encoders will be best for java
client RPC.

Serializing KeyValues
-

Key: HBASE-7233
URL: https://issues.apache.org/jira/browse/HBASE-7233
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.96.0

Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt,
7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt,
7233v6_encoder.txt, 7233v7.txt, 7233v9.txt

Undo KeyValue being a Writable.

[jira] [Commented] (HBASE-7233) Serializing KeyValues

[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573259#comment-13573259
]

stack commented on HBASE-7233:
--

I think we should close this issue now. Its scope has wandered and we got a
bunch of use out of it but I think its time has come.

The original intent was removing Writable from KV which has happened over in
another issue. How KVs went across MapReduce when not Writable came up and
that got fixed out in other issues also.

We then did a bunch of the back and forth on how to serialize KVs particularly
across RPC.

Most of the discussion has been captured here:

https://docs.google.com/document/d/1WEtrq-JTIUhlnlnvA0oYRLp0F8MKpEBeBSCFcQiacdw/edit#

...and will be realized in code over in the rpc revamp issue, HBASE-7533
(patches added here will reappear in that issue)

Regards Andrew's request regards HBASE-6222 above on being able to pass a
KeyValue version2, a v2 should make
it across the rpc and even into hfiles (after doing some work on
EncodedDataBlocks so they sling Cells instead
of KVs) but what is required will not happen in this issue.

The client and server will need to all be moved to reference the Cell Interface
rather than KV1 as they currently do. Only then could a KV2 traverse the
client and server. That is work to do (As Lars Hofhansl's found out, doing
this often makes for speedups since we
are often realizing KVs when all we need is a piece). Lets take up the effort
to change the servers to be Cell based
rather than KV elsehwere.

The EncodedDataBlocks revamp should happen elsewhere too.

Neither of the above should hold up 0.96 release (A 0.96 client should be able
to talk to
a future server that can do KV version 2).

Should we add anything to the Cell Interface before 0.96 ships; e.g. the
getTagsArray, etc., Matt suggets above for Andrew's tag work? If so, lets get
that in.

Will close in a day or two unless objection.

Serializing KeyValues
-

Key: HBASE-7233
URL: https://issues.apache.org/jira/browse/HBASE-7233
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.96.0

Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt,
7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt,
7233v6_encoder.txt, 7233v7.txt, 7233v9.txt

Undo KeyValue being a Writable.

[jira] [Commented] (HBASE-7233) Serializing KeyValues


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573267#comment-13573267
 ] 

stack commented on HBASE-7233:
--

bq. What are your thoughts on BasicCellEncoder vs KeyValueEncoder? Are you 
introducing the new BasicCell format because it's easier for a non-java 
client to decode?

Thanks [~mcorgan] for review.

I think KVEncoder will be lowest common denominator.  Should include 
BasicCellEncoder too.  The CellMessageCodec was just to see.  Not worth 
including I'd say.  I should look again at BasicCellEncoder.  Might be able to 
make it a bit better.  Your point on KV codec having an unfair advantage 
currently is a indeed the case.  Will move this code over to the rpc issue 
hbase-7533.

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.

[
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573268#comment-13573268
]

stack commented on HBASE-7723:
--

I was thinking of those places out in scripts and tools where we want to get to
hdfs where the config may have the single nn format but we're running the ha nn
-- or vice versa. Should just work rather than complain. I suppose lets just
cross that bridge when we come to it.

Remove NN URI from ZK splitlogs.

[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.


[ 
https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573278#comment-13573278
 ] 

Himanshu Vashishtha commented on HBASE-7723:


Can you point some of those tools/scripts?

 Remove NN URI from ZK splitlogs.
 

 Key: HBASE-7723
 URL: https://issues.apache.org/jira/browse/HBASE-7723
 Project: HBase
  Issue Type: Bug
  Components: hadoop2, master
Affects Versions: 0.92.0
Reporter: Kevin Odell
Assignee: Himanshu Vashishtha
 Fix For: 0.96.0, 0.94.5

 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, 
 HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, 
 HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, 
 HBASE-7723-trunk-v2.patch


 When moving to HDFS HA or removing HA we end up changing the NN namespace.  
 This can cause the HMaster not to start up fully due to trying to split 
 phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: 
 Wrong FS: error messages.  The HLogs in question might not even be on HDFS 
 anymore.  You have to go in a manually clear out the ZK splitlogs directory 
 to get HBase to properly boot up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7233) Serializing KeyValues

2013-02-06 Thread Andrew Purtell (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573283#comment-13573283
 ] 

Andrew Purtell commented on HBASE-7233:
---

I'm good if we have placeholders for tags (HBASE-7448) and can build on that as 
the Cell work takes shape. The first drop for HBASE-6222 can use out of line 
storage for the ACL metadata with some slowdown. 

 Serializing KeyValues
 -

 Key: HBASE-7233
 URL: https://issues.apache.org/jira/browse/HBASE-7233
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
 Fix For: 0.96.0

 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 
 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 
 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt


 Undo KeyValue being a Writable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7233) Serializing KeyValues