[jira] [Commented] (HBASE-7778) [snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due to interrupt flag.
[
https://issues.apache.org/jira/browse/HBASE-7778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573311#comment-13573311
]
nkeywal commented on HBASE-7778:
Yes, I inverted the boolean test in HBASE-7475. Sorry about that, and thanks
for fixing it.
I added the interrupt to prevent the cluster from staying up too long (i.e.
surviving forever the surefire build) while we actually called for a shutdown.
It does not solve all cases, because HBase sometimes messes-up with the
interrupt exception, but it usually helps.
[snapshot 130201 merge] Tests with sleep after minicluster shutdown fail due
to interrupt flag.
---
Key: HBASE-7778
URL: https://issues.apache.org/jira/browse/HBASE-7778
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.96.0, hbase-7290
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.96.0, hbase-7290
Attachments: hbase-7778.patch
Something in the merge has set the interrupted flag on the main test threads
of TestReplicationDisabledinactivePeer, TestRestartCluster, and
TestCatalogTrackerOnCluster.
These unacceptable hacks make the tests run and pass:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
b/hbase-server/src/test/java/or
index f3e57d6..a8d2ef7 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
@@ -47,6 +47,7 @@ public class TestCatalogTrackerOnCluster {
// Shutdown hbase.
UTIL.shutdownMiniHBaseCluster();
// Give the various ZKWatchers some time to settle their affairs.
+Thread.interrupted(); // HACK clear interrupt state.
Thread.sleep(1000);
// Mess with the root location in the running zk. Set it to be nonsense.
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
b/hbase-server/src/test/java/org/apache/h
index 15225e1..9f7f526 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java
@@ -108,6 +108,7 @@ public class TestRestartCluster {
UTIL.shutdownMiniHBaseCluster();
LOG.info(\n\nSleeping a bit);
+Thread.interrupted(); // HACK clear interrupt state.
Thread.sleep(2000);
LOG.info(\n\nStarting cluster the second time);
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
b/hbase-server/src/t
index b089fbe..8162f4b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationDisableInactivePeer.java
@@ -50,6 +50,7 @@ public class TestReplicationDisableInactivePeer extends
TestReplicationBase {
// enabling and shutdown the peer
admin.enablePeer(2);
utility2.shutdownMiniHBaseCluster();
+Thread.interrupted(); // HACK clear interrupted flag.
byte[] rowkey = Bytes.toBytes(disable inactive peer);
Put put = new Put(rowkey);
{code}
On the snapshot branch and on the trunk branch before the merge, these tests
passed. Need to figure out how they combination caused this behavior change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7785) rolling-restart.sh script unable to check expiration of master znode
Samir Ahmic created HBASE-7785: -- Summary: rolling-restart.sh script unable to check expiration of master znode Key: HBASE-7785 URL: https://issues.apache.org/jira/browse/HBASE-7785 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.4 Environment: Linux CentOS release 5.8 Reporter: Samir Ahmic When rolling-restart.sh script stop master it enters loop trying to detect that master znode is deleted. Since it is unable to execute check command script hangs in infinite loop. Problematic line of script is: while ! bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do bin/hbase zkcli stat can not be executed since script is run from bin directory. My suggestion is that this line should be like this in order to work: while ! $bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do After i made this change i was able to execute rolling restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7785) rolling-restart.sh script unable to check expiration of master znode
[ https://issues.apache.org/jira/browse/HBASE-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samir Ahmic updated HBASE-7785: --- Attachment: HBASE-7785.txt Here is diff for change rolling-restart.sh script unable to check expiration of master znode Key: HBASE-7785 URL: https://issues.apache.org/jira/browse/HBASE-7785 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.4 Environment: Linux CentOS release 5.8 Reporter: Samir Ahmic Attachments: HBASE-7785.txt When rolling-restart.sh script stop master it enters loop trying to detect that master znode is deleted. Since it is unable to execute check command script hangs in infinite loop. Problematic line of script is: while ! bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do bin/hbase zkcli stat can not be executed since script is run from bin directory. My suggestion is that this line should be like this in order to work: while ! $bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do After i made this change i was able to execute rolling restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead
[ https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573321#comment-13573321 ] nkeywal commented on HBASE-7462: It's likely better not to mock zookeeper. The best solution would be not to to depend on ZK at all, if it's possible. TestDrainingServer is an integration test. It should be a unit test instead --- Key: HBASE-7462 URL: https://issues.apache.org/jira/browse/HBASE-7462 Project: HBase Issue Type: Wish Components: test Affects Versions: 0.96.0 Reporter: nkeywal Priority: Trivial Labels: noob TestDrainingServer tests the function that allows to say that a regionserver should not get new regions. As it is written today, it's an integration test: it starts stops a cluster. The test would be more efficient if it would just check that the AssignmentManager does not use the drained region server; whatever the circumstances (bulk assign or not for example). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7786) Consolidate HRegion creation/opening API
Matteo Bertozzi created HBASE-7786: -- Summary: Consolidate HRegion creation/opening API Key: HBASE-7786 URL: https://issues.apache.org/jira/browse/HBASE-7786 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Currently we have 4 ways to instantiate an HRegion. * HRegion.createHRegion() * HRegion.openHRegion() * HRegion.newHRegion() + r.initialize() * new HRegion() + r.initialize() Aside from tests and HMerge and SplitTransaction code everyone use createHRegion() and openHRegion(). To avoid errors due to missing initialization calls I think we should limit the access to newHRegion() and the constructor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7462) TestDrainingServer is an integration test. It should be a unit test instead
[ https://issues.apache.org/jira/browse/HBASE-7462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573404#comment-13573404 ] Gustavo Anatoly commented on HBASE-7462: Ok Nicolas,I will follow your suggestion. But the reasons do not use all ZK mocked, it's because test focus is the AssignmentManager, call only especific pieces of codes of ZK? Therefore only mock ZK needed by AssignmentManager. Is correct? Thanks for help. TestDrainingServer is an integration test. It should be a unit test instead --- Key: HBASE-7462 URL: https://issues.apache.org/jira/browse/HBASE-7462 Project: HBase Issue Type: Wish Components: test Affects Versions: 0.96.0 Reporter: nkeywal Priority: Trivial Labels: noob TestDrainingServer tests the function that allows to say that a regionserver should not get new regions. As it is written today, it's an integration test: it starts stops a cluster. The test would be more efficient if it would just check that the AssignmentManager does not use the drained region server; whatever the circumstances (bulk assign or not for example). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7787) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
[ https://issues.apache.org/jira/browse/HBASE-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7787: --- Status: Patch Available (was: Open) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR - Key: HBASE-7787 URL: https://issues.apache.org/jira/browse/HBASE-7787 Project: HBase Issue Type: Bug Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-7787-v0.patch HLogSplitter.RECOVERED_EDITS seems to be used only in TestHLogSplit, and has the exact same meaning of HLog.RECOVERED_EDITS_DIR -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7787) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
[ https://issues.apache.org/jira/browse/HBASE-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7787: --- Attachment: HBASE-7787-v0.patch Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR - Key: HBASE-7787 URL: https://issues.apache.org/jira/browse/HBASE-7787 Project: HBase Issue Type: Bug Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-7787-v0.patch HLogSplitter.RECOVERED_EDITS seems to be used only in TestHLogSplit, and has the exact same meaning of HLog.RECOVERED_EDITS_DIR -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7787) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
Matteo Bertozzi created HBASE-7787: -- Summary: Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR Key: HBASE-7787 URL: https://issues.apache.org/jira/browse/HBASE-7787 Project: HBase Issue Type: Bug Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-7787-v0.patch HLogSplitter.RECOVERED_EDITS seems to be used only in TestHLogSplit, and has the exact same meaning of HLog.RECOVERED_EDITS_DIR -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7365) Safer table creation and deletion using .tmp dir
[ https://issues.apache.org/jira/browse/HBASE-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573498#comment-13573498 ] Jonathan Hsieh commented on HBASE-7365: --- In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be -- isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different -- creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. Safer table creation and deletion using .tmp dir Key: HBASE-7365 URL: https://issues.apache.org/jira/browse/HBASE-7365 Project: HBase Issue Type: Improvement Components: master Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: 7365-v4.patch, HBASE-7365-v0.patch, HBASE-7365-v1.patch, HBASE-7365-v2.patch, HBASE-7365-v3.patch Currently tables are created in the root directory, and the removal works on the root directory. Change the code to use a /hbase/.tmp directory to make the creation and removal a bit safer Table Creation steps * Create the table descriptor (table folder, in /hbase/.tmp/) * Create the table regions (always in temp) * Move the table from temp to the root folder * Add the regions to meta * Trigger assignment * Set enable flag in ZooKeeper Table Deletion steps * Wait for regions in transition * Remove regions from meta (use bulk delete) * Move the table in /hbase/.tmp * Remove the table from the descriptor cache * Remove table from zookeeper * Archive the table The main changes in the current code are: * Writing to /hbase/.tmp and then rename * using bulk delete in DeletionTableHandler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7365) Safer table creation and deletion using .tmp dir
[ https://issues.apache.org/jira/browse/HBASE-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573514#comment-13573514 ] Jonathan Hsieh commented on HBASE-7365: --- I'm going to file the fix to this problem as a new issue. Since this patch has laned in the snapshot branch and not landed on trunk yet, I'm going to suggest we close this issue as a subissue of the snapshot branch. Safer table creation and deletion using .tmp dir Key: HBASE-7365 URL: https://issues.apache.org/jira/browse/HBASE-7365 Project: HBase Issue Type: Improvement Components: master Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: 7365-v4.patch, HBASE-7365-v0.patch, HBASE-7365-v1.patch, HBASE-7365-v2.patch, HBASE-7365-v3.patch Currently tables are created in the root directory, and the removal works on the root directory. Change the code to use a /hbase/.tmp directory to make the creation and removal a bit safer Table Creation steps * Create the table descriptor (table folder, in /hbase/.tmp/) * Create the table regions (always in temp) * Move the table from temp to the root folder * Add the regions to meta * Trigger assignment * Set enable flag in ZooKeeper Table Deletion steps * Wait for regions in transition * Remove regions from meta (use bulk delete) * Move the table in /hbase/.tmp * Remove the table from the descriptor cache * Remove table from zookeeper * Archive the table The main changes in the current code are: * Writing to /hbase/.tmp and then rename * using bulk delete in DeletionTableHandler -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7787) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
[
https://issues.apache.org/jira/browse/HBASE-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573515#comment-13573515
]
Hadoop QA commented on HBASE-7787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568407/HBASE-7787-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestAtomicOperation
org.apache.hadoop.hbase.util.TestHBaseFsck
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4364//console
This message is automatically generated.
Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
-
Key: HBASE-7787
URL: https://issues.apache.org/jira/browse/HBASE-7787
Project: HBase
Issue Type: Bug
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Attachments: HBASE-7787-v0.patch
HLogSplitter.RECOVERED_EDITS seems to be used only in TestHLogSplit, and has
the exact same meaning of HLog.RECOVERED_EDITS_DIR
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
Jonathan Hsieh created HBASE-7788: - Summary: [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[ https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HBASE-7788: - Assignee: Jonathan Hsieh [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot --- Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files
[ https://issues.apache.org/jira/browse/HBASE-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7783: -- Attachment: hbase-7783.patch don't look at the patch -- its pretty silly. [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files - Key: HBASE-7783 URL: https://issues.apache.org/jira/browse/HBASE-7783 Project: HBase Issue Type: Sub-task Components: snapshots Affects Versions: hbase-7290 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-7290 Attachments: hbase-7783.patch Apparently, I checked in a whole bunch of binary .class files into hbase-server/bin directory. I'm just going to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files
[ https://issues.apache.org/jira/browse/HBASE-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7783: -- Attachment: hbase-7783.summary Look at hbase-7783.summary for the gist of the patch. [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files - Key: HBASE-7783 URL: https://issues.apache.org/jira/browse/HBASE-7783 Project: HBase Issue Type: Sub-task Components: snapshots Affects Versions: hbase-7290 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-7290 Attachments: hbase-7783.patch, hbase-7783.summary Apparently, I checked in a whole bunch of binary .class files into hbase-server/bin directory. I'm just going to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files
[ https://issues.apache.org/jira/browse/HBASE-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7783: -- Status: Patch Available (was: Open) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files - Key: HBASE-7783 URL: https://issues.apache.org/jira/browse/HBASE-7783 Project: HBase Issue Type: Sub-task Components: snapshots Affects Versions: hbase-7290 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: hbase-7290 Attachments: hbase-7783.patch, hbase-7783.summary Apparently, I checked in a whole bunch of binary .class files into hbase-server/bin directory. I'm just going to remove them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[ https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7788: -- Attachment: hbase-7788.patch Nice and simple patch. [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot --- Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7788.patch In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[ https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7788: -- Status: Patch Available (was: Open) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot --- Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7788.patch In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[
https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573529#comment-13573529
]
Hadoop QA commented on HBASE-7788:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568416/hbase-7788.debug.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4365//console
This message is automatically generated.
[snapshot 130201 merge] Fix flakey
TestRestore*SnapshotFromClient#testCloneSnapshot
---
Key: HBASE-7788
URL: https://issues.apache.org/jira/browse/HBASE-7788
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7788.debug.patch, hbase-7788.patch
In the current implementation the clone call waits until after the newly
created table isTableEnabled. However there is another state (apparently
orthogonal) that a newly created table is assumed to be – isTableAvailable
(all regions assigned). The logic for checking after table creation and after
clone creation are slightly different – creation does the equivalent of
isTableAvailable but clone does not check this availability condition.
This causes flaky failures in tests that quickly try to use/delete a newly
cloned table.
TestRestoreSnapshotFromClient#testCloneSnapshot
TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot
I believe there also are race conditions because of the
postTableCreateHandler corpco and postTableDeleteHandler coproc hooks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[ https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7788: -- Attachment: hbase-7788.debug.patch the hbase-7788.debug.patch file adds some debugging information that is not intended for commit, but was useful for diagnosing the problem. [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot --- Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7788.debug.patch, hbase-7788.patch In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7783) [snapshot 130201 merge] Remove accidentally committed hbase-server/bin files
[
https://issues.apache.org/jira/browse/HBASE-7783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573530#comment-13573530
]
Hadoop QA commented on HBASE-7783:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568414/hbase-7783.summary
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 836
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4366//console
This message is automatically generated.
[snapshot 130201 merge] Remove accidentally committed hbase-server/bin files
-
Key: HBASE-7783
URL: https://issues.apache.org/jira/browse/HBASE-7783
Project: HBase
Issue Type: Sub-task
Components: snapshots
Affects Versions: hbase-7290
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: hbase-7290
Attachments: hbase-7783.patch, hbase-7783.summary
Apparently, I checked in a whole bunch of binary .class files into
hbase-server/bin directory. I'm just going to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[ https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573531#comment-13573531 ] Matteo Bertozzi commented on HBASE-7788: yeah, it looks simple :) +1 on the patch [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot --- Key: HBASE-7788 URL: https://issues.apache.org/jira/browse/HBASE-7788 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-7788.debug.patch, hbase-7788.patch In the current implementation the clone call waits until after the newly created table isTableEnabled. However there is another state (apparently orthogonal) that a newly created table is assumed to be – isTableAvailable (all regions assigned). The logic for checking after table creation and after clone creation are slightly different – creation does the equivalent of isTableAvailable but clone does not check this availability condition. This causes flaky failures in tests that quickly try to use/delete a newly cloned table. TestRestoreSnapshotFromClient#testCloneSnapshot TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot I believe there also are race conditions because of the postTableCreateHandler corpco and postTableDeleteHandler coproc hooks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7788) [snapshot 130201 merge] Fix flakey TestRestore*SnapshotFromClient#testCloneSnapshot
[
https://issues.apache.org/jira/browse/HBASE-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573539#comment-13573539
]
Hadoop QA commented on HBASE-7788:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568416/hbase-7788.debug.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4367//console
This message is automatically generated.
[snapshot 130201 merge] Fix flakey
TestRestore*SnapshotFromClient#testCloneSnapshot
---
Key: HBASE-7788
URL: https://issues.apache.org/jira/browse/HBASE-7788
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: hbase-7788.debug.patch, hbase-7788.patch
In the current implementation the clone call waits until after the newly
created table isTableEnabled. However there is another state (apparently
orthogonal) that a newly created table is assumed to be – isTableAvailable
(all regions assigned). The logic for checking after table creation and after
clone creation are slightly different – creation does the equivalent of
isTableAvailable but clone does not check this availability condition.
This causes flaky failures in tests that quickly try to use/delete a newly
cloned table.
TestRestoreSnapshotFromClient#testCloneSnapshot
TestRestoreFlushSnapshotFromCleitn#testCloneSnapshot
I believe there also are race conditions because of the
postTableCreateHandler corpco and postTableDeleteHandler coproc hooks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7786) Consolidate HRegion creation/opening API
[ https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7786: --- Attachment: HBASE-7786-v0.patch Consolidate HRegion creation/opening API Key: HBASE-7786 URL: https://issues.apache.org/jira/browse/HBASE-7786 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-7786-v0.patch Currently we have 4 ways to instantiate an HRegion. * HRegion.createHRegion() * HRegion.openHRegion() * HRegion.newHRegion() + r.initialize() * new HRegion() + r.initialize() Aside from tests and HMerge and SplitTransaction code everyone use createHRegion() and openHRegion(). To avoid errors due to missing initialization calls I think we should limit the access to newHRegion() and the constructor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7786) Consolidate HRegion creation/opening API
[ https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7786: --- Status: Patch Available (was: Open) Consolidate HRegion creation/opening API Key: HBASE-7786 URL: https://issues.apache.org/jira/browse/HBASE-7786 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-7786-v0.patch Currently we have 4 ways to instantiate an HRegion. * HRegion.createHRegion() * HRegion.openHRegion() * HRegion.newHRegion() + r.initialize() * new HRegion() + r.initialize() Aside from tests and HMerge and SplitTransaction code everyone use createHRegion() and openHRegion(). To avoid errors due to missing initialization calls I think we should limit the access to newHRegion() and the constructor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573548#comment-13573548 ] Matteo Bertozzi commented on HBASE-4676: Is there a reason why the prefix-tree and cell code is inside org.apache.hbase instead of the traditional org.apache.hadoop.hbase? Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, Performance, regionserver Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Priority: Critical Attachments: 4676-prefix-tree-trunk-v16.patch, 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch, HBASE-4676-prefix-tree-trunk-v10.patch, HBASE-4676-prefix-tree-trunk-v11.patch, HBASE-4676-prefix-tree-trunk-v12.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v14.patch, HBASE-4676-prefix-tree-trunk-v15.patch, HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version might build on this by also applying further (suffix, etc) compression on the trie nodes at the cost of slower write speed. Even further compression could be obtained by using all VInts instead of FInts with a sacrifice on random seek speed (though not huge). One current drawback is the current write speed. While programmed with good constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not programmed with the same level of optimization as the read path. Work will need to be done to optimize the data structures used for encoding and could probably show a 10x increase. It
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573605#comment-13573605
]
Hadoop QA commented on HBASE-7786:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568418/HBASE-7786-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4368//console
This message is automatically generated.
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7789) Clean DeadServer.java and add a Jitter method in ConnectionUtils
nkeywal created HBASE-7789: -- Summary: Clean DeadServer.java and add a Jitter method in ConnectionUtils Key: HBASE-7789 URL: https://issues.apache.org/jira/browse/HBASE-7789 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 I need to do some changes in DeadServer because of HBASE-7390. To minimize the patch size and simplifies the feedback, I prefer to isolate the issue. Changes are: - Add the time when the server was declared as dead. It's what I need in HBASE-7390, but it makes sense even without it, for example to be shown in the UI. - suppress the extends on Set clean up all the not used methods - use directly the object instead of a copy. For connection utils, we currently have a jitter of 1%. I need a bigger one for sure in one case, but I wonder if we should not increase it in all cases? instead of plus 1%, we should have plus or minus 10% imho. Tests are in progress locally, I will add the patch when they're ok. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573631#comment-13573631 ] stack commented on HBASE-7723: -- Do a 'grep -r makeQualified' under hbase checkout. This is us filling out the full path to a file (including scheme, etc.). That should lead you to compares of paths that factor in the filesystem scheme, host and port. Let me know if you need more pointers H. Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.5 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7790) Refactor OpenRegionHandler so that the cleanup happens in one place - the finally block
ramkrishna.s.vasudevan created HBASE-7790: - Summary: Refactor OpenRegionHandler so that the cleanup happens in one place - the finally block Key: HBASE-7790 URL: https://issues.apache.org/jira/browse/HBASE-7790 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 This is based on discussion in HBASE-7698. Jimmy suggested this improvment. Look at https://issues.apache.org/jira/browse/HBASE-7698?focusedCommentId=13572736page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13572736 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck
[ https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573649#comment-13573649 ] ramkrishna.s.vasudevan commented on HBASE-7698: --- Raised HBASE-7790 for the improvement. race between RS shutdown thread and openregionhandler causes region to get stuck Key: HBASE-7698 URL: https://issues.apache.org/jira/browse/HBASE-7698 Project: HBase Issue Type: Bug Affects Versions: 0.94.4 Reporter: Sergey Shelukhin Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch 2013-01-22 17:59:03,237 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08 ... 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): Closing IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.: disabling compactions amp; flushes 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): Updates disabled for region IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. 2013-01-22 17:59:03,415 ERROR [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): Caught throwable while processing event M_RS_OPEN_REGION java.io.IOException: java.io.IOException: java.io.IOException: Filesystem closed at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never called and region can get stuck. As an added benefit, the meta is already written by that time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6073) Add support for scan filters in Thrift2
[ https://issues.apache.org/jira/browse/HBASE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Polzunov updated HBASE-6073: --- Tags: Thrift2 Scan Get Filters (was: Thrift2 Scan Filters) Release Note: The patch adds support for server-side HBase filters to TScan and TGet Status: Patch Available (was: Open) This patch lacks unit test code just because trunk uses some new features that are not available in Cloudera distribution release cdh4.1.2 Add support for scan filters in Thrift2 --- Key: HBASE-6073 URL: https://issues.apache.org/jira/browse/HBASE-6073 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.94.0 Reporter: Jay Talreja Labels: thrift2 With HBase 0.94 a new thrift API was added (thrift2). This API is more akin to the Java HBase API. Thrift (version1) had added filterString to the TScan struct as part of HBase release 0.92 . Thrift2 TScan object doesn't have filterString parameter. Hence executing server side filters using thrift2 API is currently not possible. It would be great to have filtering capabilities added to TScan struct in thrift2 as well to maintain feature compatibility between two thrift versions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6073) Add support for scan filters in Thrift2
[ https://issues.apache.org/jira/browse/HBASE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Polzunov updated HBASE-6073: --- Attachment: hbase-thrift2-filters-src.patch hbase-thrift2-filters-scheme.patch Add support for scan filters in Thrift2 --- Key: HBASE-6073 URL: https://issues.apache.org/jira/browse/HBASE-6073 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.94.0 Reporter: Jay Talreja Labels: thrift2 Attachments: hbase-thrift2-filters-scheme.patch, hbase-thrift2-filters-src.patch With HBase 0.94 a new thrift API was added (thrift2). This API is more akin to the Java HBase API. Thrift (version1) had added filterString to the TScan struct as part of HBase release 0.92 . Thrift2 TScan object doesn't have filterString parameter. Hence executing server side filters using thrift2 API is currently not possible. It would be great to have filtering capabilities added to TScan struct in thrift2 as well to maintain feature compatibility between two thrift versions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573668#comment-13573668 ] Lars Hofhansl commented on HBASE-7723: -- Let's cross that bridge when needed I'd say. I think is only an issue where we store fully qualified paths outside of HDFS (such as in ZK). Thinking now maybe we should check the replication code's usage of ZK (although it is not immediately clear how that *should* behave). Should I push to 0.94.6, since we're still discussing? Maybe at least document? (It did cost the Salesforce Ops folks some gray hair in a late night session, before they finally called me up). Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.5 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7786) Consolidate HRegion creation/opening API
[ https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7786: --- Attachment: HBASE-7786-v1.patch Consolidate HRegion creation/opening API Key: HBASE-7786 URL: https://issues.apache.org/jira/browse/HBASE-7786 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch Currently we have 4 ways to instantiate an HRegion. * HRegion.createHRegion() * HRegion.openHRegion() * HRegion.newHRegion() + r.initialize() * new HRegion() + r.initialize() Aside from tests and HMerge and SplitTransaction code everyone use createHRegion() and openHRegion(). To avoid errors due to missing initialization calls I think we should limit the access to newHRegion() and the constructor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7785) rolling-restart.sh script unable to check expiration of master znode
[ https://issues.apache.org/jira/browse/HBASE-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7785: - Fix Version/s: 0.94.5 0.96.0 rolling-restart.sh script unable to check expiration of master znode Key: HBASE-7785 URL: https://issues.apache.org/jira/browse/HBASE-7785 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.4 Environment: Linux CentOS release 5.8 Reporter: Samir Ahmic Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7785.txt When rolling-restart.sh script stop master it enters loop trying to detect that master znode is deleted. Since it is unable to execute check command script hangs in infinite loop. Problematic line of script is: while ! bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do bin/hbase zkcli stat can not be executed since script is run from bin directory. My suggestion is that this line should be like this in order to work: while ! $bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do After i made this change i was able to execute rolling restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7785) rolling-restart.sh script unable to check expiration of master znode
[ https://issues.apache.org/jira/browse/HBASE-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573681#comment-13573681 ] Lars Hofhansl commented on HBASE-7785: -- Yep. We do this everywhere else in this script. This is the only exception. Going to commit in a few. rolling-restart.sh script unable to check expiration of master znode Key: HBASE-7785 URL: https://issues.apache.org/jira/browse/HBASE-7785 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.4 Environment: Linux CentOS release 5.8 Reporter: Samir Ahmic Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7785.txt When rolling-restart.sh script stop master it enters loop trying to detect that master znode is deleted. Since it is unable to execute check command script hangs in infinite loop. Problematic line of script is: while ! bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do bin/hbase zkcli stat can not be executed since script is run from bin directory. My suggestion is that this line should be like this in order to work: while ! $bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do After i made this change i was able to execute rolling restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck
[ https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573683#comment-13573683 ] Jimmy Xiang commented on HBASE-7698: @Ram, sure, +1 for commit. race between RS shutdown thread and openregionhandler causes region to get stuck Key: HBASE-7698 URL: https://issues.apache.org/jira/browse/HBASE-7698 Project: HBase Issue Type: Bug Affects Versions: 0.94.4 Reporter: Sergey Shelukhin Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch 2013-01-22 17:59:03,237 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08 ... 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): Closing IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.: disabling compactions amp; flushes 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): Updates disabled for region IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. 2013-01-22 17:59:03,415 ERROR [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): Caught throwable while processing event M_RS_OPEN_REGION java.io.IOException: java.io.IOException: java.io.IOException: Filesystem closed at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never called and region can get stuck. As an added benefit, the meta is already written by that time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7741) Don't use bulk assigner if assigning just several regions
[ https://issues.apache.org/jira/browse/HBASE-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-7741: --- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Stack and Ram for the review. Don't use bulk assigner if assigning just several regions - Key: HBASE-7741 URL: https://issues.apache.org/jira/browse/HBASE-7741 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: trunk-7741.patch, trunk-7741_v2.patch, trunk-7741_v3_1.patch, trunk-7741_v3.patch If just assign one region, bulk assigner may be slower. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6782) HBase shell's 'status 'detailed'' should escape the printed keys
[ https://issues.apache.org/jira/browse/HBASE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6782: --- Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated into trunk. Thanks Viji for the patch. HBase shell's 'status 'detailed'' should escape the printed keys Key: HBASE-6782 URL: https://issues.apache.org/jira/browse/HBASE-6782 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.90.1 Reporter: Viji Assignee: Viji Priority: Minor Fix For: 0.96.0 Attachments: HBASE-6782.patch Currently the HBase shell's status command prints unescaped keys on the terminal causing the terminal to print garbage characters. We should escape the printed keys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573702#comment-13573702
]
Sergey Shelukhin commented on HBASE-7786:
-
{quote}
+ * Used only by tests and SplitTransaction to reopen the region.
+ * You should use createHRegion() or openHRegion()
{quote}
Is it possible to get rid of it instead? It just does redirect to different
method, might as well have it in the tests themselves.
{quote}
- /**
+ /*
{quote}
Is this intentional?
{quote}
HRegion createDaughterRegion(final HRegionInfo hri, final Path daughterTmpDir)
{quote}
Is this used anywhere except for the other overload?
Otherwise looks reasonable.
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[ https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573703#comment-13573703 ] Sergey Shelukhin commented on HBASE-7786: - ...assuming the tests pass :) Consolidate HRegion creation/opening API Key: HBASE-7786 URL: https://issues.apache.org/jira/browse/HBASE-7786 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch Currently we have 4 ways to instantiate an HRegion. * HRegion.createHRegion() * HRegion.openHRegion() * HRegion.newHRegion() + r.initialize() * new HRegion() + r.initialize() Aside from tests and HMerge and SplitTransaction code everyone use createHRegion() and openHRegion(). To avoid errors due to missing initialization calls I think we should limit the access to newHRegion() and the constructor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Periodically flush the Memstore?
[
https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573705#comment-13573705
]
Hadoop QA commented on HBASE-5930:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568108/5930-2.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestAccessControlFilter
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4369//console
This message is automatically generated.
Periodically flush the Memstore?
Key: HBASE-5930
URL: https://issues.apache.org/jira/browse/HBASE-5930
Project: HBase
Issue Type: Improvement
Reporter: Lars Hofhansl
Assignee: Devaraj Das
Priority: Minor
Fix For: 0.96.0
Attachments: 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch,
5930-2.3.patch, 5930-wip.patch
A colleague of mine ran into an interesting issue.
He inserted some data with the WAL disabled, which happened to fit in the
aggregate Memstores memory.
Two weeks later he a had problem with the HDFS cluster, which caused the
region servers to abort. He found that his data was lost. Looking at the log
we found that the Memstores were not flushed at all during these two weeks.
Should we have an option to flush memstores periodically. There are obvious
downsides to this, like many small storefiles, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.
[ https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573713#comment-13573713 ] Sergey Shelukhin commented on HBASE-7763: - There's also a third thing, you can pick up more files, that way you get read perf, low write amplification, but bursty compactions. Which is also the problem for many people as far as I know, since they run major compactions manually. Agree with all the improvements above being configurable except sorting, for sorting I am not sure. I am usually in favor of allowing people to improve perf at the risk of shooting themselves in the foot, so as long as we make it very explicit that they will get less guarantees than the refguide says, maybe it should be configurable too. Not certain how important this guarantee is. Compactions not sorting based on size anymore. -- Key: HBASE-7763 URL: https://issues.apache.org/jira/browse/HBASE-7763 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.96.0, 0.94.4 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch Currently compaction selection is not sorting based on size. This causes selection to choose larger files to re-write than are needed when bulk loads are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573715#comment-13573715
]
Matteo Bertozzi commented on HBASE-7786:
{quote}
{code}
Used only by tests and SplitTransaction to reopen the region.
You should use createHRegion() or openHRegion()
{code}
Is it possible to get rid of it instead? It just does redirect to different
method, might as well have it in the tests themselves.
{quote}
Nah... the idea was naving initialize() private, since the user should not know
about that... but a couple of tests uses the constructor new HRegion() {
@override method() } to slightly change the behaviour... I guess I can mock it,
or set the REGION_IMPL but it makes the test less readable.
{quote}
HRegion createDaughterRegion(final HRegionInfo hri, final Path daughterTmpDir)
Is this used anywhere except for the other overload?
{quote}
This is used only by the split code, and inside HRegion just to make
newHRegion() private
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7290) Online snapshots
[
https://issues.apache.org/jira/browse/HBASE-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573720#comment-13573720
]
Ted Yu commented on HBASE-7290:
---
ZKProcedureUtil has a reference to ZooKeeperWatcher.
This reference is obtained in two places.
1. SnapshotManager ctor:
{code}
ProcedureCoordinatorRpcs comms = new ZKProcedureCoordinatorRpcs(
master.getZooKeeper(),
SnapshotManager.ONLINE_SNAPSHOT_CONTROLLER_DESCRIPTION, name);
{code}
2. RegionServerSnapshotManager ctor:
{code}
ZooKeeperWatcher zkw = rss.getZooKeeper();
String nodeName = rss.getServerName().toString();
this.memberRpcs = new ZKProcedureMemberRpcs(zkw,
SnapshotManager.ONLINE_SNAPSHOT_CONTROLLER_DESCRIPTION, nodeName);
{code}
This means the watcher would be closed when master / region server goes down.
Looks like ZKProcedureUtil.close() doesn't have to call watcher.close()
Online snapshots
-
Key: HBASE-7290
URL: https://issues.apache.org/jira/browse/HBASE-7290
Project: HBase
Issue Type: Bug
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
HBASE-6055 will be closed when the offline snapshots pieces get merged with
trunk. This umbrella issue has all the online snapshot specific patches.
This will get merged once one of the implementations makes it into trunk.
Other flavors of online snapshots can then be done as normal patches instead
of on a development branch. (was: HBASE-6055 will be closed when the online
snapshots pieces get merged with trunk. This umbrella issue has all the
online snapshot specific patches. This will get merged once one of the
implementations makes it into trunk. Other flavors of online snapshots can
then be done as normal patches instead of on a development branch.)
(not a fan of the quick edit descirption jira feature)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7764) [snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
[
https://issues.apache.org/jira/browse/HBASE-7764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573723#comment-13573723
]
Ted Yu commented on HBASE-7764:
---
I applied Enis' HBASE-7721 patch to see if the new split logic makes a
difference.
TestSnapshotCloneIndependence still failed.
Specifying ConstantSizeRegionSplitPolicy made the test pass.
[snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
-
Key: HBASE-7764
URL: https://issues.apache.org/jira/browse/HBASE-7764
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 7764.txt, 7764-v2.txt
Here is log snippet for
TestSnapshotCloneIndependence#testOnlineSnapshotRegionOperationsIndependent
(pay attention to region 1360020297284.2e43e47a882d3cff601eb222cad41f20.):
{code}
2013-02-04 15:24:58,369 INFO
[MASTER_SERVER_OPERATIONS-10.11.2.194,61955,1360020289835-0]
handler.SplitRegionHandler(115): Handled SPLIT event;
parent=test1360020295791,,1360020295793.794d37c0445b61619b5056623228827d.
daughter
a=test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.daughter
b=test1360020295791,dgb,1360020297284.b87834cee60702d883aa287df6aaeaef.
...
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(78): Starting region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(81): Flush Snapshotting
region test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
started...
2013-02-04 15:25:02,005 DEBUG [member:
apos;10.11.2.194,61958,1360020290064apos; subprocedure-pool3-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(318): Completed
2/3 local region snapshots.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2485): Storing region-info for snapshot.
2013-02-04 15:25:02,006 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
util.FSUtils(166): Creating
file=hdfs://localhost:61942/user/tyu/hbase/.snapshot/.tmp/snapshot_test1360020295791/2e43e47a882d3cff601eb222cad41f20/.tmp/.regioninfo
with permission=rwxrwxrwx
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2489): Creating references for hfiles
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2502): Adding snapshot references for
[hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02]
hfiles
2013-02-04 15:25:02,015 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2516): Creating reference for file (1/1) :
hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(84): ... Flush
Snapshotting region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. completed.
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(86): Closing region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
...
2013-02-04 15:25:03,836 INFO [PRI IPC Server handler 7 on 61958]
regionserver.HRegionServer(3600): Splitting
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.HStore(1686): cannot split because midkey is the same as first
or last row
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.CompactSplitThread(172): Region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. not
splittable because midkey=null
2013-02-04 15:25:03,837 DEBUG [main] catalog.CatalogTracker(231): Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5c602d9d
2013-02-04 15:25:03,838 INFO [main]
client.TestSnapshotCloneIndependence(307): split requested for
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:03,892 DEBUG [main] client.MetaScanner(199): Scanning .META.
starting at row=test1360020295791,,00 for max=2147483647 rows
using hconnection 0x53d9f80
2013-02-04 15:25:03,895 DEBUG [main]
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573729#comment-13573729
]
stack commented on HBASE-7786:
--
Thanks for doing housekeeping.
{code}
+ * Used only by tests and SplitTransaction to reopen the region.
+ * You should use createHRegion() or openHRegion()
{code}
Deprecate if you are not going to remove as Sergey suggests?
+1
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Corgan updated HBASE-4676: --- Release Note: Eperimental: please use on non-critical data for now. PREFIX_TREE encoder stores row keys in a trie layout and uses other encoding techniques for the cells inside a row. It provides similar memory savings to existing encoders, but has faster random access at a cost of slower encoding speed. Set encoding = PREFIX_TREE to enable for a column family. Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, Performance, regionserver Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Priority: Critical Attachments: 4676-prefix-tree-trunk-v16.patch, 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch, HBASE-4676-prefix-tree-trunk-v10.patch, HBASE-4676-prefix-tree-trunk-v11.patch, HBASE-4676-prefix-tree-trunk-v12.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v14.patch, HBASE-4676-prefix-tree-trunk-v15.patch, HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version might build on this by also applying further (suffix, etc) compression on the trie nodes at the cost of slower write speed. Even further compression could be obtained by using all VInts instead of FInts with a sacrifice on random seek speed (though not huge). One current drawback is the current write speed. While programmed with good constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not programmed with the
[jira] [Updated] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4676: -- Fix Version/s: 0.96.0 Release Note: Experimental: please use on non-critical data for now. PREFIX_TREE encoder stores row keys in a trie layout and uses other encoding techniques for the cells inside a row. It provides similar memory savings to existing encoders, but has faster random access at a cost of slower encoding speed. Set encoding = PREFIX_TREE to enable for a column family. (was: Eperimental: please use on non-critical data for now. PREFIX_TREE encoder stores row keys in a trie layout and uses other encoding techniques for the cells inside a row. It provides similar memory savings to existing encoders, but has faster random access at a cost of slower encoding speed. Set encoding = PREFIX_TREE to enable for a column family.) Hadoop Flags: Reviewed Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, Performance, regionserver Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Priority: Critical Fix For: 0.96.0 Attachments: 4676-prefix-tree-trunk-v16.patch, 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch, HBASE-4676-prefix-tree-trunk-v10.patch, HBASE-4676-prefix-tree-trunk-v11.patch, HBASE-4676-prefix-tree-trunk-v12.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v14.patch, HBASE-4676-prefix-tree-trunk-v15.patch, HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version
[jira] [Updated] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck
[ https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-7698: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review Sergey, Jimmy and Lars. Committed to trunk and 0.94. race between RS shutdown thread and openregionhandler causes region to get stuck Key: HBASE-7698 URL: https://issues.apache.org/jira/browse/HBASE-7698 Project: HBase Issue Type: Bug Affects Versions: 0.94.4 Reporter: Sergey Shelukhin Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch 2013-01-22 17:59:03,237 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08 ... 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): Closing IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.: disabling compactions amp; flushes 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): Updates disabled for region IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. 2013-01-22 17:59:03,415 ERROR [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): Caught throwable while processing event M_RS_OPEN_REGION java.io.IOException: java.io.IOException: java.io.IOException: Filesystem closed at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never called and region can get stuck. As an added benefit, the meta is already written by that time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4676) Prefix Compression - Trie data block encoding
[ https://issues.apache.org/jira/browse/HBASE-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4676: -- Resolution: Fixed Status: Resolved (was: Patch Available) Prefix Compression - Trie data block encoding - Key: HBASE-4676 URL: https://issues.apache.org/jira/browse/HBASE-4676 Project: HBase Issue Type: New Feature Components: io, Performance, regionserver Affects Versions: 0.96.0 Reporter: Matt Corgan Assignee: Matt Corgan Priority: Critical Fix For: 0.96.0 Attachments: 4676-prefix-tree-trunk-v16.patch, 4676-prefix-tree-trunk-v17.patch, HBASE-4676-0.94-v1.patch, HBASE-4676-common-and-server-v8.patch, HBASE-4676-prefix-tree-trunk-v10.patch, HBASE-4676-prefix-tree-trunk-v11.patch, HBASE-4676-prefix-tree-trunk-v12.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v13.patch, HBASE-4676-prefix-tree-trunk-v14.patch, HBASE-4676-prefix-tree-trunk-v15.patch, HBASE-4676-prefix-tree-trunk-v1.patch, HBASE-4676-prefix-tree-trunk-v2.patch, HBASE-4676-prefix-tree-trunk-v3.patch, HBASE-4676-prefix-tree-trunk-v4.patch, HBASE-4676-prefix-tree-trunk-v5.patch, HBASE-4676-prefix-tree-trunk-v6.patch, HBASE-4676-prefix-tree-trunk-v7.patch, HBASE-4676-prefix-tree-trunk-v8.patch, HBASE-4676-prefix-tree-trunk-v9.patch, hbase-prefix-trie-0.1.jar, PrefixTrie_Format_v1.pdf, PrefixTrie_Performance_v1.pdf, SeeksPerSec by blockSize.png The HBase data block format has room for 2 significant improvements for applications that have high block cache hit ratios. First, there is no prefix compression, and the current KeyValue format is somewhat metadata heavy, so there can be tremendous memory bloat for many common data layouts, specifically those with long keys and short values. Second, there is no random access to KeyValues inside data blocks. This means that every time you double the datablock size, average seek time (or average cpu consumption) goes up by a factor of 2. The standard 64KB block size is ~10x slower for random seeks than a 4KB block size, but block sizes as small as 4KB cause problems elsewhere. Using block sizes of 256KB or 1MB or more may be more efficient from a disk access and block-cache perspective in many big-data applications, but doing so is infeasible from a random seek perspective. The PrefixTrie block encoding format attempts to solve both of these problems. Some features: * trie format for row key encoding completely eliminates duplicate row keys and encodes similar row keys into a standard trie structure which also saves a lot of space * the column family is currently stored once at the beginning of each block. this could easily be modified to allow multiple family names per block * all qualifiers in the block are stored in their own trie format which caters nicely to wide rows. duplicate qualifers between rows are eliminated. the size of this trie determines the width of the block's qualifier fixed-width-int * the minimum timestamp is stored at the beginning of the block, and deltas are calculated from that. the maximum delta determines the width of the block's timestamp fixed-width-int The block is structured with metadata at the beginning, then a section for the row trie, then the column trie, then the timestamp deltas, and then then all the values. Most work is done in the row trie, where every leaf node (corresponding to a row) contains a list of offsets/references corresponding to the cells in that row. Each cell is fixed-width to enable binary searching and is represented by [1 byte operationType, X bytes qualifier offset, X bytes timestamp delta offset]. If all operation types are the same for a block, there will be zero per-cell overhead. Same for timestamps. Same for qualifiers when i get a chance. So, the compression aspect is very strong, but makes a few small sacrifices on VarInt size to enable faster binary searches in trie fan-out nodes. A more compressed but slower version might build on this by also applying further (suffix, etc) compression on the trie nodes at the cost of slower write speed. Even further compression could be obtained by using all VInts instead of FInts with a sacrifice on random seek speed (though not huge). One current drawback is the current write speed. While programmed with good constructs like TreeMaps, ByteBuffers, binary searches, etc, it's not programmed with the same level of optimization as the read path. Work will need to be done to optimize the data structures used for encoding and could probably show a 10x increase. It will still be slower than delta encoding, but with a much higher decode speed. I have not yet created a thorough
[jira] [Commented] (HBASE-7789) Clean DeadServer.java and add a Jitter method in ConnectionUtils
[ https://issues.apache.org/jira/browse/HBASE-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573737#comment-13573737 ] stack commented on HBASE-7789: -- Sounds good except for 'use directly the object instead of a copy.' Whats wrong w/ making new instance if something changes? Don't we only make these once? Clean DeadServer.java and add a Jitter method in ConnectionUtils Key: HBASE-7789 URL: https://issues.apache.org/jira/browse/HBASE-7789 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 I need to do some changes in DeadServer because of HBASE-7390. To minimize the patch size and simplifies the feedback, I prefer to isolate the issue. Changes are: - Add the time when the server was declared as dead. It's what I need in HBASE-7390, but it makes sense even without it, for example to be shown in the UI. - suppress the extends on Set clean up all the not used methods - use directly the object instead of a copy. For connection utils, we currently have a jitter of 1%. I need a bigger one for sure in one case, but I wonder if we should not increase it in all cases? instead of plus 1%, we should have plus or minus 10% imho. Tests are in progress locally, I will add the patch when they're ok. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7787) Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR
[ https://issues.apache.org/jira/browse/HBASE-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573739#comment-13573739 ] stack commented on HBASE-7787: -- +1 Remove HLogSplitter.RECOVERED_EDITS duplicate of HLog.RECOVERED_EDITS_DIR - Key: HBASE-7787 URL: https://issues.apache.org/jira/browse/HBASE-7787 Project: HBase Issue Type: Bug Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-7787-v0.patch HLogSplitter.RECOVERED_EDITS seems to be used only in TestHLogSplit, and has the exact same meaning of HLog.RECOVERED_EDITS_DIR -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573741#comment-13573741
]
Hadoop QA commented on HBASE-7786:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568432/HBASE-7786-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4371//console
This message is automatically generated.
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573744#comment-13573744 ] stack commented on HBASE-7233: -- [~mcorgan] Yeah, I think that the way to go. Client can send over that it favors v2 but that it can do v1 too... Server will do to the best of its abilities. This won't help [~andrew.purt...@gmail.com] if he wants to refer to tags via the Cell Interface? He'll have to cast to KV2 which will 'break'? So, wondering if we should add to Cell Interface now support for tags? Only other feature we have ever talked of is adding a mvcc into the key but Interface already has that so a shift in how it is implemented should have no effect on the Cell Interface. Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7764) [snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
[
https://issues.apache.org/jira/browse/HBASE-7764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573745#comment-13573745
]
Jonathan Hsieh commented on HBASE-7764:
---
Can you give a quick summary of why this policy vs the new default cause this
to fail/succeed? I can read the code but future readers it would be really
helpful.
[snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
-
Key: HBASE-7764
URL: https://issues.apache.org/jira/browse/HBASE-7764
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 7764.txt, 7764-v2.txt
Here is log snippet for
TestSnapshotCloneIndependence#testOnlineSnapshotRegionOperationsIndependent
(pay attention to region 1360020297284.2e43e47a882d3cff601eb222cad41f20.):
{code}
2013-02-04 15:24:58,369 INFO
[MASTER_SERVER_OPERATIONS-10.11.2.194,61955,1360020289835-0]
handler.SplitRegionHandler(115): Handled SPLIT event;
parent=test1360020295791,,1360020295793.794d37c0445b61619b5056623228827d.
daughter
a=test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.daughter
b=test1360020295791,dgb,1360020297284.b87834cee60702d883aa287df6aaeaef.
...
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(78): Starting region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(81): Flush Snapshotting
region test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
started...
2013-02-04 15:25:02,005 DEBUG [member:
apos;10.11.2.194,61958,1360020290064apos; subprocedure-pool3-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(318): Completed
2/3 local region snapshots.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2485): Storing region-info for snapshot.
2013-02-04 15:25:02,006 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
util.FSUtils(166): Creating
file=hdfs://localhost:61942/user/tyu/hbase/.snapshot/.tmp/snapshot_test1360020295791/2e43e47a882d3cff601eb222cad41f20/.tmp/.regioninfo
with permission=rwxrwxrwx
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2489): Creating references for hfiles
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2502): Adding snapshot references for
[hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02]
hfiles
2013-02-04 15:25:02,015 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2516): Creating reference for file (1/1) :
hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(84): ... Flush
Snapshotting region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. completed.
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(86): Closing region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
...
2013-02-04 15:25:03,836 INFO [PRI IPC Server handler 7 on 61958]
regionserver.HRegionServer(3600): Splitting
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.HStore(1686): cannot split because midkey is the same as first
or last row
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.CompactSplitThread(172): Region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. not
splittable because midkey=null
2013-02-04 15:25:03,837 DEBUG [main] catalog.CatalogTracker(231): Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5c602d9d
2013-02-04 15:25:03,838 INFO [main]
client.TestSnapshotCloneIndependence(307): split requested for
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:03,892 DEBUG [main] client.MetaScanner(199): Scanning .META.
starting at row=test1360020295791,,00 for max=2147483647 rows
using hconnection 0x53d9f80
2013-02-04 15:25:03,895 DEBUG [main]
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573751#comment-13573751
]
Matteo Bertozzi commented on HBASE-7786:
{quote}Deprecate if you are not going to remove as Sergey suggests?{quote}
It will be nice having some decorator to mark methods as Exposed for testing
do we have something similar?
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573755#comment-13573755 ] Himanshu Vashishtha commented on HBASE-7723: There are two use cases here: a) Supporting a rolling restart b) Change in NN URI. They are mutually exclusive. This jira deals only with b). This requires a restart of the entire cluster, which is OK. I grepped the above tools, and they are using conf object, which is fine too. The only problem is the zondes which contains absolute path now become irrelevant because of the URI change. One needs to manually remove those stale znodes. This patch intends to use relative path to avoid such behavior. bq. where we want to get to hdfs where the config may have the single nn format but we're running the ha nn – or vice versa. We need a restart in this case, which should serve the purpose. All the nodes will have right conf property then, right? For a), we don't need to change anything for this imo. Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.5 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573756#comment-13573756 ] Andrew Purtell commented on HBASE-7233: --- If versioning the codecs like BasicCellEncoderV1, then could we have accessors for tags in the Cell interface but BasicCellEncoderV1 would throw UnsupportedOperationException, while a BasicCellEncoderV2 would support it? And/or a method in the interface that a user can interrogate for capabilities, i.e. can do tags or not? Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573757#comment-13573757 ] stack commented on HBASE-7233: -- [~andrew.purt...@gmail.com] I was thinking would just return no tags in v1 rather than unsupported? Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573760#comment-13573760 ] Andrew Purtell commented on HBASE-7233: --- bq. I was thinking would just return no tags in v1 rather than unsupported? Sure, easy. Let's at least get the placeholders in. Both HBASE-7662 and HBASE-7663 would need efficiently stored and accessed tags in KVs, and [~enis] was also talking about using KV tags for holding metadata for grouping rows recently if I recall correctly. It would be even better if there was a BasicCellEncoderV2 that could actually store and retrieve tags, at least in unit tests, even if not baked enough to actually use until a later release. Something to build on. Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573760#comment-13573760 ] Andrew Purtell edited comment on HBASE-7233 at 2/7/13 6:37 PM: --- bq. I was thinking would just return no tags in v1 rather than unsupported? Sure, easy. Let's at least get the placeholders in. Both HBASE-7662 and HBASE-7663 would need efficiently stored and accessed tags in KVs, and [~enis] was also talking about using KV tags for holding metadata for grouping rows recently if I recall correctly. It would be even better if there was a BasicCellEncoderV2 that could actually store and retrieve tags, at least in unit tests, even if not baked enough to actually use until a later release. Enough to prove the tag accessor methods workable. Something to build on. was (Author: apurtell): bq. I was thinking would just return no tags in v1 rather than unsupported? Sure, easy. Let's at least get the placeholders in. Both HBASE-7662 and HBASE-7663 would need efficiently stored and accessed tags in KVs, and [~enis] was also talking about using KV tags for holding metadata for grouping rows recently if I recall correctly. It would be even better if there was a BasicCellEncoderV2 that could actually store and retrieve tags, at least in unit tests, even if not baked enough to actually use until a later release. Something to build on. Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7415) [snapshots] Add task information to snapshot operation
[ https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573765#comment-13573765 ] Jesse Yates commented on HBASE-7415: I've been caught up in other work and haven't had a chance to do the refactor for online snapshots. Thinking about holding off until [~jmhsieh], [~mbertozzi], and co. get the tests passing consistently (I'd rather not make things more difficult for you fellas). Sound reasonable? [snapshots] Add task information to snapshot operation -- Key: HBASE-7415 URL: https://issues.apache.org/jira/browse/HBASE-7415 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Fix For: hbase-6055, 0.96.0 Attachments: hbase-7415-v0.patch, hbase-7415-v1.patch, HBase_Snapshot_Task_UI.png Snapshot operations should have some sort of progresss information available via the WebUI so admins can track progress of operations. This should go a long way to enable 'good' admins to not hose their clusters by running concurrent snapshot operations (e.g. rename while a clone is in progress). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7764) [snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
[
https://issues.apache.org/jira/browse/HBASE-7764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573766#comment-13573766
]
Ted Yu commented on HBASE-7764:
---
After reverting HBASE-7412, the test passed.
Here is hbase.hregion.max.filesize entry from
hbase-server/src/test/resources/hbase-site.xml:
{code}
property
namehbase.hregion.max.filesize/name
value67108864/value
{code}
Here is the default value:
{code}
namehbase.hregion.max.filesize/name
value10737418240/value
{code}
Jimmy's change made the custom value (67MB) effective.
[snapshot 130201 merge] Fix TestSnapshotCloneIndependence failure
-
Key: HBASE-7764
URL: https://issues.apache.org/jira/browse/HBASE-7764
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 7764.txt, 7764-v2.txt
Here is log snippet for
TestSnapshotCloneIndependence#testOnlineSnapshotRegionOperationsIndependent
(pay attention to region 1360020297284.2e43e47a882d3cff601eb222cad41f20.):
{code}
2013-02-04 15:24:58,369 INFO
[MASTER_SERVER_OPERATIONS-10.11.2.194,61955,1360020289835-0]
handler.SplitRegionHandler(115): Handled SPLIT event;
parent=test1360020295791,,1360020295793.794d37c0445b61619b5056623228827d.
daughter
a=test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.daughter
b=test1360020295791,dgb,1360020297284.b87834cee60702d883aa287df6aaeaef.
...
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(78): Starting region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(81): Flush Snapshotting
region test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
started...
2013-02-04 15:25:02,005 DEBUG [member:
apos;10.11.2.194,61958,1360020290064apos; subprocedure-pool3-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(318): Completed
2/3 local region snapshots.
2013-02-04 15:25:02,005 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2485): Storing region-info for snapshot.
2013-02-04 15:25:02,006 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
util.FSUtils(166): Creating
file=hdfs://localhost:61942/user/tyu/hbase/.snapshot/.tmp/snapshot_test1360020295791/2e43e47a882d3cff601eb222cad41f20/.tmp/.regioninfo
with permission=rwxrwxrwx
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2489): Creating references for hfiles
2013-02-04 15:25:02,014 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2502): Adding snapshot references for
[hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02]
hfiles
2013-02-04 15:25:02,015 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
regionserver.HRegion(2516): Creating reference for file (1/1) :
hdfs://localhost:61942/user/tyu/hbase/test1360020295791/2e43e47a882d3cff601eb222cad41f20/fam/946b75e5f0ba445aa1c646b8cbc87e02
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(84): ... Flush
Snapshotting region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. completed.
2013-02-04 15:25:02,017 DEBUG
[rs(10.11.2.194,61958,1360020290064)-snapshot-pool5-thread-1]
snapshot.FlushSnapshotSubprocedure$RegionSnapshotTask(86): Closing region
operation on
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
...
2013-02-04 15:25:03,836 INFO [PRI IPC Server handler 7 on 61958]
regionserver.HRegionServer(3600): Splitting
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.HStore(1686): cannot split because midkey is the same as first
or last row
2013-02-04 15:25:03,836 DEBUG [PRI IPC Server handler 7 on 61958]
regionserver.CompactSplitThread(172): Region
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20. not
splittable because midkey=null
2013-02-04 15:25:03,837 DEBUG [main] catalog.CatalogTracker(231): Stopping
catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@5c602d9d
2013-02-04 15:25:03,838 INFO [main]
client.TestSnapshotCloneIndependence(307): split requested for
test1360020295791,,1360020297284.2e43e47a882d3cff601eb222cad41f20.
2013-02-04
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573767#comment-13573767 ] Sergey Shelukhin commented on HBASE-7725: - left some non-blocking comments in /r/. I am assuming Stack meant [~apurtell] in the last comment? :) Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573771#comment-13573771
]
Sergey Shelukhin commented on HBASE-7495:
-
latest patch looks reasonable but what about the tests?
TestCoprocessorScanPolicy has a ctor that enables new feature, but I don't see
it called.
parallel seek in StoreScanner
-
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: Liang Xie
Assignee: Liang Xie
Attachments: HBASE-7495-0.94.txt, HBASE-7495.txt, HBASE-7495.txt,
HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt, HBASE-7495-v2.txt,
HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt, HBASE-7495-v5.txt,
HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt, HBASE-7495-v9.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7786) Consolidate HRegion creation/opening API
[
https://issues.apache.org/jira/browse/HBASE-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573774#comment-13573774
]
Sergey Shelukhin commented on HBASE-7786:
-
{quote}
bq. createDaughterRegion(final HRegionInfo hri, final Path daughterTmpDir)
This is used only by the split code, and inside HRegion just to make
newHRegion() private
{quote}
Oh, I see, they are in different files.
+1 then, +1 on deprecating the test call.
Consolidate HRegion creation/opening API
Key: HBASE-7786
URL: https://issues.apache.org/jira/browse/HBASE-7786
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-7786-v0.patch, HBASE-7786-v1.patch
Currently we have 4 ways to instantiate an HRegion.
* HRegion.createHRegion()
* HRegion.openHRegion()
* HRegion.newHRegion() + r.initialize()
* new HRegion() + r.initialize()
Aside from tests and HMerge and SplitTransaction code everyone use
createHRegion() and openHRegion(). To avoid errors due to missing
initialization calls I think we should limit the access to newHRegion() and
the constructor.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573778#comment-13573778
]
Ted Yu commented on HBASE-7495:
---
Review request is at https://reviews.apache.org/r/9272/
The new TestCoprocessorScanPolicy ctor would be called by JUnit since
TestCoprocessorScanPolicy is parameterized.
parallel seek in StoreScanner
-
Key: HBASE-7495
URL: https://issues.apache.org/jira/browse/HBASE-7495
Project: HBase
Issue Type: Bug
Components: Scanners
Affects Versions: 0.94.3, 0.96.0
Reporter: Liang Xie
Assignee: Liang Xie
Attachments: HBASE-7495-0.94.txt, HBASE-7495.txt, HBASE-7495.txt,
HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt, HBASE-7495-v2.txt,
HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt, HBASE-7495-v5.txt,
HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt, HBASE-7495-v9.txt
seems there's a potential improvable space before doing scanner.next:
{code:title=StoreScanner.java|borderStyle=solid}
if (explicitColumnQuery lazySeekEnabledGlobally) {
for (KeyValueScanner scanner : scanners) {
scanner.requestSeek(matcher.getStartKey(), false, true);
}
} else {
for (KeyValueScanner scanner : scanners) {
scanner.seek(matcher.getStartKey());
}
}
{code}
we can do scanner.requestSeek or scanner.seek in parallel, instead of current
serialization, to reduce latency for special case.
Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7785) rolling-restart.sh script unable to check expiration of master znode
[ https://issues.apache.org/jira/browse/HBASE-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-7785. -- Resolution: Fixed Committed to 0.94 and 0.96. Thanks for the patch. rolling-restart.sh script unable to check expiration of master znode Key: HBASE-7785 URL: https://issues.apache.org/jira/browse/HBASE-7785 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.4 Environment: Linux CentOS release 5.8 Reporter: Samir Ahmic Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7785.txt When rolling-restart.sh script stop master it enters loop trying to detect that master znode is deleted. Since it is unable to execute check command script hangs in infinite loop. Problematic line of script is: while ! bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do bin/hbase zkcli stat can not be executed since script is run from bin directory. My suggestion is that this line should be like this in order to work: while ! $bin/hbase zkcli stat $zmaster 21 | grep Node does not exist; do After i made this change i was able to execute rolling restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[
https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573784#comment-13573784
]
Matt Corgan commented on HBASE-7233:
{quote}So, wondering if we should add to Cell interface now support for
tags?{quote}Guess I'm confused why we need to add it to the Cell interface now
since the reason for the versioning is to enable us to add it later. It would
help future-proof the KeyValueEncoderV1? I think Andy has found a way to work
the tags into the current KeyValue serialization, so might not even need a V2
for that.
Serializing KeyValues
-
Key: HBASE-7233
URL: https://issues.apache.org/jira/browse/HBASE-7233
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.96.0
Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt,
7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt,
7233v6_encoder.txt, 7233v7.txt, 7233v9.txt
Undo KeyValue being a Writable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573785#comment-13573785 ] Lars Hofhansl commented on HBASE-7723: -- There's also c) support changing of HBase root (also requires an HBase restart) b) would be handled by my -minimal patch as well. for c) your patch is needed. Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.5 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-3171: -- Status: Patch Available (was: Open) Drop ROOT and instead store META location(s) directly in ZooKeeper -- Key: HBASE-3171 URL: https://issues.apache.org/jira/browse/HBASE-3171 Project: HBase Issue Type: Improvement Components: Client, master, regionserver, Zookeeper Reporter: Jonathan Gray Attachments: HBASE-3171.patch, HBASE-3171-v2.patch, HBASE-3171-v3.patch Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and reading the META location, we should just store the META location directly in ZooKeeper. The purpose of the root region from the bigtable paper was to support multiple meta regions. Currently, we explicitly only support a single meta region, so the translation from our current code of a single root location to a single meta location will be very simple. Long-term, it seems reasonable that we could store several meta region locations in ZK. There's been some discussion in HBASE-1755 about actually moving META into ZK, but I think this jira is a good step towards taking some of the complexity out of how we have to deal with catalog tables everywhere. As-is, a new client already requires ZK to get the root location, so this would not change those requirements in any way. The primary motivation for this is to simplify things like CatalogTracker. The way we can handle root in that class is really simple but the tracking of meta is difficulty and a bit hacky. This hack on tracking of the meta location is what caused one of the bugs over in HBASE-3159. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7415) [snapshots] Add task information to snapshot operation
[ https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573788#comment-13573788 ] Jonathan Hsieh commented on HBASE-7415: --- [~jesse_yates] sounds reasonable. when we get tests passing, depending on if the merge happens we'll promote this (and any other subtasks) to a fully fledged issues as opposed to a subtask. [snapshots] Add task information to snapshot operation -- Key: HBASE-7415 URL: https://issues.apache.org/jira/browse/HBASE-7415 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Fix For: hbase-6055, 0.96.0 Attachments: hbase-7415-v0.patch, hbase-7415-v1.patch, HBase_Snapshot_Task_UI.png Snapshot operations should have some sort of progresss information available via the WebUI so admins can track progress of operations. This should go a long way to enable 'good' admins to not hose their clusters by running concurrent snapshot operations (e.g. rename while a clone is in progress). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573792#comment-13573792 ] Andrew Purtell commented on HBASE-7725: --- I think downcasting is a code smell and should be avoided unless necessary. This is not necessarily better: What about passing the CompactionRequest through the CP hooks and making it so the CompactionRequest type can be extended? Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573793#comment-13573793 ] Jean-Daniel Cryans commented on HBASE-3171: --- Found that TestAdmin has a new testRootTableSplit test and it fails. Leading me to do a lot more cleanup around that part of the code. Drop ROOT and instead store META location(s) directly in ZooKeeper -- Key: HBASE-3171 URL: https://issues.apache.org/jira/browse/HBASE-3171 Project: HBase Issue Type: Improvement Components: Client, master, regionserver, Zookeeper Reporter: Jonathan Gray Attachments: HBASE-3171.patch, HBASE-3171-v2.patch, HBASE-3171-v3.patch Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and reading the META location, we should just store the META location directly in ZooKeeper. The purpose of the root region from the bigtable paper was to support multiple meta regions. Currently, we explicitly only support a single meta region, so the translation from our current code of a single root location to a single meta location will be very simple. Long-term, it seems reasonable that we could store several meta region locations in ZK. There's been some discussion in HBASE-1755 about actually moving META into ZK, but I think this jira is a good step towards taking some of the complexity out of how we have to deal with catalog tables everywhere. As-is, a new client already requires ZK to get the root location, so this would not change those requirements in any way. The primary motivation for this is to simplify things like CatalogTracker. The way we can handle root in that class is really simple but the tracking of meta is difficulty and a bit hacky. This hack on tracking of the meta location is what caused one of the bugs over in HBASE-3159. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573795#comment-13573795 ] Jesse Yates commented on HBASE-7725: Thanks [~sershe]! Since I've got +1s from stack and Ted (and it looks like [~sershe] blessing too?), I'm planning on committing v1 to trunk and 0.94 tonight. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573795#comment-13573795 ] Jesse Yates edited comment on HBASE-7725 at 2/7/13 7:09 PM: Thanks [~sershe]! Since I've got +1s from stack and Ted (and it looks like [~sershe] blessing too?), I'm planning on committing v1 to trunk and 0.94 tonight, unless there are any objections. was (Author: jesse_yates): Thanks [~sershe]! Since I've got +1s from stack and Ted (and it looks like [~sershe] blessing too?), I'm planning on committing v1 to trunk and 0.94 tonight. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573796#comment-13573796 ] Andrew Purtell commented on HBASE-7725: --- Fine, -1. Consider the above. Then let's talk. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7741) Don't use bulk assigner if assigning just several regions
[ https://issues.apache.org/jira/browse/HBASE-7741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573817#comment-13573817 ] Hudson commented on HBASE-7741: --- Integrated in HBase-TRUNK #3861 (See [https://builds.apache.org/job/HBase-TRUNK/3861/]) HBASE-7741 Don't use bulk assigner if assigning just several regions (Revision 1443617) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java Don't use bulk assigner if assigning just several regions - Key: HBASE-7741 URL: https://issues.apache.org/jira/browse/HBASE-7741 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: trunk-7741.patch, trunk-7741_v2.patch, trunk-7741_v3_1.patch, trunk-7741_v3.patch If just assign one region, bulk assigner may be slower. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6782) HBase shell's 'status 'detailed'' should escape the printed keys
[ https://issues.apache.org/jira/browse/HBASE-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573818#comment-13573818 ] Hudson commented on HBASE-6782: --- Integrated in HBase-TRUNK #3861 (See [https://builds.apache.org/job/HBase-TRUNK/3861/]) HBASE-6782 HBase shell's 'status 'detailed'' should escape the printed keys (Viji) (Revision 1443620) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/ruby/hbase/admin.rb HBase shell's 'status 'detailed'' should escape the printed keys Key: HBASE-6782 URL: https://issues.apache.org/jira/browse/HBASE-6782 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.90.1 Reporter: Viji Assignee: Viji Priority: Minor Fix For: 0.96.0 Attachments: HBASE-6782.patch Currently the HBase shell's status command prints unescaped keys on the terminal causing the terminal to print garbage characters. We should escape the printed keys. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7698) race between RS shutdown thread and openregionhandler causes region to get stuck
[ https://issues.apache.org/jira/browse/HBASE-7698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573819#comment-13573819 ] Hudson commented on HBASE-7698: --- Integrated in HBase-TRUNK #3861 (See [https://builds.apache.org/job/HBase-TRUNK/3861/]) HBASE-7698 race between RS shutdown thread and openregionhandler causes region to get stuck (Ram) (Revision 1443637) Result = FAILURE ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/handler/TestOpenRegionHandler.java race between RS shutdown thread and openregionhandler causes region to get stuck Key: HBASE-7698 URL: https://issues.apache.org/jira/browse/HBASE-7698 Project: HBase Issue Type: Bug Affects Versions: 0.94.4 Reporter: Sergey Shelukhin Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0, 0.94.5 Attachments: HBASE-7698_0.94.patch, HBASE-7698.patch, HBASE-7698_trunk_final.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase_1.patch, HBASE-7698_withtestcase.patch 2013-01-22 17:59:03,237 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5984cf08] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5984cf08 ... 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1001): Closing IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb.: disabling compactions amp; flushes 2013-01-22 17:59:03,411 DEBUG [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] regionserver.HRegion(1023): Updates disabled for region IntegrationTestRebalanceAndKillServersTargeted,6660,1358906196709.0200b366bc37c5afd1185f7d487c7dfb. 2013-01-22 17:59:03,415 ERROR [RS_OPEN_REGION-10.11.2.92,50661,1358906192942-0] executor.EventHandler(205): Caught throwable while processing event M_RS_OPEN_REGION java.io.IOException: java.io.IOException: java.io.IOException: Filesystem closed at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1058) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:974) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:945) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.cleanupFailedOpen(OpenRegionHandler.java:459) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:143) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) tryTransitionFromOpeningToFailedOpen or transitionToOpened below is never called and region can get stuck. As an added benefit, the meta is already written by that time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573823#comment-13573823 ] Andrew Purtell commented on HBASE-7233: --- Yeah I guess there is a little bit of confusion here. I was thinking Cell can support getting tag data, and the encoders might not support it (v1) or could (v2). Should tags be a concern of Cell or KV? As maybe an interesting consideration, I only need tags in on disk representation and actually should not send the ones I'd be working with over the wire to the client. bq. I think Andy has found a way to work the tags into the current KeyValue serialization I did, but it is ugly IMHO: I store the value length as negative, prepend delimited tag data to the user value data, and parse the tags into in-memory metadata and fix up offsets on KV instantiation. Do we actually want this? If so, then I guess we can have tagged KVs mixed with old KVs in a backwards compatible way. Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3171) Drop ROOT and instead store META location(s) directly in ZooKeeper
[
https://issues.apache.org/jira/browse/HBASE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573825#comment-13573825
]
Hadoop QA commented on HBASE-3171:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568322/HBASE-3171-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 24 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegionInfo
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4372//console
This message is automatically generated.
Drop ROOT and instead store META location(s) directly in ZooKeeper
--
Key: HBASE-3171
URL: https://issues.apache.org/jira/browse/HBASE-3171
Project: HBase
Issue Type: Improvement
Components: Client, master, regionserver, Zookeeper
Reporter: Jonathan Gray
Attachments: HBASE-3171.patch, HBASE-3171-v2.patch,
HBASE-3171-v3.patch
Rather than storing the ROOT region location in ZooKeeper, going to ROOT, and
reading the META location, we should just store the META location directly in
ZooKeeper.
The purpose of the root region from the bigtable paper was to support
multiple meta regions. Currently, we explicitly only support a single meta
region, so the translation from our current code of a single root location to
a single meta location will be very simple. Long-term, it seems reasonable
that we could store several meta region locations in ZK. There's been some
discussion in HBASE-1755 about actually moving META into ZK, but I think this
jira is a good step towards taking some of the complexity out of how we have
to deal with catalog tables everywhere.
As-is, a new client already requires ZK to get the root location, so this
would not change those requirements in any way.
The primary motivation for this is to simplify things like CatalogTracker.
The way we can handle root in that class is really simple but the tracking of
meta is difficulty and a bit hacky. This hack on tracking of the meta
location is what caused one of the bugs over in HBASE-3159.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573823#comment-13573823 ] Andrew Purtell edited comment on HBASE-7233 at 2/7/13 7:28 PM: --- Yeah I guess there is a little bit of confusion here. I was thinking Cell can support getting tag data, and the encoders might not support it (v1) or could (v2). Should tags be a concern of Cell or KV? As maybe an interesting consideration, I only need tags in on disk representation and actually should not send the ones I'd be working with over the wire to the client. bq. I think Andy has found a way to work the tags into the current KeyValue serialization I did, but it is ugly IMHO: I store the value length as negative to flag the KV as one containing tag data, prepend delimited tag data to the user value data, and parse the tags into in-memory metadata and fix up offsets on KV instantiation. Do we actually want this? If so, then I guess we can have tagged KVs mixed with old KVs in a backwards compatible way. was (Author: apurtell): Yeah I guess there is a little bit of confusion here. I was thinking Cell can support getting tag data, and the encoders might not support it (v1) or could (v2). Should tags be a concern of Cell or KV? As maybe an interesting consideration, I only need tags in on disk representation and actually should not send the ones I'd be working with over the wire to the client. bq. I think Andy has found a way to work the tags into the current KeyValue serialization I did, but it is ugly IMHO: I store the value length as negative, prepend delimited tag data to the user value data, and parse the tags into in-memory metadata and fix up offsets on KV instantiation. Do we actually want this? If so, then I guess we can have tagged KVs mixed with old KVs in a backwards compatible way. Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573833#comment-13573833
]
Jesse Yates commented on HBASE-7725:
I'd need to control a compaction request from inception to execution. Looking
at the way we make compactions, there are the pre/postCompactSelection which
would need to get the attributes. Then we could definitely add a hook at the
end where we can override the compactionRequest:
{code}
...
// everything went better than expected. create a compaction request
int pri = getCompactPriority(priority);
ret = new CompactionRequest(region, this, filesToCompact, isMajor, pri,
compactionAttributes);
...
{code}
to something like:
{code}
...
// everything went better than expected. create a compaction request
int pri = getCompactPriority(priority);
if (region.getCoprocessorHost() != null) {
ret = region.getCoprocessorHost().preCompactionRequest(region, this,
filesToCompact, isMajor, pri, compactionAttributes)
}
if(ret == null){
ret = new CompactionRequest(region, this, filesToCompact, isMajor, pri,
compactionAttributes);
}
{code}
Then we'd just have to pass in the compaction request to all the compaction CP
hooks, rather than the compactionAttributes object so we can handle special
case compaction setup (for methods like preCompactScannerOpen and preCompact).
So the same overall usage and layout, but we don't need to modify
CompactionRequest and let people add their own finish hooks and store
attributes if the want.
I dig it Andrew. Sound reasonable to everyone else?
Add generic attributes to CP initiated compaction request AND latch on
compaction completion
Key: HBASE-7725
URL: https://issues.apache.org/jira/browse/HBASE-7725
Project: HBase
Issue Type: Bug
Components: Compaction, Coprocessors, regionserver
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.6
Attachments: example.java, hbase-7725_0.94-v0.patch,
hbase-7725-v0.patch, hbase-7725-v1.patch,
hbase-7725_with-attributes-0.94-v0.patch,
hbase-7725_with-attributes-0.94-v1.patch
You can request that a compaction be started, but you can't be sure when that
compaction request completes. This is a simple update to the
CompactionRequest interface and the compact-split thread on the RS that
doesn't actually impact the RS exposed interface.
This is particularly useful for CPs so they can control starting/running a
compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573837#comment-13573837 ] Andrew Purtell commented on HBASE-7725: --- I do like it better, thanks for listening, +0. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573840#comment-13573840 ] Sergey Shelukhin commented on HBASE-7725: - hmm, allowing the user to create compaction requests seems more dangerous to me than having an opaque context/attributes. The latter is a common pattern when calling plugins that cannot preserve information across calls. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573841#comment-13573841 ] Himanshu Vashishtha commented on HBASE-7723: Yes, I considered it to be a special case of b) :) But yes, you're right. Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.5 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573844#comment-13573844 ] Sergey Shelukhin commented on HBASE-7725: - Another alternative, uglier but with more general applicability, is to create some ID for the operation and pass it to coprocessor methods if they belong to the same logical operation. If coprocessor has its own context, it can manage it, for example in a Map. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7723) Remove NN URI from ZK splitlogs.
[ https://issues.apache.org/jira/browse/HBASE-7723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7723: - Fix Version/s: (was: 0.94.5) 0.94.6 Remove NN URI from ZK splitlogs. Key: HBASE-7723 URL: https://issues.apache.org/jira/browse/HBASE-7723 Project: HBase Issue Type: Bug Components: hadoop2, master Affects Versions: 0.92.0 Reporter: Kevin Odell Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.6 Attachments: 7723-0.94-minimal.txt, 7723-0.94-simple.txt, HBASE-7723-94.patch, HBASE-7723-94-v2.patch, HBASE-7723-94-v3.patch, HBASE-7723-94-v4.patch, HBASE-7723-94-v5.patch, HBASE-7723-trunk-v1.patch, HBASE-7723-trunk-v2.patch When moving to HDFS HA or removing HA we end up changing the NN namespace. This can cause the HMaster not to start up fully due to trying to split phantom HLogs pointing to the wrong FS - java.lang.IllegalArgumentException: Wrong FS: error messages. The HLogs in question might not even be on HDFS anymore. You have to go in a manually clear out the ZK splitlogs directory to get HBase to properly boot up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573860#comment-13573860
]
Jesse Yates commented on HBASE-7725:
{quote}
Another alternative, uglier but with more general applicability, is to create
some ID for the operation and pass it to coprocessor methods if they belong to
the same logical operation. If coprocessor has its own context, it can manage
it, for example in a Map.
{quote}
This would be far better managed as a set of attributes attached to a
CompactionRequest. Generally, you aren't going to start a compaction from one
of the generic RegionObserver methods, so its going to be an Endpoint, which
mean another class, so you then need to keep a static Map shared across the
classes.
{quote}
allowing the user to create compaction requests seems more dangerous to me than
having an opaque context/attributes.
{quote}
Sure, but CPs are inherently more dangerous. They are there for you people able
to futz around with the internals and come with no warranty ;) In our use case,
we will probably just subclass the exsiting CompactionRequest and add the
things that I added in the attached patches (v0, v1). I think adding this
subclass would be a reasonable inclusion, at the least in the hbase-examples
module, if not in the standard directories.
Add generic attributes to CP initiated compaction request AND latch on
compaction completion
Key: HBASE-7725
URL: https://issues.apache.org/jira/browse/HBASE-7725
Project: HBase
Issue Type: Bug
Components: Compaction, Coprocessors, regionserver
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.6
Attachments: example.java, hbase-7725_0.94-v0.patch,
hbase-7725-v0.patch, hbase-7725-v1.patch,
hbase-7725_with-attributes-0.94-v0.patch,
hbase-7725_with-attributes-0.94-v1.patch
You can request that a compaction be started, but you can't be sure when that
compaction request completes. This is a simple update to the
CompactionRequest interface and the compact-split thread on the RS that
doesn't actually impact the RS exposed interface.
This is particularly useful for CPs so they can control starting/running a
compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573861#comment-13573861 ] Andrew Purtell commented on HBASE-7725: --- bq. allowing the user to create compaction requests seems more dangerous to me than having an opaque context/attributes I see it as being more powerful. This is Coprocessors, dangerous is implied. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573861#comment-13573861 ] Andrew Purtell edited comment on HBASE-7725 at 2/7/13 7:58 PM: --- bq. allowing the user to create compaction requests seems more dangerous to me than having an opaque context/attributes I see it as being more powerful, and a bit better structured. This is Coprocessors, dangerous is implied. was (Author: apurtell): bq. allowing the user to create compaction requests seems more dangerous to me than having an opaque context/attributes I see it as being more powerful. This is Coprocessors, dangerous is implied. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573866#comment-13573866 ] Sergey Shelukhin commented on HBASE-7725: - Hmm, sure. +1. CompactionRequest then also needs InterfaceAudience, and ctor cannot be removed later. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573870#comment-13573870 ] Jesse Yates commented on HBASE-7725: Not necessarily - we already have a couple cases where we override an InterfaceAudience.Private. See RegionCoprocessorHost# postScannerOpen(final Scan scan, RegionScanner s), which returns a RegionScanner. Add generic attributes to CP initiated compaction request AND latch on compaction completion Key: HBASE-7725 URL: https://issues.apache.org/jira/browse/HBASE-7725 Project: HBase Issue Type: Bug Components: Compaction, Coprocessors, regionserver Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.6 Attachments: example.java, hbase-7725_0.94-v0.patch, hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725_with-attributes-0.94-v0.patch, hbase-7725_with-attributes-0.94-v1.patch You can request that a compaction be started, but you can't be sure when that compaction request completes. This is a simple update to the CompactionRequest interface and the compact-split thread on the RS that doesn't actually impact the RS exposed interface. This is particularly useful for CPs so they can control starting/running a compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573871#comment-13573871
]
Andrew Purtell commented on HBASE-7725:
---
We don't have consensus on if interface annotations are applicable to
coprocessors. I'd be cool with LimitedPrivate({coprocessors}) and Evolving.
Add generic attributes to CP initiated compaction request AND latch on
compaction completion
Key: HBASE-7725
URL: https://issues.apache.org/jira/browse/HBASE-7725
Project: HBase
Issue Type: Bug
Components: Compaction, Coprocessors, regionserver
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.6
Attachments: example.java, hbase-7725_0.94-v0.patch,
hbase-7725-v0.patch, hbase-7725-v1.patch,
hbase-7725_with-attributes-0.94-v0.patch,
hbase-7725_with-attributes-0.94-v1.patch
You can request that a compaction be started, but you can't be sure when that
compaction request completes. This is a simple update to the
CompactionRequest interface and the compact-split thread on the RS that
doesn't actually impact the RS exposed interface.
This is particularly useful for CPs so they can control starting/running a
compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573883#comment-13573883 ] stack commented on HBASE-7233: -- Tags should be concern of Cell else you will have to cast to KV2 before getting them. I'd think we'd add tag support to Cell so we don't have to change it later (that'd be painful, wouldn't it?) Serializing KeyValues - Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7233sketch.txt, 7233.txt, 7233v10.txt, 7233-v2.txt, 7233v3_encoders.txt, 7233v4_encoders.txt, 7233v5_encoders.txt, 7233v6_encoder.txt, 7233v7.txt, 7233v9.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7763) Compactions not sorting based on size anymore.
[ https://issues.apache.org/jira/browse/HBASE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573887#comment-13573887 ] Elliott Clark commented on HBASE-7763: -- For the bulk load scenario, sorting by size is very important. Take this example: Bulk loads every hour of small files: (ratio 1.2, max 3 files, min 2) [5000 20 20 20 20 20 ] The largest file is from compactions but has -1 as it's seqNum. So we'll be compacting [ 500 20 20 ] then [5040 20 20] then [5080 20 20] So I think we should always have a sort by size after the sort by seqid comparator, in addition to configurable additions talked about above. I've never seen a user put the same ts and expect to obscure the older one. (Sample size of 1 and all) But it seems like if we come up with something that's much better but have to give up that one line in the ref guide. It would be a fair trade. Compactions not sorting based on size anymore. -- Key: HBASE-7763 URL: https://issues.apache.org/jira/browse/HBASE-7763 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.96.0, 0.94.4 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch, HBASE-7763-trunk-TESTING.patch Currently compaction selection is not sorting based on size. This causes selection to choose larger files to re-write than are needed when bulk loads are involved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
