[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552231#comment-13552231 ] Devaraj Das commented on HBASE-7213: I am looking at the TestMultiParallel. I am at this point just can't see how this patch could cause the test failure.. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch, 7213-2.14.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552249#comment-13552249
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564621/TEST-org.apache.hadoop.hbase.client.TestMultiParallel.xml
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3993//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch,
7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.client.TestMultiParallel.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552418#comment-13552418
]
chunhui shen commented on HBASE-7213:
-
From
{code}
public HLog getWAL(HRegionInfo regionInfo) throws IOException {
//TODO: at some point this should delegate to the HLogFactory
//currently, we don't care about the region as much as we care about the
//table.. (hence checking the tablename below)
if (regionInfo != null
Arrays.equals(regionInfo.getTableName(), HConstants.META_TABLE_NAME)) {
return getMetaWAL();
}
{code}
what about ROOT region? we put its logs with other user regions?
In MetaServerShutdownHadnler, we consider META and ROOT both are meta regions.
Correct me if wrong
Thanks.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch,
7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.client.TestMultiParallel.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552424#comment-13552424 ] chunhui shen commented on HBASE-7213: - After fixing the above problem, passed TestMultiParallel for 15 times on my local PC. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch, 7213-2.14.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.client.TestMultiParallel.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551997#comment-13551997 ] Devaraj Das commented on HBASE-7213: bq. I have also attached the diff file that does a diff between the earlier committed patch (7213-2.11.patch) and the current one, for quick reference on what changed. Deleted the file since it might confuse hadoopqa.. (but yeah the difference between my earlier patch a version of which got committed and this new one is only in the HRegionServer.closeWAL, and a line length fix) Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552012#comment-13552012
]
Ted Yu commented on HBASE-7213:
---
Patch v14 looks good.
I ran the following tests locally and they passed:
{code}
684 mt -Dtest=TestCatalogTrackerOnCluster#testBadOriginalRootLocation
685 mt -Dtest=TestUpgradeFromHFileV1ToEncoding#testUpgrade
686 mt -Dtest=TestRestartCluster
{code}
There was one test failure which may not be related to the patch:
{code}
testReplayEditsAfterRegionMovedWithMultiCF(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
Time elapsed: 41.99 sec ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=10, exceptions:
Sat Jan 12 11:25:58 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
org.apache.hadoop.hbase.RegionMovedException: Region moved to:
hostname=10.120.104.184 port=56941.
Sat Jan 12 11:25:59 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.io.IOException: Call to /10.120.104.184:56941 failed on local exception:
java.io.EOFException
Sat Jan 12 11:26:00 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:01 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This server is
in the failed servers list: /10.120.104.184:56941
Sat Jan 12 11:26:03 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:05 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:09 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:13 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:22 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
Sat Jan 12 11:26:38 PST 2013, org.apache.hadoop.hbase.client.HTable$3@3c993730,
java.net.ConnectException: Connection refused
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:186)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:561)
at
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay.testReplayEditsAfterRegionMovedWithMultiCF(TestWALReplay.java:198)
{code}
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552017#comment-13552017
]
Ted Yu commented on HBASE-7213:
---
{code}
+ if (this.hlogForMeta != null) {
...
+this.hlogForMeta.close();
+ }
if (this.hlog != null) {
{code}
For the first if statement above, should we check whether this.hlog is null ?
If this.hlog is null, we should be calling this.hlogForMeta.closeAndDelete().
Since there is constraint on order of closing multiple WALs, it would better if
this logic is handled by FSHLog - can be done in another JIRA.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552024#comment-13552024
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564585/7213-2.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3989//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552031#comment-13552031 ] Devaraj Das commented on HBASE-7213: bq. For the first if statement above, should we check whether this.hlog is null ? If this.hlog is null, we should be calling this.hlogForMeta.closeAndDelete(). I don't think we need to worry about it. We will not have a case where _hlogForMeta_ is non-null and _hlog_ is null (_hlog_ is initialized before _hlogForMeta_). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552042#comment-13552042 ] Devaraj Das commented on HBASE-7213: Verified that the tests pass locally. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552074#comment-13552074
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564591/7213-2.14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.master.TestRollingRestart
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZKInternals(TestSplitTransactionOnCluster.java:738)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:541)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3991//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.14.patch, 7213-2.14.patch, 7213-2.4.patch, 7213-2.6.patch,
7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13552131#comment-13552131
]
chunhui shen commented on HBASE-7213:
-
Run the failed test TestMultiParallel on local PC for several times.
Without patch,passed 15 times;
With 7213-2.14.patch,4 failed of total 10;
Following is the failed tests' stacktrace:
{code}
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 37
actions: ExecutionException: 37 times,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.makeException(HConnectionManager.java:2106)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.rethrowIfAny(HConnectionManager.java:2089)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2071)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1850)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1839)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:601)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithPut(TestMultiParallel.java:303)
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 321
actions:
ExecutionException: 321 times,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.makeException(HConnectionManager.java:2106)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.rethrowIfAny(HConnectionManager.java:2089)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2071)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1850)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1839)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:601)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithDelete(TestMultiParallel.java:331)
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 321
actions:
ExecutionException: 321 times,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.makeException(HConnectionManager.java:2106)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.rethrowIfAny(HConnectionManager.java:2089)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2071)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1850)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1839)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:601)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testHTableDeleteWithList(TestMultiParallel.java:360)
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 100
actions:
ExecutionException: 100 times,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.makeException(HConnectionManager.java:2106)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.rethrowIfAny(HConnectionManager.java:2089)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2071)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1850)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1839)
at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:601)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithManyColsInOneRowGetAndPut(TestMultiParallel.java:394)
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 358
actions:
ExecutionException: 358 times,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process$BatchErrors.makeException(HConnectionManager.java:2106)
at
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551018#comment-13551018 ] Hudson commented on HBASE-7213: --- Integrated in HBase-TRUNK #3730 (See [https://builds.apache.org/job/HBase-TRUNK/3730/]) HBASE-7213 Have HLog files for .META. edits only (Revision 1431935) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551150#comment-13551150 ] Ted Yu commented on HBASE-7213: --- The test failure in TestSplitTransaction of QA run prevented large tests from running. I guess the failed tests (TestWALReplay, TestScannerTimeout) in trunk build #3730 were related to this patch. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551202#comment-13551202 ] Devaraj Das commented on HBASE-7213: I will look at the test failures shortly. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551302#comment-13551302 ] stack commented on HBASE-7213: -- [~ted_yu] Thanks Ted. Let me back out the patch till we get green light from DD. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551306#comment-13551306 ] stack commented on HBASE-7213: -- Or, I'll let it for a few hours... if you get a chance to look see and make an addendum, that'd be sweet [~devaraj] Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551387#comment-13551387
]
Ted Yu commented on HBASE-7213:
---
TestCatalogTrackerOnCluster failed in trunk builds #3730 and #3731. It fails
locally as well.
I saw the following in test output:
{code}
2013-01-11 10:42:33,443 ERROR [Shutdown of
org.apache.hadoop.hbase.fs.HFileSystem@2dc4df0b] hdfs.DFSClient(416): Failed to
close file
/user/tyu/hbase/.logs/10.10.8.161,51511,1357929747103/10.10.8.161%2C51511%2C1357929747103.1357929752224.meta
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/user/tyu/hbase/.logs/10.10.8.161,51511,1357929747103/10.10.8.161%2C51511%2C1357929747103.1357929752224.meta
File does not exist. [Lease. Holder:
DFSClient_hb_rs_10.10.8.161,51511,1357929747103_891897357_107, pendingcreates:
1]
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:729)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
{code}
Will attach test output.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch,
7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551412#comment-13551412 ] stack commented on HBASE-7213: -- I reverted for now. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551427#comment-13551427 ] Devaraj Das commented on HBASE-7213: Yeah the revert makes sense. Things may have changed in the codebase in the last couple of weeks leading to some test failures. I'm investigating more deeply now. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551495#comment-13551495 ] Hudson commented on HBASE-7213: --- Integrated in HBase-TRUNK #3733 (See [https://builds.apache.org/job/HBase-TRUNK/3733/]) HBASE-7213 Have HLog files for .META. edits only; REVERT (Revision 1432234) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551669#comment-13551669 ] Hudson commented on HBASE-7213: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #344 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/344/]) HBASE-7213 Have HLog files for .META. edits only; REVERT (Revision 1432234) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetaLogRoller.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch, 7213v13.txt, TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551772#comment-13551772
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564534/7213-2.12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
org.apache.hadoop.hbase.regionserver.wal.TestWALReplay
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3988//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch,
7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch,
7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551794#comment-13551794
]
Ted Yu commented on HBASE-7213:
---
{code}
+ private static void closeWAL(final boolean delete, HLog hlog) {
try {
- if (this.hlog != null) {
+ if (hlog != null) {
{code}
The above code deals with hlog parameter which happens to eclipse this.hlog
It would be better to distinguish the parameter more clearly from member
variable.
Currently HRegionServer has knowledge about the WAL for .META. and normal WAL.
We can revisit this when multi-WAL is to be supported.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.12.patch,
7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch,
7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch,
7213v13.txt,
TEST-org.apache.hadoop.hbase.catalog.TestCatalogTrackerOnCluster.xml
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550423#comment-13550423
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564237/7213-2.10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZKInternals(TestSplitTransactionOnCluster.java:738)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:541)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3965//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.4.patch, 7213-2.6.patch,
7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550873#comment-13550873 ] stack commented on HBASE-7213: -- I was trying this [~devaraj] and if fails to apply to trunk. I tried to get it in but there is a big difference in this file: hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java.rej Mind taking look see sir? Looking at the patch again, you might move the define that is out in HConstants local to the wal package: public static final String META_HLOG_FILE_EXTN = .meta; Put it in HLog.java Interface? In MasterFileSystem, should a few of the public methods have some javadoc or comment? For instance, I see splitMetaLog twice but one takes a ServerName and another takes a list of ServerNames but their bodies do different things. It is a little confusing. Else looks good. Lets get it in quick. Elsewhere fellas are talking about removing MetaServerShutdown handler so would require yet another patch moving stuff around. Good stuff. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550923#comment-13550923 ] Devaraj Das commented on HBASE-7213: Sorry, forgot to mention that I took into consideration the last 2-3 comments from [~stack] in the last patch (and in the process, removed one unnecessary splitMetaLog). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550942#comment-13550942
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564369/7213-2.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3979//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.10.patch, 7213-2.11.patch, 7213-2.4.patch,
7213-2.6.patch, 7213-2.8.patch, 7213-2.9.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1354#comment-1354 ] Devaraj Das commented on HBASE-7213: Folks, should we commit this. It has been nearly 1.5 months since the patch was reviewed and +1'ed. Since this patch is somewhat big, I'd prefer this be closed sooner than later.. (can I do anything to expedite the commit for this patch). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549181#comment-13549181
]
Ted Yu commented on HBASE-7213:
---
There're a few conflicts for patch v8:
{code}
-rw-r--r-- 1 tyu staff 1578 Jan 9 15:24
./hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java.rej
-rw-r--r-- 1 tyu staff 3086 Jan 9 15:24
./hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java.rej
-rw-r--r-- 1 tyu staff 1380 Jan 9 15:24
./hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java.rej
-rw-r--r-- 1 tyu staff 323 Jan 9 15:24
./hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java.rej
{code}
After they're fixed, let Hadoop QA run the test suite.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549302#comment-13549302
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564079/7213-2.9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestScannerTimeout
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.master.TestMasterFailover
org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZKInternals(TestSplitTransactionOnCluster.java:738)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:541)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3956//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549344#comment-13549344
]
Ted Yu commented on HBASE-7213:
---
Running tests mentioned above against latest patch:
{code}
test3686b(org.apache.hadoop.hbase.client.TestScannerTimeout) Time elapsed:
0.007 sec ERROR!
java.lang.ArrayIndexOutOfBoundsException: -1
at
java.util.concurrent.CopyOnWriteArrayList.get(CopyOnWriteArrayList.java:343)
at java.util.Collections$UnmodifiableList.get(Collections.java:1152)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getRSForFirstRegionInTable(HBaseTestingUtility.java:1456)
at
org.apache.hadoop.hbase.client.TestScannerTimeout.test3686b(TestScannerTimeout.java:199)
{code}
It may not be related to the patch.
The following two may need closer look:
{code}
testDataCorrectnessReplayingRecoveredEdits(org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster)
Time elapsed: 85.445 sec ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
action: ExecutionException: 1 time,
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.processBatchCallback(HConnectionManager.java:2063)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$Process.access$900(HConnectionManager.java:1850)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1839)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1818)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:885)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:695)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:670)
at
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster.putDataAndVerify(TestHRegionOnCluster.java:127)
at
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster.testDataCorrectnessReplayingRecoveredEdits(TestHRegionOnCluster.java:115)
{code}
{code}
testCorrectnessWhenMasterFailOver(org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing)
Time elapsed: 23.843 sec FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at
org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing.testCorrectnessWhenMasterFailOver(TestRSKilledWhenMasterInitializing.java:188)
{code}
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-2.9.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530719#comment-13530719
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12556217/7213-2.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
102 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 23 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster
org.apache.hadoop.hbase.TestDrainingServer
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3511//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch,
7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510706#comment-13510706
]
stack commented on HBASE-7213:
--
Do other logs have a '.log' suffix? You add .hlog with the below?
+ /** The META region's HLog filename extension */
+ public static final String META_HLOG_FILE_EXTN = .meta.hlog;
The refactoring in MasterFileSystem to make a new getLogDirs method is nice.
The SSH changes look good. There is a hole though at the moment around -ROOT-
handling? We need to wait on -ROOT- removal before this can go in, right?
(The RootServerShutdownHandler would goor should RSSH be calling MSSH when
it is done w/ its process?)
This below is given to an executor? Should have a better name than handler?
+ private UncaughtExceptionHandler handler;
Will we always create the metahlog just because getMetaHLog was called? Even
if we are NOT carrying .META.:
+ private HLog getMetaWAL() throws IOException {
+if (this.hlogForMeta == null) {
If .META. moves away from a RS and then comes back, we'll just use the already
made meta log roller, etc.?
Replication will skip this .META. log?
It is intentional that MetaServices is still in the patch?
Changing param name from logName to logDir is good stuff
In SplitLogManager, we have added splitMetaLogDistributed. A more generic
method might have been splitLogDistributed that took a file path filter
instead...
There is a define for .META. HRegionInfo in HRegionInfo#FIRST_META_REGIONINFO
so you don't have to make it each time, FYI.
Don't have to deprecate the below I'd say. This is for 0.96 the singularity
and this is on a class with annotation @InterfaceAudience.Private
/** @return the HLog */
+ @Deprecated
public HLog getWAL();
+
+ /** @return the HLog for a particular region. Pass null for getting the
+ * default (common) WAL */
+ public HLog getWAL(HRegionInfo regionInfo) throws IOException;
Here
-return new Path(dir, prefix + . + filenum);
+String child = prefix + . + filenum;
+if (forMeta) {
+ child += HConstants.META_HLOG_FILE_EXTN;
+}
+return new Path(dir, child);
the 'normal' WALs do not seem to pick up the suffix in here. Is it possible
that the '.log' is appended elsewhere? Are your .META. logs getting double
'.log'? Just wondering.
Good stuff
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510829#comment-13510829
]
Devaraj Das commented on HBASE-7213:
bq. Do other logs have a '.log' suffix? You add .hlog with the below?
Currently (in the trunk codebase), the log files don't have extension like .log
or something (the files look like
192.168.1.8%2C60020%2C1354695930505.1354695931168). They are in directory
called .logs/region-server. So I am wondering whether I should just have
.meta as the suffix for the meta log files, and be done with it (as I had it
in the previous patch). In the current patch, the meta logs have .meta.hlog as
the extension. Or, would it make sense to change the log files to have .hlog
extension as well? Please let me know..
bq. The SSH changes look good. There is a hole though at the moment around ROOT
handling? We need to wait on ROOT removal before this can go in, right? (The
RootServerShutdownHandler would goor should RSSH be calling MSSH when it is
done w/ its process?)
It can be done either way in theory (HBASE-3171 first or this first). On the
hole, yes, if this patch goes in first, and if there is a case where the same
RS hosts both the root and meta regions, and that RS goes down, and the root's
edits weren't already persisted, there will be a problem. But given that the
root doesn't have edits other than the one row to do with .META., it is
unlikely that we will run into the hole in practice. But am okay to wait till
HBASE-3171 goes in (and that's a better way to line the commits, but would like
to close on on the other comments).
bq. This below is given to an executor? Should have a better name than handler?
bq. + private UncaughtExceptionHandler handler;
Will take a look and update.
bq. Will we always create the metahlog just because getMetaHLog was called?
Even if we are NOT carrying .META.:
No. It'll be created on the first call to getMetaWAL (and getMetaWAL gets
indirectly called only when we are asked to open the meta region in
OpenRegionHandler)
bq. If .META. moves away from a RS and then comes back, we'll just use the
already made meta log roller, etc.?
Yes.
bq. Replication will skip this .META. log?
Yes. The MetaLogRoller doesn't register with the Replication folks when
instantiated.
bq. It is intentional that MetaServices is still in the patch?
Mistake.. Will update.
bq. In SplitLogManager, we have added splitMetaLogDistributed. A more generic
method might have been splitLogDistributed that took a file path filter
instead...
Hmm.. Let me see..
bq. There is a define for .META. HRegionInfo in
HRegionInfo#FIRST_META_REGIONINFO so you don't have to make it each time, FYI.
I had seen this one but had forgotten to update code to use it.. will update.
bq. Don't have to deprecate the below I'd say. This is for 0.96 the singularity
and this is on a class with annotation @InterfaceAudience.Private
bq. /** @return the HLog */
bq. + @Deprecated
bq. public HLog getWAL();
Okay..
{quote}
Here
return new Path(dir, prefix + . + filenum);
+ String child = prefix + . + filenum;
+ return new Path(dir, child);
the 'normal' WALs do not seem to pick up the suffix in here. Is it possible
that the '.log' is appended elsewhere? Are your .META. logs getting double
'.log'? Just wondering.
{quote}
No.. as I said before, the log files don't have the string extensions.. So what
do you think - should I just remove the .hlog from the .meta.hlog extension
that I did for meta hlogs? Or, leave .meta.hlog as it is and add .hlog to the
non-meta hlog files, or, leave the non-meta hlog names as they are now?
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.0
Attachments: 7213-2.4.patch, 7213-in-progress.2.2.patch,
7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510843#comment-13510843 ] stack commented on HBASE-7213: -- bq. So I am wondering whether I should just have .meta as the suffix for the meta log files, and be done with it (as I had it in the previous patch). Yes. The '.hlog' is extraneous (I thought all log files had .log suffix -- sorry if I mislead) One more spin and I'd say this patch is ready to go in. Good on you DD. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-2.4.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511030#comment-13511030 ] Devaraj Das commented on HBASE-7213: I ran tests locally with the patch and they passed. I also ran some manual tests (killing region servers after making sure some edits weren't persisted, etc.). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13511109#comment-13511109 ] stack commented on HBASE-7213: -- This looks new (and important): +splitLog(serverNames, META_FILTER); +splitLog(serverNames, NON_META_FILTER); I can remove this on commit: /** @return the HLog */ public HLog getWAL(); Patch looks good to me. Lets wait some time on hbase-3171. If it doesn't show up soon, we'll commit this. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509573#comment-13509573 ] Devaraj Das commented on HBASE-7213: bq. .META. file ends in '.META' and not '.META.'? Can't it end in .log? xx.log.META looks a bit odd and I'm sure will trip up someone else trying to find WALs for whatever reason. Hmm.. .log.META sounds better.. bq. How does splitMetaLog work? It is passed a server name and if we find logs in the filesystem, then we'll split its logs? Yes.. (much like how it's currently done for the non-meta logs) bq. What if .META. is currently assigned? The splitting will be to no avail? Can you please explain when this can happen (that a RS hosting the .META. crashes, and before we do the log splitting, the .META. gets assigned to someone else)? But in general, the behavior shouldn't change in the patch from what the behavior currently is... bq. When would we ever log this message: bq. LOG.info(No logs to split); This is the same log that gets emitted for the general case (in the current codebase). Look at splitLog method in MasterFileSystem. Seems like when the logdir (.logs) doesn't exist, getLogDirs returns an empty list. bq. Should that be .META. logs to split? Yes.. bq. Methods usually have a space between them rather than: Ack. I missed adding a space. I usually always add spaces without fail (smile) bq. Why emit two log lines when you could have one that varies with whether meta or not: This was mostly for my debugging. I'll update this.. bq. Should be a isMetaWAL method? to do the looking for META_LOG_FILE_EXTN... since used in a few places. Yeah.. bq. We maybe should have done it before but was a bit tougher since had to do the log splitting first... could have broke up the process more .. I went that route and backtracked.. The problem is that when a RS crashes, we will probably have that RS hosting many regions (including the catalog ones). To recover we have to call process() methods of both SSH and MSSH as a pair. That seemed harder to maintain.. So I kept the existing semantics of one SSH process doing the work for both META and non-meta regions. As I have noted in a previous comment, I have explicitly not handled -ROOT- since the root region is going to disappear post HBASE-3171. bq. Is this needed? Yeah, since the process() method handles both .META. and non-meta regions. My earlier comment explains why I am not very inclined to separate the handling in process(). But I'll reconsider this part and see if I can do something better. bq. MetaServices Interface doesn't seem right? Its all about a WAL. You wouldn't have to add new methods? Especially public ones? Yeah, I shouldn't have made that a public class, etc. and maybe called it MetaWALServices or something. Would doing that alleviate this concern? bq. When we lose the .META. region, does the RS clean up its .META. log? I need to check on this. bq. I don't like extending the RegionServerServices because makes it harder mocking up an RSS... more stuff to add in. The new methods could just return null (and that's what I have done in the existing mock classes). But I'll consider this. bq. Should it be MetaFSHLog rather than FSLog w/ boolean args in constructor? Just wondering. You mean another class.. Seems overkill to me.. bq. We have to add this method? bq. public static HLog createMetaHLog(final FileSystem fs, final Path root, final String logName, bq. Nothing in say the log name that would clue us its a .META. file? Could we preface the logName with .META. or something and then we don't have to pass flags to FSHLog? Could have static isMetaWAL method that looks at log name and can tell it .META.? The logname (and this is there in the current trunk) is kind of misleading. It actually signifies the directory name (and maybe I should update the code to call it logdir everywhere)... So in that sense, there is nothing indicative of meta. Also, I'd prefer passing explicit booleans rather than string based conditionals.. bq. Is this used: private static final Pattern pattern = Pattern.compile(.\\. d); + private static final Pattern pattern = Pattern.compile(.\\. d(.META)*); Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509575#comment-13509575 ] Devaraj Das commented on HBASE-7213: Sorry, I hit the submit button too soon. Ignore the last few lines in the last comment (starting from Is this used:). On the last bit: bq. Is this used: bq. private static final Pattern pattern = Pattern.compile(.\\.d); bq. + private static final Pattern pattern = Pattern.compile(.\\.d(.META)*); Yes, otherwise the master deletes the .META hlog files.. (this makes it so that files are accepted with .META extensions). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510057#comment-13510057 ] Devaraj Das commented on HBASE-7213: bq. When we lose the .META. region, does the RS clean up its .META. log? I don't think I need to do anything specific here. The existing code should handle this already. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508999#comment-13508999 ] Enis Soztutar commented on HBASE-7213: -- bq. and in case folks feel that ROOT should be handled as well, it probably can be done in a follow up I think JD has updated the patch for removing ROOT in favor of ZK. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509513#comment-13509513
]
Hadoop QA commented on HBASE-7213:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555829/7213-in-progress.2.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.catalog.TestMetaReaderEditor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3443//console
This message is automatically generated.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.0
Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509536#comment-13509536 ] Devaraj Das commented on HBASE-7213: TestMetaReaderEditor (timed out in the hadoopqa run) passes locally.. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509537#comment-13509537
]
stack commented on HBASE-7213:
--
.META. file ends in '.META' and not '.META.'? Can't it end in .log?
xx.log.META looks a bit odd and I'm sure will trip up someone else trying
to find WALs for whatever reason.
How does splitMetaLog work? It is passed a server name and if we find logs in
the filesystem, then we'll split its logs? What if .META. is currently
assigned? The splitting will be to no avail? When would we ever log this
message:
{code}
+ LOG.info(No logs to split);
{code}
Should that be .META. logs to split?
Methods usually have a space between them rather than:
{code}
+ }
+ /**
{code}
... in hbase codebase boss.
Why emit two log lines when you could have one that varies with whether meta or
not:
{code}
LOG.info(Done splitting + path);
+if (path.endsWith(HConstants.META_HLOG_FILE_EXTN)) {
+ LOG.info(Done splitting meta region);
+}
{code}
Should be a isMetaWAL method? to do the looking for META_LOG_FILE_EXTN...
since used in a few places.
In process, we have code for meta and root handling in the general
ServerShutdownHandler. Should the root handling be up in
RootServerShutdownHandler, in its process, which then calls through to the
general SSH#process and ditto for the .META.? (Might mean ROOT needs to
subclass Meta for now... ). We maybe should have done it before but was a bit
tougher since had to do the log splitting first... could have broke up the
process more ... but might be easier now these root and meta handlings are
first thing done in the process method.
Is this needed?
{code}
-this.services.getExecutorService().submit(this);
+//typecast to SSH so that we don't do the above meta log split again
+this.services.getExecutorService().submit((ServerShutdownHandler)this);
{code}
Would make sense if you move the meta specific code to metaSSH, and ditto on
ROOT?
MetaServices Interface doesn't seem right? Its all about a WAL. What else
would ever go in here? Who uses this new Interface and its methods? I don't
see anything else in this patch using it. I suppose MetaLogRoller does but it
is sitting beside RegionServer -- could be protected or package private -- yet
its public in the Interface. Hmm... i see that OpenRegionHandler uses it.
Can't RegionServer notice when its a meta region and do the setup etc.,
internally? Ugh, maybe not looking at this OpenRegionHandlerhow it needs
the exact WAL? What if you passed in the regionname to getWAL? Then it
internally would do the right thing. You wouldn't have to add new methods?
Especially public ones?
When we lose the .META. region, does the RS clean up its .META. log?
I don't like extending the RegionServerServices because makes it harder mocking
up an RSS... more stuff to add in.
Should it be MetaFSHLog rather than FSLog w/ boolean args in constructor? Just
wondering.
You have this: +LOG.info(CREATED writer + path); //REMOVETHIS
We have to add this method?
+public static HLog createMetaHLog(final FileSystem fs, final Path root,
final String logName,
+final Configuration conf, final ListWALActionsListener listeners,
+final String prefix) throws IOException {
+ return new FSHLog(fs, root, logName, HConstants.HREGION_OLDLOGDIR_NAME,
+conf, listeners, false, prefix, true);
+}
Nothing in say the log name that would clue us its a .META. file? Could we
preface the logName with .META. or something and then we don't have to pass
flags to FSHLog? Could have static isMetaWAL method that looks at log name and
can tell it .META.?
Import this in LogSplitter but not used: +import
org.apache.hadoop.fs.PathFilter;
Is this used:
- private static final Pattern pattern = Pattern.compile(.*\\.\\d*);
+ private static final Pattern pattern = Pattern.compile(.*\\.\\d*(.META)*);
Good stuff.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Fix For: 0.96.0
Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch,
7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA,
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508425#comment-13508425
]
Ted Yu commented on HBASE-7213:
---
{code}
+ListServerName serverNames = new ArrayListServerName();
+serverNames.add(serverName);
+ListPath logDirs = getLogDirs(serverNames);
{code}
The first two lines above can be moved below the if block which checks
logDirs.isEmpty()
{code}
+ * @throws IOException
+ * if there was an error while splitting any log file
+ * @return cumulative size of the logfiles split
+ * @throws IOException
+ */
+ public long splitMetatLogDistributed(final ListPath logDirs) throws
IOException {
{code}
Please group the @throws lines together. The method name is misspelled - I
couldn't find where it is called.
I think the above method is eclipsed by splitMetaLogDistributed() which appears
later.
{code}
+ if (isCarryingMeta() || isCarryingRoot()) {
+try {
+ LOG.info(Splitting META logs for + serverName);
+ if (this.shouldSplitHlog) {
+this.services.getMasterFileSystem().splitMetaLog(serverName);
{code}
The check for ROOT region above is for assigning ROOT. Suggest making the check
of ROOT and .META. separate so that the logic is clearer.
{code}
+ protected HLog instantiateMetaHLog(Path rootdir, String logName) throws
IOException {
+return HLogFactory.createMetaHLog(this.fs.getBackingFs(), rootdir,
logName, this.conf,
+getMETAWALActionListeners(), this.serverNameFromMasterPOV.toString());
+ }
{code}
The above method is only called by setupMetaWAL(). Can we merge it with
setupMetaWAL() ?
nit: getMETAWALActionListeners(), do we need to make all 4 letters of Meta
capitalized ?
{code}
+ LOG.info(ISALIVE: + leases.isAlive() + +cacheFlusher.isAlive() +
+ this.compactionChecker.isAlive() + + hlogRoller.isAlive() + +
metaHlogRoller.isAlive()); //REMOVETHIS
{code}
As the comment says: remove in next patch.
{code}
+class MetaLogRoller extends LogRoller {
{code}
Consider putting MetaLogRoller in its own class.
For MetaServices.java, please add license and audience annotation.
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Attachments: 7213-in-progress.2.patch, 7213-in-progress.patch
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507250#comment-13507250 ] nkeywal commented on HBASE-7213: To me, there are some stuff that are nice to have in multiwal but very useful here: - it's better to have only one table in the wal with .meta. - it's would be great to be able to configure the replication factor for this wal - having a separate wal for meta should be the only option (it could make sense short term to have it as an option for safety, but it would be temporary) - there is no real split of the .meta. wal (at least conceptually: all regions ends up on the same server..). There is some room for (premature?) optimization there. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506314#comment-13506314 ] Ted Yu commented on HBASE-7213: --- I wonder what kind of data structure should be in place for the FSHLog instances (two for the region server hosting .META.) so that multiple WAL implementation would be more intuitive. Previously, close / deletion of HLog operates on the single instance. Now things start to get more interesting. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506730#comment-13506730 ] Todd Lipcon commented on HBASE-7213: I agree it seems to make sense to lump this with the multi-WAL work. Perhaps an interface like WALFactory or WALProvider, which, given a region name, gives back a WAL instance? The basic implementation would always provide the single WAL. Then, we could add the feature that returns a different WAL for META alone. More complex implementations could choose to give different tenants of a cluster separate WALs, etc. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506884#comment-13506884 ] Ted Yu commented on HBASE-7213: --- We already have HLogFactory in trunk. bq. to lump this with the multi-WAL work Can I interpret the above as saying that multi-WAL work should be done at the same time, if not earlier ? Since HLogFactory can hand out the unique instance for .META., it is not far from handing out different instances (for different regions) which is what HBASE-5699 tries to do. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506923#comment-13506923 ] Devaraj Das commented on HBASE-7213: [~tlipcon], good points there. But I'd like to separate the META SPOF work from the full fledged multiwal work (for the full multi-wal case, we'd need to fix up things like replication, and those can be skipped from the meta-only design that this jira attempts to do). [~yuzhih...@gmail.com], I guess you are right. In theory, one could read the HLogFactory as the WALFactory... Thoughts? (i am in the process of extending the patch I previously posted to a fully functional one) Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506953#comment-13506953 ] Todd Lipcon commented on HBASE-7213: Yea, I don't think you need to strictly sequence this after the multi-WAL work. But it would be nice to have the end goal in mind while doing this work. Sorry, haven't had time to look at the in-progress patch, but if there's a simple solution that works OK now, no sense blocking it for the perfect end-game solution later. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505501#comment-13505501 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- @Deva Patch looks good.. We may need to check on log rolling, log splitting etc... One thing is if the META has just moved from one RS to another RS and the first RS crashes. We will be having a the HLog for the META also with it(because it was holding it).. So in this i think we need not split that META hlog right? Correct me if am wrong here. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505785#comment-13505785 ] Devaraj Das commented on HBASE-7213: [~ram_krish], thanks for looking. Yes, this patch is not complete yet and the next iteration will have the missing functionalities. I'll also answer your questions in the next iteration. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505791#comment-13505791 ] stack commented on HBASE-7213: -- [~ram_krish] On if a .META. moved from one RS to another, and first crashes, then when we go to recover, we will see it wasn't carrying .META. so we won't try and recover it (as is the case w/ any other region). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505803#comment-13505803 ] stack commented on HBASE-7213: -- On the approach in the sketch patch: Passing a flag to instantiateHLog when its meta is a little awkward. Ripples all the ways down. Would instantiateMetaHLog and createMetaHLog in the factory be more plain about what is going on? Runs into a wall at FSHLog though though here, is there anything in the log name that would denote it a .META. WAL? Patch doesn't look bad so far. Not too intrusive for big payback. I was thinking this facility would come in as part of the multiwal work... if multiwals, make provision for an extra if carrying .meta. but this seems like shorter path. Do you think this work will get in the way of the multiwal feature? Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505896#comment-13505896 ] Devaraj Das commented on HBASE-7213: [~stack], yes, I am trying to keep the patch as less intrusive as I can. The fact that the multiwal is being done for META only helps towards that (for example, replication won't need to be tweaked since replication is not done for META entries, etc.). I'll check on the flag comment and see if I can introduce new methods rather than use a flag.. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506204#comment-13506204 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- bq.we will see it wasn't carrying .META. so we won't try and recover it Yes that is true...MetaServerServerShutDownHandler alone should be handling the split of META hlog. The normal SSH will handle only the other HLog. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505217#comment-13505217 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- Deva, I could not apply the code to trunk as i don have the trunk code with me now.. (will apply it later once i go home). Just one question i have here.. If the META region moves newly to an RS we instantiate the HLog for META. If an RS that had the META once and again gets back the META in that case we should be pointing to the same HLog right? May be if the code does it already pardon mebecause i could not apply the code i had this doubt. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505247#comment-13505247 ] Devaraj Das commented on HBASE-7213: bq. If an RS that had the META once and again gets back the META in that case we should be pointing to the same HLog right? Yes, [~ram_krish], it should be pointing to the same HLog. I am wondering though when could this happen. I can see the obvious case where a RS dies and the meta gets assigned to some other RS, but when does the former get it back? Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505253#comment-13505253 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- Thanks Deva. Normally not possible. But seeing the 0.94 code atleast we dont block in issuing an assign or move command to META. !! Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505254#comment-13505254 ] nkeywal commented on HBASE-7213: Is it something forbidden? Because there is a (fixed) jira that says we should be able to do it (HBASE-3756). It's obviously critical, but I think it's requires for rolling restart, where we move all the regions from a server before stopping/starting it. We could also imagine balancers doing this... Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505256#comment-13505256 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- @N Yes balancer is also one thing which is doing it. just verified the code. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505261#comment-13505261 ] Devaraj Das commented on HBASE-7213: Thanks for the clarifications, [~ram_krish] and [~nkeywal]. The thought of moving meta sounds scary to me though :-) [but this patch doesn't intend to affect that part]. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504032#comment-13504032 ] stack commented on HBASE-7213: -- Yes Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504230#comment-13504230 ] Enis Soztutar commented on HBASE-7213: -- This can also allow us to directly deploy META without waiting for log splitting (if we have separate WAL for every META region, there is nothing to split). Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504398#comment-13504398 ] Devaraj Das commented on HBASE-7213: Apologies for not responding earlier.. But yes, the high level idea is to make the recovery of META table fast and hence reduce the downtime of META should the regionserver hosting the RS crash. By having a dedicated HLog for META, the log splitting will be really simple (maybe just simple rename operations of files; need to look at the code here)... I am in the middle of plumbing through the code to make this work. Will hopefully have something in a day or two. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504404#comment-13504404 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- @Deva Thanks for the update. I remeber there is a hlog created inside a hregion also which is never getting used? Can we make use of it? Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504413#comment-13504413
]
Devaraj Das commented on HBASE-7213:
Will take a look [~ram_krish], but is there any particular reason why that
should be reused? (are you referring to the field {noformat}final HLog
log;{noformat}).
Have HLog files for .META. edits only
-
Key: HBASE-7213
URL: https://issues.apache.org/jira/browse/HBASE-7213
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Devaraj Das
Assignee: Devaraj Das
Over on HBASE-6774, there is a discussion on separating out the edits for
.META. regions from the other regions' edits w.r.t where the edits are
written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503342#comment-13503342 ] ramkrishna.s.vasudevan commented on HBASE-7213: --- Is the idea to write it in a seperate WAL file? So that i can be picked up first? Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
