[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450188#comment-13450188 ] Hudson commented on HBASE-6659: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #163 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/163/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt, 6508-v9.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449887#comment-13449887 ] Hudson commented on HBASE-6659: --- Integrated in HBase-TRUNK #3310 (See [https://builds.apache.org/job/HBase-TRUNK/3310/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt, 6508-v9.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449828#comment-13449828 ] Ted Yu commented on HBASE-6659: --- Integrated to trunk. Thanks for the review, Stack and Ram. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt, 6508-v9.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449824#comment-13449824
]
Ted Yu commented on HBASE-6659:
---
{code}
Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.17 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.567 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogFiltering
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.173 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.719 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 234.204 sec
Results :
Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
{code}
Going to commit.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
> 6508-v7.txt, 6508-v8.txt, 6508-v9.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449819#comment-13449819 ] ramkrishna.s.vasudevan commented on HBASE-6659: --- @Ted Nice useful backport. Thanks for taking it up. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt, 6508-v9.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449817#comment-13449817 ] stack commented on HBASE-6659: -- I took a quick look. +1. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt, 6508-v9.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449809#comment-13449809 ] stack commented on HBASE-6659: -- Maybe check out the TestHLogMethods failure... unlikely it this change but worth a look I'd say. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449808#comment-13449808
]
stack commented on HBASE-6659:
--
+1
To me at least, we encapsulate the sequenceid counting inside ServerManager
rather than have it spread between SM and HM (also, don't need to add an
esoteric method to the generic MasterServices Interface). HLogSplitter is
passed a specific Interface for getting last sequenceid rather than a protocol
implementation. I think this better. Thanks for entertaining my reviews.
On commit remove '+ * Copyright The Apache Software Foundation' and change this
'+ * @param master' (param name is different).
This change seems gratuitous but it is from original patch so its ok:
{code}
-regionInfo.getTableName(), completeSequenceId,
+regionInfo.getTableName(), completeSeqId,
{code}
Are the imports in MasterServices needed? (they don't seem to be referred to by
this patch). Remove on commit.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
> 6508-v7.txt, 6508-v8.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449805#comment-13449805 ] Hadoop QA commented on HBASE-6659: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544072/6508-v8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 107 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//console This message is automatically generated. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v8.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449453#comment-13449453 ] stack commented on HBASE-6659: -- bq. They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager. Sorry. I don't follow what you are saying above. updateLastFlushedSequenceIds is implemented in HMaster and used by ServerManager. So ServerManager can use updateLastFlushedSequenceIds you must add it to MasterServices. Can you better encapsulate this new sequenceid to region accounting by moving it all into ServerManager so you do not have to do this back and forth between SM and HM and you can avoid adding more facility to the MasterServices Interface? Looking at updateLastFlushedSequenceIds implementation I do not see an obvious reason why we could not but I may be missing something. bq. We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter. HLogSplitter would be able to consume this new method. The above is basically right (We'd pass a reference to a LastSequenceId Interface -- though the actual instance may be a RegionServerServices that implements LastSequenceId). Do you think this makes sense? Would it be a PITA to do? We'd pass this Interface instead of passing in the crazy RegionServerStatusProtocol masterRef (A LastSequenceId would be easy to mock, a RegionServerStatusProtocol would be a pain) > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449251#comment-13449251 ] Ted Yu commented on HBASE-6659: --- For last flushed sequence Id, another option is to embed it in HRegionInfo. This way, there is no need to modify RegionLoad. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447995#comment-13447995 ] Ted Yu commented on HBASE-6659: --- @Stack: What do you think of my proposal above ? Thanks > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446640#comment-13446640
]
Ted Yu commented on HBASE-6659:
---
bq. load is carrying non-load attributes
I will add javadoc as you suggested on review board.
bq. a bit of improvement over having to pass info from HMaster to SM and then
back to HMaster
Sequence Id information is passed from region servers to ServerManager, from
ServerManager to Master, if that is what you meant.
I checked a few methods in ServerManager, such as:
{code}
public Map getOnlineServers() {
public Set getDeadServers() {
{code}
They are all consumed in org.apache.hadoop.hbase.master namespace. Since
ServerManager doesn't implement interface, currently I don't see how
LastFlushedSequenceIds can be encapsulated within ServerManager.
w.r.t. creating LastSequenceId interface, currently neither SplitLogWorker nor
HLogSplitter has reference to RegionServerServices.
If I understand your suggestion correctly, 'long getLastSequenceId(regionname)'
can be added to RegionServerServices. We pass reference of RegionServerServices
from region server to SplitLogWorker, then to HLogSplitter.
HLogSplitter would be able to consume this new method.
Once I get your confirmation, I would come up with patch v8.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Zhihong Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
> 6508-v7.txt, 6508-v7.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446609#comment-13446609 ] stack commented on HBASE-6659: -- bq. I searched AssignmentManager and don't see regions field. My bad. It was in 0.94. Its not in trunk (Jimmy removed it recently) bq. w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the patch is due to the fact that regionServerReport() is in ServerManager. ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to deliver status update to HMaster. I do not follow. What is the above justifying? Should the map of sequenceids to regions be in ServerManager and not in Master so you don't have to leak it out to the HMaster wrapper class? (Would this mean you would not have to add updateLastFlushedSequenceIds to MasterServices? ServerManager deals w/ Load as it is... its tough because this stuff is so messy... load is carrying non-load attributes... keeping the mess confined to SM would be a bit of improvement over having to pass info from HMaster to SM and then back to HMaster) bq. Could we put the rpc behind an Interface at least? What I meant is would it make sense making an Interface named LastSequenceId whose content was a method named: long getLastSequenceId(regionname); Its implementation would have RS invoke method on Master but at least we'd be making it so HLogSplitter didn't have to be polluted w/ some protocol implementation. Interface would also make it easier testing HLogSplitter. I took a look at v7. Most of my comments above are informed by a review of it up in rb > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446384#comment-13446384 ] Ted Yu commented on HBASE-6659: --- TestReplication was missing in PreCommit build #2756. Other tests passed. I also updated https://reviews.apache.org/r/6768 with patch v7. Please provide your comments. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446364#comment-13446364 ] Hadoop QA commented on HBASE-6659: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543331/6508-v7.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 108 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 11 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2756//console This message is automatically generated. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v7.txt, 6508-v7.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446142#comment-13446142
]
Ted Yu commented on HBASE-6659:
---
bq. Do we not have the methods getRegionsByServer in HBaseTestingUtility already
'In HBaseTestingUtility, I don't see getRegionsByServer()' was for above
comment.
bq. You can pass null to the first IIRC... at least the second should call the
first.
bq. you're right that flushRegion(final RpcController controller, final
FlushRegionRequest request) requires the composition of FlushRegionRequest.
I will remove flushRegion(byte[] regionName) method and replace the call with
one to the existing method.
bq. There is at least AM#regions
I searched AssignmentManager and don't see regions field.
w.r.t. adding updateLastFlushedSequenceIds(), the current formation of the
patch is due to the fact that regionServerReport() is in ServerManager.
ServerManager is marked @InterfaceAudience.Private, so ServerManager needs to
deliver status update to HMaster.
bq. Could we put the rpc behind an Interface at least?
To my understanding, RegionServerStatusProtocol is an interface:
{code}
public interface RegionServerStatusProtocol extends
{code}
Will upload new patch.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Zhihong Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
> 6508-v6.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446111#comment-13446111 ] stack commented on HBASE-6659: -- Your responses are many and do not cite to what you are responding which makes it tough aligning what your response refers to. For example, this comment: bq. In HBaseTestingUtility, I don't see getRegionsByServer(). To what of my remarks is it in reply to? On the keeping TestHLogFiltering, thats fine. You give good rationale. bq. The timing of this call is not in sync with region server heartbeat. Nevermind my comment; I had the workings mixed up. bq. But I found that code would get more convoluted that way. Yes. Its super ugly rpc'ing the master from down deep in hlogsplitter. Could we put the rpc behind an Interface at least? bq. w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client. I don't follow what you are saying above. bq. Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable. My rationale for not adding methods has nothing to do w/ whether a release has been made or not; its about keeping these Interfaces as basic as possible so mocking does not become burdensome (when mocking is burdensome, fellas will skip on writing tests) bq. w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name). There is at least AM#regions Its use is different but I'd think some effort would be expended trying to make it so we don't proliferate state keepers per facility added to hbase. Maybe its not possible. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, > 6508-v6.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446016#comment-13446016 ] Ted Yu commented on HBASE-6659: --- I mentioned the need for adding updateLastFlushedSequenceIds() @ 29/Aug/12 23:01 Since 0.96 hasn't been released, I think adding methods to MasterServices should be acceptable. w.r.t. flushedSequenceIdByRegion map, there is no other map in HMaster which is keyed by byte[] (region name). > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445962#comment-13445962 ] Ted Yu commented on HBASE-6659: --- In HBaseTestingUtility, I don't see getRegionsByServer(). TestHLog is a large test. There would be more tests added to TestHLog in the future. According to N's motto, we should keep runtime for large tests in check. So I keep TestHLogFiltering for now. New method getLastFlushedSequenceId() is called by HLogSplitter. The timing of this call is not in sync with region server heartbeat. bq. We make a call to the master per region from down inside HLogSplitter to ask for last known sequence id? For 'protected final RegionServerStatusProtocol master', I agree there is something to be desired. Previously there was no need for HLogSplitter to know the actual status for any region in the WALs it splits. This feature adds this requirement. We can do buffering of calls to master. But I found that code would get more convoluted that way. w.r.t. introduction of void flushRegion(byte[] regionName), you're right that flushRegion(final RpcController controller, final FlushRegionRequest request) requires the composition of FlushRegionRequest. So that method is called by client. Will answer the remaining questions. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445647#comment-13445647 ] Ted Yu commented on HBASE-6659: --- bq. or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later. I would choose this route. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445633#comment-13445633 ] stack commented on HBASE-6659: -- There are a few approaches I think. We could hold up this patch till we figure alternate channel for metrics and attributes -- e.g. have the master listen to the metrics emitted by regionservers rather than have the regionserver blast load out on a metrics channel and duplicate their compostion putting same info out on the regionserver heartbeat -- or we commit the patch as is (after the non-architectural issues raised in the issue have been addressed) and address the messaging problems later. It looks like it could help a bunch with MTTR even though its totally ugly the way we are calling master up out of an HLog (how do you put that facility behind an Interface?) > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445599#comment-13445599 ] Ted Yu commented on HBASE-6659: --- @Stack: What should be the next step for this feature ? > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444810#comment-13444810 ] Hudson commented on HBASE-6659: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #154 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/154/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time, revert (Revision 1378752) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631) Result = FAILURE Tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444521#comment-13444521 ] Gregory Chanan commented on HBASE-6659: --- Just so I'm sure I understand -- lower footprint in ZK is better because ZK persists the data (absent ZOOKEEPER-866) or because of the overhead for consensus (I'm assuming the other costs, e.g. rpc time are equivalent in ZK vs HMaster, but maybe that's not correct)? Or both? > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444508#comment-13444508 ] Ted Yu commented on HBASE-6659: --- bq. or we could put it in a data structure apart from serverload Do we have a route through which such data structure can be channeled to master nowadays ? bq. out of metrics w/ master pulling the info when its interested in particular The use case for this feature is that log splitters may request sequence Ids in a short period of time. I am not sure if I understand the above comment enough in this context. I agree with Stack's analysis on using zookeeper. In a large cluster under heavy write load, sufficient experiment / profiling is needed to justify the approach. Again, the reason I came with the port was that I found this feature very helpful, even if HBASE-5699 is implemented. I think the patch didn't introduce fresh design deficiency. I can address Stack's other comments in the next patch. When there is good alternative to retiring heartbeat, all the payload can be moved in one shot. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1384#comment-1384 ] stack commented on HBASE-6659: -- I thought about putting it into zk. The good thing is that it'd be persisted so would make it across a master failover. In general the less footprint in zk, the better. If we went this route, it'd be a znode per region on the cluster. We'd write this region znode every time we flush. During high write times could flush a file per second or so If 100k regions on cluster... could be a bunch of zk writes. I suppose an rpc to the master to ask it about a long that its keeping in an in-memory map is way cheaper than going to fs, finding the region, and getting the sequenceid out of the hfile metadata. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1377#comment-1377 ] Gregory Chanan commented on HBASE-6659: --- On Stack's comments: * I agree it's silly to duplicate stats. I even noticed once the stats weren't quite duplicated (I think the metrics calculates requestsPerSecond differently than ServerLoad does), which is pointless and confusing * On this specific issue: should we put this info into ZK? it's "transient state" in the sense that we can recover fine if it gets blown away, and it has the added advantage that if both the Master and RS go down (but ZK stays up), we can still run this optimization. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1371#comment-1371 ] stack commented on HBASE-6659: -- Again, sorry w/ my late review. Thanks for reverting the patch for now. Yeah, seems odd adding currentsequenceid to RegionLoad? Its not part of region load, I don't think. Its more a region attribute (this would not be the first parameter that cannot be properly called a 'load'). High-level, we'd been trying to shrink the heartbeat with the hope that one day it'd be so small we could drown it in a bathtub. Maybe thats not possible. Outstanding is the fat serverload object updating the master on each heartbeat. I don't have specific issues to hand but we've talked about making it so this serverload could be obtained say via metrics/jmx since we are publishing the bulk of it there as well as over here up on the heartbeat; seems silly duplicating info. ServerLoad updating the master can be a bit lossy... same with this new sequenceid attribute. We could put it on the heartbeat -- either as is where serverload is a dumping ground of all we want to pass from regionserver to master or we could put it in a data structure apart from serverload -- and then maybe later do refactor of heartbeat to get its info elsewhere (from zk if unchanging post regionserver registration) and out of metrics w/ master pulling the info when its interested in particular? > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1360#comment-1360 ] Ted Yu commented on HBASE-6659: --- w.r.t. updateLastFlushedSequenceIds(), since regionServerReport() is in ServerManager, we need a way to let master know region <-> sequence Id mapping. I have reverted the patch. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1358#comment-1358
]
Ted Yu commented on HBASE-6659:
---
In hbase.proto, sequence Id is added to:
{code}
message RegionLoad {
{code}
ServerLoad.RegionLoad is just another form of representation for the above.
In regionServerReport(), we have:
{code}
HBaseProtos.ServerLoad sl = request.getLoad();
this.serverManager.regionServerReport(ProtobufUtil.toServerName(request.getServer()),
new ServerLoad(sl));
{code}
If the regionserver to master heartbeat is not the place to channel sequence Id
information, please remind me of the correct channel to use.
I can revert the patch to facilitate further discussion.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Zhihong Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1340#comment-1340
]
stack commented on HBASE-6659:
--
Sorry. Meant to review this patch before it went in.
I have a bit of problem w/ this patch in that it loads up new payloads on top
of a channel we've been trying to shut down, the regionserver to master
heartbeat.
Here are some other comments:
Are all the splitLogFile changes to add an extra null arg needed? Could do an
overload for case where arg is non-null? (with default null)
Do we not have the methods getRegionsByServer in HBaseTestingUtility already --
or at least in the hbase cluster object?
Could TestHLogFiltering tests not be added to the TestHLog? Seems related?
We have to add this new regionserver call getLastFlushedSequenceId even though
the regionservers are heartbeating every second (by default)?
This seems like an ugly entanglement:
+ // For checking the latest flushed sequence id
+ protected final RegionServerStatusProtocol master;
... a master reference down deep inside in HLogSplitter. Its needed?
We make a call to the master per region from down inside HLogSplitter to ask
for last known sequence id?
What is difference between:
public FlushRegionResponse flushRegion(final RpcController controller,
final FlushRegionRequest request) throws ServiceException {
and
+ public void flushRegion(byte[] regionName)
+ throws NotServingRegionException, IOException {
Can you not do what you want w/ the first method? Its pb'd? Not sure how you
get at the second (You can pass null to the first IIRC... at least the second
should call the first.
Do we have to add
+ void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl);
to MasterServices? I'd like it if we could keep MasterServices curt.
Everytime we add a new method, it makes for a bunch of changes in tests such as
the bleow and it also makes mocking that little bit tougher.
@Override
+public void updateLastFlushedSequenceIds(ServerName sn, ServerLoad hsl) {
+ // no-op
+}
Do we have to add a new map like this:
+ private final SortedMap flushedSequenceIdByRegion =
+new ConcurrentSkipListMap(Bytes.BYTES_COMPARATOR);
Isn't there elsewhere where we keep list of regions out on cluster. Could we
add the sequenceid as an attribute on that Map rather than introduce a new one?
Putting sequenceid into serverload doesn't seem right. ServerLoad already is a
dumping ground. Now we have pbs, could add as attribute easy enough.
> Port HBASE-6508 Filter out edits at log split time
> --
>
> Key: HBASE-6659
> URL: https://issues.apache.org/jira/browse/HBASE-6659
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>Assignee: Zhihong Ted Yu
> Fix For: 0.96.0
>
> Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt
>
>
> HBASE-6508 is for 0.89-fb branch.
> This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444218#comment-13444218 ] Hudson commented on HBASE-6659: --- Integrated in HBase-TRUNK #3291 (See [https://builds.apache.org/job/HBase-TRUNK/3291/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631) Result = FAILURE Tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444166#comment-13444166 ] Ted Yu commented on HBASE-6659: --- Integrated to trunk. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442784#comment-13442784 ] Hadoop QA commented on HBASE-6659: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542656/6508-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2708//console This message is automatically generated. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v4.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442487#comment-13442487 ] Hadoop QA commented on HBASE-6659: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542608/6508-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 92 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2700//console This message is automatically generated. > Port HBASE-6508 Filter out edits at log split time > -- > > Key: HBASE-6659 > URL: https://issues.apache.org/jira/browse/HBASE-6659 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu >Assignee: Zhihong Ted Yu > Fix For: 0.96.0 > > Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v4.txt > > > HBASE-6508 is for 0.89-fb branch. > This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
