[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560906#comment-13560906 ] Sergey Shelukhin commented on HBASE-7329: - Hmm... as I mentioned above actually I am not sure whether safety on close is necessary. This was just to preserve the logic of existing behavior. cacheFlushLock taken on close interacted with things as such - close would wait for flush; close would wait for log rolling; log rolling would wait for close and then exit because .closed is set; cache flush will wait for close and then proceed(?). Judging by lack of bugs from the later case, I am assuming it is deliberately, or by coincidence, ensured externally that it doesn't happen. With barrier, close would still wait for both operations. Both flush and log roll will not start if close has started. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560889#comment-13560889 ] ramkrishna.s.vasudevan commented on HBASE-7329: --- Had time check this patch today. Nice one. Covering all cases. One question, What was the motivation in introducing a barrier? What was the major problem wrt to close operation prior to this patch. Thanks Sergey. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559274#comment-13559274 ] Hudson commented on HBASE-7329: --- Integrated in HBase-TRUNK #3776 (See [https://builds.apache.org/job/HBase-TRUNK/3776/]) HBASE-7329 Remove duplicate code in newly added files (Revision 1436679) HBASE-7329 remove flush-related records from WAL and make locking more granular (Sergey) (Revision 1436678) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559250#comment-13559250 ] Hudson commented on HBASE-7329: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #365 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/365/]) HBASE-7329 Remove duplicate code in newly added files (Revision 1436679) HBASE-7329 remove flush-related records from WAL and make locking more granular (Sergey) (Revision 1436678) HBASE-7329 Revert due to compilation error (Revision 1436633) HBASE-7329 remove flush-related records from WAL and make locking more granular (Sergey) (Revision 1436632) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch,
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559205#comment-13559205 ] Hudson commented on HBASE-7329: --- Integrated in HBase-TRUNK #3775 (See [https://builds.apache.org/job/HBase-TRUNK/3775/]) HBASE-7329 Revert due to compilation error (Revision 1436633) HBASE-7329 remove flush-related records from WAL and make locking more granular (Sergey) (Revision 1436632) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/DrainBarrier.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestDrainBarrier.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559192#comment-13559192 ] Ted Yu commented on HBASE-7329: --- Integrated to trunk. Thanks for the patch, Sergey. Thanks for the review, Stack and Chunhui. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559183#comment-13559183
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565856/7329-v7.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4118//console
This message is automatically generated.
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch,
> HBASE-7329-v6.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559137#comment-13559137 ] Ted Yu commented on HBASE-7329: --- Patch v7 compiles based on latest trunk. TestHLog passes. Let Hadoop QA tell us the result. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, 7329-v7.txt, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559122#comment-13559122 ] Ted Yu commented on HBASE-7329: --- HBASE-7268 has been checked in. I checked in and then reverted patch v6 because of compilation error. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559104#comment-13559104 ] stack commented on HBASE-7329: -- [~sershe] Thanks for running tests. +1 on commit. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559067#comment-13559067
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565832/HBASE-7329-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4115//console
This message is automatically generated.
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch,
> HBASE-7329-v6.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559055#comment-13559055 ] Sergey Shelukhin commented on HBASE-7329: - Integration tests have run without problems... I can see a lot of rolling/flushing in the logs, although of course this is not a rigorous test for this patch in particular. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch, > HBASE-7329-v6.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559054#comment-13559054
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565825/HBASE-7329-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4113//console
This message is automatically generated.
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch, HBASE-7329-v5.patch, HBASE-7329-v6.patch,
> HBASE-7329-v6.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558984#comment-13558984 ] stack commented on HBASE-7329: -- bq. You mean rollWriter? I meant the fat log message we make under the log. Maybe this log could be down outside the lock? I think running integration tests a good idea. The spaghetti locking and state management that is in place around flush/close and log roll was hard won. Changing it around may bite in unexpected ways. Regards the internal comment, it talks about calling beginOp then you must call...I'm not sure. looks like two possibilities. I was thinking the class comment would include how you would use the class? This is a nit. Not important. Thanks for persisting w/ this one Sergey. Its a nice patch. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558973#comment-13558973 ] Sergey Shelukhin commented on HBASE-7329: - bq. Have you done much testing of this patch SS? I've run mvn tests; I will run some integration tests. bq. Nit: Below looks like it should be class comment rather than internal implementation comment or do you think otherwise? It's actually an implementation comment since it describes the implementation :) Class and method javadoc describe the external stuff. bq.Initially, the number of operations is 1. bq. Is it right having it at 1 when we construct the class? Should we wait on first beginOp call? Having 1 at start allows us to maintain a simple invariant that decrement to 0 is always the last. bq. It is not your fault but that is sure an ugly name on a method, getCompleteCacheFlushSequenceId. From its name you would not know what it is for. bq. In fact, if you want to remove it it looks like you could since 'TransactionalRegion' is a facility that no longer exists and going forward if you wanted to do this kinda thing, you'd do it via a coprocessor. Removed. bq. Is it right getting seqid before we advance memstore? We used to do it other way around. Fixed. bq. I wonder if the log of the roll should be outside of the lock...? Could be a bit sloppy but maybe it does not have to be too precise? You mean rollWriter? The only reason it's inside the lock is to prevent two concurrent rolls. It can be done differently e.g. we could lock around a boolean check and set, and bail duplicates. However current logic worked as having rollWriter-s queue up, not sure if anything relies on that. I will submit another patch, then run some integration tests on real cluster. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13558167#comment-13558167 ] stack commented on HBASE-7329: -- Patch looks great. I like the analysis that is done in the below citation. I think the below would work well as release note for this issue. https://issues.apache.org/jira/browse/HBASE-7329?focusedCommentId=13554658&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13554658 That'd be cool if you can close hbase-7011 too. From 7011, "You say the zombie RS 'should fail' when the dir is renamed under it. I haven't looked but if we done't already, sounds like we should add a test that proves it." Is there such a test in this issue or, do you know if we rename the directory before we start log splitting? Have you done much testing of this patch SS? Nit: Below looks like it should be class comment rather than internal implementation comment or do you think otherwise? + /** + * Contains the number of outstanding operations, as well as flags. + * Initially, the number of operations is 1. Each beginOp increments, and endOp decrements it. + * beginOp does not proceed when it sees the draining flag. When stop is called, it atomically + * decrements the number of operations (the initial 1) and sets the draining flag. If stop did + * the decrement to zero, that means there are no more operations outstanding, so stop is done. + * Otherwise, stop blocks, and the endOp that decrements the count to 0 unblocks it. + */ bq. Initially, the number of operations is 1. Is it right having it at 1 when we construct the class? Should we wait on first beginOp call? Nice test. It is not your fault but that is sure an ugly name on a method, getCompleteCacheFlushSequenceId. From its name you would not know what it is for. In fact, if you want to remove it it looks like you could since 'TransactionalRegion' is a facility that no longer exists and going forward if you wanted to do this kinda thing, you'd do it via a coprocessor. Is it right getting seqid before we advance memstore? We used to do it other way around. Thanks for doing this rename: - private final ConcurrentSkipListMap lastSeqWritten = + private final ConcurrentSkipListMap oldestUnflushedSeqNums = I wonder if the log of the roll should be outside of the lock...? Could be a bit sloppy but maybe it does not have to be too precise? Let me try and do another review before monday (but don't let this hold up commit I'd say...) > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557941#comment-13557941 ] Ted Yu commented on HBASE-7329: --- Will integrate Monday (21st) morning if there is no further review comment. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557888#comment-13557888 ] chunhui shen commented on HBASE-7329: - +1 on patch v5 > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557725#comment-13557725 ] Ted Yu commented on HBASE-7329: --- [~stack], [~zjushch]: Do you have further comments ? > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch, HBASE-7329-v5.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557715#comment-13557715
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565563/HBASE-7329-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4094//console
This message is automatically generated.
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch, HBASE-7329-v5.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557426#comment-13557426 ] Sergey Shelukhin commented on HBASE-7329: - [~zjushch] good point, I'll move it inside the lock. I was actually intending it for quick exit. [~ted_yu] yes; FindBugs in(?)correctly infers that synchronized method is protecting some particular field and then complains that it's not similarly protected elsewhere, from what I can see. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556969#comment-13556969
]
Ted Yu commented on HBASE-7329:
---
Was this part of code changed due to the following findbugs warning ?
{code}
{code}
Such warning appears in AuthenticationTokenSecretManager, etc.
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556959#comment-13556959 ] chunhui shen commented on HBASE-7329: - bq.It's the same as synchronized method It's not completely same. Suppose following scenario: Thread 1 block at synchronized (rollWriterLock) Thread 2 running rollWriter. Thread 1 will run rollWriter again after thread 2 complete it even if this.numEntries.get() <= 0. But it won't happen if synchronized method Correct me if wrong or no necessary Thanks > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556918#comment-13556918 ] Sergey Shelukhin commented on HBASE-7329: - Whoa, green precommit build! I thought they were a myth? :) > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556916#comment-13556916 ] Sergey Shelukhin commented on HBASE-7329: - No, this lock only protects concurrent execution of two rollWriter-s. It's the same as synchronized method, except that the latter produces bogus findbugs warning that will have to be suppressed on class level. > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556881#comment-13556881
]
chunhui shen commented on HBASE-7329:
-
Should we do the check {code} if (!force && this.writer != null &&
this.numEntries.get() <= 0) {{code}
again after
{code}
synchronized (rollWriterLock)
{code}
> remove flush-related records from WAL and make locking more granular
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and make locking more granular
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556854#comment-13556854 ] Ted Yu commented on HBASE-7329: --- [~stack]: Do you want to take another look ? > remove flush-related records from WAL and make locking more granular > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch, > HBASE-7329-v4.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556851#comment-13556851
]
Ted Yu commented on HBASE-7329:
---
Potentially we could have had a green PreCommit build. The failure was due to:
{code}
Caused by: java.io.IOException: Remote call on hadoop3 failed
at hudson.remoting.Channel.call(Channel.java:681)
at hudson.FilePath.act(FilePath.java:841)
... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space
{code}
+1 on patch v4.
> remove flush-related records from WAL and reduce locking
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556839#comment-13556839
]
Hadoop QA commented on HBASE-7329:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565376/HBASE-7329-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4081//console
This message is automatically generated.
> remove flush-related records from WAL and reduce locking
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 0.96.0
>
> Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch,
> HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch,
> HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch,
> HBASE-7329-v4.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556495#comment-13556495 ] Ted Yu commented on HBASE-7329: --- This was how I generated the findbugs diff: diff 4066-newPatchFindbugsWarningshbase-server.xml 4065-newPatchFindbugsWarningshbase-server.xml > 7329-findbugs.diff Warnings in HRegion and FSLog should be checked. > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: 7329-findbugs.diff, HBASE-7329-v0.patch, > HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, > HBASE-7329-v1.patch, HBASE-7329-v2.patch, HBASE-7329-v3.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555950#comment-13555950 ] chunhui shen commented on HBASE-7329: - +1 on pacth v3 if warning is ok > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch, HBASE-7329-v3.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555888#comment-13555888 ] Ted Yu commented on HBASE-7329: --- You can produce diff between findbugs output of PreCommit build #4065 and https://builds.apache.org/job/PreCommit-HBASE-Build/4066//testReport/ which passed findbugs check > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch, HBASE-7329-v3.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555877#comment-13555877 ] Ted Yu commented on HBASE-7329: --- Since the patch doesn't involve generated protobuf files, line length warning is legit. Then there is the findbugs warning. > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch, HBASE-7329-v3.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555808#comment-13555808
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565239/HBASE-7329-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4065//console
This message is automatically generated.
> remove flush-related records from WAL and reduce locking
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch,
> HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch,
> HBASE-7329-v2.patch, HBASE-7329-v3.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555451#comment-13555451 ] Sergey Shelukhin commented on HBASE-7329: - When I diff the findbugs warnings from patch files I don't get any real diffs (other then dates and timing). Is there any way to see where it gets the 6 warnings? > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554877#comment-13554877 ] chunhui shen commented on HBASE-7329: - Agree with removing that flush-related log. PatchV2 is good after a review > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554721#comment-13554721
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565062/HBASE-7329-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 6 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4038//console
This message is automatically generated.
> remove flush-related records from WAL and reduce locking
>
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch,
> HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch,
> HBASE-7329-v2.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL and reduce locking
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554665#comment-13554665 ] Sergey Shelukhin commented on HBASE-7329: - /r/ at https://reviews.apache.org/r/8967/ > remove flush-related records from WAL and reduce locking > > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch, > HBASE-7329-v2.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551390#comment-13551390 ] Sergey Shelukhin commented on HBASE-7329: - It looks like the patch doesn't contain the test for new class, which in all likelihood is nuked because I never committed it to local git :( I am going to rebase the patch, recover or rewrite the test, write up some explanation and post it on /r/, for now please continue disregarding this JIRA :) > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546557#comment-13546557
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563668/HBASE-7329-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHTableMultiplexer
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3922//console
This message is automatically generated.
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch,
> HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546508#comment-13546508 ] Sergey Shelukhin commented on HBASE-7329: - rebased wrong patch, and then the correct patch (v1) > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0.patch, > HBASE-7329-v0-tmp.patch, HBASE-7329-v1.patch, HBASE-7329-v1.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546440#comment-13546440 ] Sergey Shelukhin commented on HBASE-7329: - Hi. Ping? :) > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch, > HBASE-7329-v1.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538657#comment-13538657
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562204/HBASE-7329-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 21 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3663//console
This message is automatically generated.
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch,
> HBASE-7329-v1.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538601#comment-13538601
]
Sergey Shelukhin commented on HBASE-7329:
-
The race condition is actually the same as the one already fixed for
writer.append - we get writer w/o updateLock, so it s possible that it will be
invalid by the time we used it. I added the same handling in the writer as
already done in append (catch NPE).
bq. On the patch, Sergey, are you convinced we can't make use of these edits?
You think the right thing to do is remove them rather than try and make use of
this entry at WAL log splitting time? It is the sequenceid of the last file
flushed so we should replay all edits with a sequenceid that is great that this
one. At WAL log splitting time, I'm not sure how we'd find the last one of
these edits in a WAL without first reading all WALs... and if reading the WAL
anyways, might as well replay its edits? Or is that true? We could save a bunch
not deserializing edits we will discard anyways?
bq. Or, we don't need this edit because recently a mechanism was added where
the regionserver tells the master what its most recently flushed edit was and
master knows by other means what is the last flushed edit so this special edit
it WALs is now redundant?
Yeah, that's HBASE-6508.
bq. We need to synchronize rolling now? Only one thread hereabouts anyways?
Hmm... is this guaranteed?
bq. Is this because you want to narrow the lock...to the body of cleanOldLogs?
The lock was actually used for two things - to avoid rolling log during flush,
which is no longer needed, and to safely operate on the map with oldest seqnums.
I narrowed the lock to the latter.
The lock was also taken in close(), for whatever reason. I will add a barrier
to block flushes and rolls after close is called.
bq. You mean to leave this in the patch
bq. + LOG.info("TMP: rollWriter has nothing to do");
No, will remove that.
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538569#comment-13538569
]
stack commented on HBASE-7329:
--
On the patch, Sergey, are you convinced we can't make use of these edits? You
think the right thing to do is remove them rather than try and make use of this
entry at WAL log splitting time? It is the sequenceid of the last file flushed
so we should replay all edits with a sequenceid that is great that this one.
At WAL log splitting time, I'm not sure how we'd find the last one of these
edits in a WAL without first reading all WALs... and if reading the WAL
anyways, might as well replay its edits? Or is that true? We could save a
bunch not deserializing edits we will discard anyways?
Or, we don't need this edit because recently a mechanism was added where the
regionserver tells the master what its most recently flushed edit was and
master knows by other means what is the last flushed edit so this special edit
it WALs is now redundant?
We need to synchronize rolling now? Only one thread hereabouts anyways?
You remove this
-this.cacheFlushLock.lock();
Is this because you want to narrow the lock...to the body of cleanOldLogs?
Will be back later. This is a refactoring of a critical bit of code. Need to
spend some time looking.
Good on you Sergey.
You mean to leave this in the patch
+ LOG.info("TMP: rollWriter has nothing to do");
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538561#comment-13538561 ] stack commented on HBASE-7329: -- Reapply the patch should get your patch rerun by hadoopqa. > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538475#comment-13538475 ] Sergey Shelukhin commented on HBASE-7329: - well, this is a real race condition... let me check > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538388#comment-13538388
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561965/HBASE-7329-v0-tmp.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 18 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3657//console
This message is automatically generated.
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538335#comment-13538335 ] Sergey Shelukhin commented on HBASE-7329: - hmm... how do I nudge Hudson properly? Does anyone know? > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch, HBASE-7329-v0-tmp.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537263#comment-13537263 ] Sergey Shelukhin commented on HBASE-7329: - TestDelayedRpc is flaky. TestLogRolling failure looks like it could be caused by this patch but I cannot reproduce it. Looking... > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7329-v0.patch > > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[
https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536765#comment-13536765
]
Hadoop QA commented on HBASE-7329:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12561835/HBASE-7329-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 27 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
org.apache.hadoop.hbase.ipc.TestDelayedRpc
{color:red}-1 core zombie tests{color}. There are zombie tests. See build
logs for details.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3629//console
This message is automatically generated.
> remove flush-related records from WAL
> -
>
> Key: HBASE-7329
> URL: https://issues.apache.org/jira/browse/HBASE-7329
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HBASE-7329-v0.patch
>
>
> Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush
> records in WAL are not useful. If so, they should be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13536305#comment-13536305 ] Sergey Shelukhin commented on HBASE-7329: - ok, have to read a lot of code to remove a lot of code, so the patch may be slower in the making :) > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7329) remove flush-related records from WAL
[ https://issues.apache.org/jira/browse/HBASE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13534610#comment-13534610 ] Sergey Shelukhin commented on HBASE-7329: - Will have a patch tomorrow/later this week unless something intervenes. A lot can be removed, maybe even the lock itself. > remove flush-related records from WAL > - > > Key: HBASE-7329 > URL: https://issues.apache.org/jira/browse/HBASE-7329 > Project: HBase > Issue Type: Improvement > Components: wal >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Comments from many people in HBASE-6466 and HBASE-6980 indicate that flush > records in WAL are not useful. If so, they should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
