[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490390#comment-13490390 ] Hudson commented on HBASE-5867: --- Integrated in HBase-0.94-security-on-Hadoop-23 #9 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/9/]) HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Fix For: 0.96.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D2943.1.patch, > HBASE-5867-trunk.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484347#comment-13484347 ] Hudson commented on HBASE-5867: --- Integrated in HBase-0.94 #554 (See [https://builds.apache.org/job/HBase-0.94/554/]) HBASE-7040 Port HBASE-5867 Improve Compaction Throttle Default to 0.94 (Sergey Shelukhin) (Revision 1402215) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Fix For: 0.96.0 > > Attachments: ASF.LICENSE.NOT.GRANTED--D2943.1.patch, > HBASE-5867-trunk.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276347#comment-13276347 ] Hudson commented on HBASE-5867: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #5 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/5/]) [HBASE-5867] Improve Compaction Throttle Default Summary: We recently had a production issue where our compactions fell behind because our compaction throttle was improperly tuned and accidentally upgraded all compactions to the large pool. The default from HBASE-3877 makes 1 bad assumption: the default number of flushed files in a compaction. MinFilesToCompact should be taken into consideration. As a default, it is less damaging for the large thread to be slightly higher than it needs to be and only get timed-majors versus having everything accidentally promoted. Test Plan: - mvn test Reviewers: JIRA, Kannan, Liyin Reviewed By: Kannan CC: stack Differential Revision: https://reviews.facebook.net/D2943 (Revision 1338809) Result = FAILURE nspiegelberg : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Fix For: 0.96.0 > > Attachments: D2943.1.patch, HBASE-5867-trunk.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268935#comment-13268935 ] Phabricator commented on HBASE-5867: mbautin has committed the revision "[jira] [HBASE-5867] [89-fb] Improve Compaction Throttle Default". REVISION DETAIL https://reviews.facebook.net/D2943 COMMIT https://reviews.facebook.net/rHBASEEIGHTNINEFBBRANCH1334388 > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Attachments: D2943.1.patch, HBASE-5867-trunk.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268522#comment-13268522 ] Hadoop QA commented on HBASE-5867: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525641/HBASE-5867-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1764//console This message is automatically generated. > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Attachments: D2943.1.patch, HBASE-5867-trunk.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13268458#comment-13268458 ] Nicolas Spiegelberg commented on HBASE-5867: @stack: this should be a much better out-of-the-box than we currently have. If anything, we could make the value higher (4 or 8 instead of 2). However, this should be a substantially-better default than the current. > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Attachments: D2943.1.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264123#comment-13264123 ] Phabricator commented on HBASE-5867: stack has commented on the revision "[jira] [HBASE-5867] [89-fb] Improve Compaction Throttle Default". Should we pull this into trunk Nicolas? Would it serve as good default for out-of-the-box hbase? REVISION DETAIL https://reviews.facebook.net/D2943 BRANCH (no branch) > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Attachments: D2943.1.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[ https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261966#comment-13261966 ] Phabricator commented on HBASE-5867: Kannan has accepted the revision "[jira] [HBASE-5867] [89-fb] Improve Compaction Throttle Default". looks good! REVISION DETAIL https://reviews.facebook.net/D2943 BRANCH (no branch) > Improve Compaction Throttle Default > --- > > Key: HBASE-5867 > URL: https://issues.apache.org/jira/browse/HBASE-5867 > Project: HBase > Issue Type: Improvement >Reporter: Nicolas Spiegelberg >Assignee: Nicolas Spiegelberg >Priority: Minor > Attachments: D2943.1.patch > > > We recently had a production issue where our compactions fell behind because > our compaction throttle was improperly tuned and accidentally upgraded all > compactions to the large pool. The default from HBASE-3877 makes 1 bad > assumption: the default number of flushed files in a compaction. Currently > the algorithm is: > throttleSize ~= flushSize * 2 > This assumes that the basic compaction utilizes 3 files and that all 3 files > are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the > values were not very compressible. Both conditions should be taken into > consideration. As a default, it is less damaging for the large thread to be > slightly higher than it needs to be versus having everything accidentally > promoted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5867) Improve Compaction Throttle Default
[
https://issues.apache.org/jira/browse/HBASE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261066#comment-13261066
]
Nicolas Spiegelberg commented on HBASE-5867:
The most common type of compaction is compacting only flushed files. Assuming
that there is no compression (the default), then the common compaction size
should be:
{code}
minFiles * flushSize
{code}
The current idea is to support this operation and supporting a compaction with
1 previously-compacted file. Assuming no overlap, this size would be:
(minFiles-1) * flushSize + minFiles * flushSize ==>
{code}
2 * minFiles * flushSize - ε
{code}
> Improve Compaction Throttle Default
> ---
>
> Key: HBASE-5867
> URL: https://issues.apache.org/jira/browse/HBASE-5867
> Project: HBase
> Issue Type: Improvement
>Reporter: Nicolas Spiegelberg
>Assignee: Nicolas Spiegelberg
>Priority: Minor
>
> We recently had a production issue where our compactions fell behind because
> our compaction throttle was improperly tuned and accidentally upgraded all
> compactions to the large pool. The default from HBASE-3877 makes 1 bad
> assumption: the default number of flushed files in a compaction. Currently
> the algorithm is:
> throttleSize ~= flushSize * 2
> This assumes that the basic compaction utilizes 3 files and that all 3 files
> are compressed. In this case, "hbase.hstore.compaction.min" == 6 && the
> values were not very compressible. Both conditions should be taken into
> consideration. As a default, it is less damaging for the large thread to be
> slightly higher than it needs to be versus having everything accidentally
> promoted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
