[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14078882#comment-14078882 ] Hudson commented on HBASE-11120: FAILURE: Integrated in HBase-TRUNK #5353 (See [https://builds.apache.org/job/HBase-TRUNK/5353/]) HBASE-11316 Expand info about compactions beyond HBASE-11120 ADDENDUM TO FIX BROKE TESTS (stack: rev 85853f69c88c08db61af7fb6ef3b5432f926e531) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch, hbase-11120.addendum.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14077338#comment-14077338 ] Hudson commented on HBASE-11120: FAILURE: Integrated in HBase-TRUNK #5349 (See [https://builds.apache.org/job/HBase-TRUNK/5349/]) HBASE-11316 Expand info about compactions beyond HBASE-11120 (Misty Stanley-Jones) (jdcryans: rev afae1e2583b8b32d828efbadce363247c784a93e) * hbase-common/src/main/resources/hbase-default.xml * src/main/docbkx/book.xml Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch, hbase-11120.addendum.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026649#comment-14026649 ] Jonathan Hsieh commented on HBASE-11120: Patch seems to have broken trunk. Posting and committing trivial addendum to fix shortly. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026661#comment-14026661 ] Jonathan Hsieh commented on HBASE-11120: jimmy seems to have beaten me to the commit. :) thanks! Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Jonathan Hsieh Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch, hbase-11120.addendum.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026681#comment-14026681 ] stack commented on HBASE-11120: --- Thanks for fix up [~jxiang] and [~jmhsieh]. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch, hbase-11120.addendum.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026876#comment-14026876 ] Hudson commented on HBASE-11120: SUCCESS: Integrated in HBase-TRUNK #5191 (See [https://builds.apache.org/job/HBase-TRUNK/5191/]) HBASE-11120 Addendum for fixing compilation error (jxiang: rev ee762257835a060e9e01244fb8b0dc97682af098) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreConfig.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch, hbase-11120.addendum.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025292#comment-14025292 ] Jean-Daniel Cryans commented on HBASE-11120: bq. I don't see hbase.hstore.compaction.min.size in code base any more Yeah I was confused by this too, [~stack], until I found this thread that you started for this reason a while back: http://osdir.com/ml/general/2013-06/msg41068.html Basically, {{CompactionConfiguration}} was written with a configuration prefix so doing a grep on that config (or hbase.hstore.compaction.max.size) won't return anything. I think we should undo the refactoring in that class since it obviously breaks a pattern we're used to, which is to completely spell out the configuration names so we can easily search for them. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025525#comment-14025525 ] stack commented on HBASE-11120: --- [~jdcryans] Thanks for the digging. Agree. Let me change it. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025561#comment-14025561 ] stack commented on HBASE-11120: --- So now [~misty], why did the file size shrink so much? (I can commit my addendum along w/ your doc patch when done). Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025815#comment-14025815 ] Misty Stanley-Jones commented on HBASE-11120: - I don't know why it shrunk. At the time I was very confused and it seemed as though maybe somehow the HBASE-11120-3-rebased.patch got applied to master. Let me try to replay it and make sure that everything is really there so I don't lose any work. I will let you know. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025890#comment-14025890 ] Misty Stanley-Jones commented on HBASE-11120: - I am figuring that out by doing 'git annotate' and searching for the changes. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025888#comment-14025888 ] Misty Stanley-Jones commented on HBASE-11120: - OK it shrunk because somehow a lot of these changes got applied to master by your commit on HBASE-11199, [~stack]. Can you please confirm? Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026117#comment-14026117 ] Hudson commented on HBASE-11120: FAILURE: Integrated in HBase-TRUNK #5190 (See [https://builds.apache.org/job/HBase-TRUNK/5190/]) HBASE-11120 Update documentation about major compaction algorithm (Misty Stanley-Jones) + addendum fixing CompactionConfiguration (stack: rev 948b22f082e1c598fc553a34772226de31b8b0b7) * src/main/docbkx/book.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: 11120_addendum.txt, HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14021081#comment-14021081 ] stack commented on HBASE-11120: --- Why did the patch go from 21k to 6k [~misty]? Did you get less ambitious? Looking the below: {code} entryhbase.hstore.compaction.min.size/entry entryA StoreFile smaller than this size (in bytes) will always be eligible for minor compaction./entry -entry10/entry +entry128 MB/entry /row row entryhbase.hstore.compaction.max.size/entry entryA StoreFile larger than this size (in bytes) will be excluded from minor compaction./entry -entry1000/entry +entryLong.MAX_VALUE/entry {code} ... I don't see hbase.hstore.compaction.min.size in code base any more and you can't write Long.MAX_VALUE or 128 MB as values... our configuration system can't digest anything but numbers. Let me know why big difference in size. Maybe this patch should have had more in it? Thanks. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019516#comment-14019516 ] Hadoop QA commented on HBASE-11120: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648582/HBASE-11120-4-rebased.patch against trunk revision . ATTACHMENT ID: 12648582 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + xlink:href=https://issues.apache.org/jira/browse/HBASE-7842;ExploringCompaction Policy/link/title +termIf this compaction was user-requested, perform the requested type of compaction./term +compaction is requested manually, HBase always runs that type of compaction. If the +user requests a major compaction, the major compaction still runs even if the are {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9700//console This message is automatically generated. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4-rebased.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012875#comment-14012875 ] Jean-Daniel Cryans commented on HBASE-11120: Spent some more time reading through and I saw that I had missed a few things. BTW, thanks again Misty for working on this, it is much needed. I'm not sure about this part: {quote} However, if the normal compaction +algorithm do not find any normally-eligible StoreFiles, a major compaction is +the only way to get out of this situation, and is forced. This is also called a +size-based or size-triggered major compaction. {quote} The exploration compaction will select the smallest set of files to compact and it won't trigger a major compaction. Also, I looked for size-based and size-triggered in the code but couldn't find anything so I'm not sure what this is referring to. This is wrong: {quote} +termIf this compaction was user-requested, do a major compaction./term +listitem + paraCompactions can run on a schedule or can be initiated manually. If a +compaction is requested manually, it is always a major compaction. If the +compaction is user-requested, the major compaction still happens even if the are +more than varnamehbase.hstore.compaction.max/varname files that need +compaction./para +/listitem {quote} It will do a minor compaction if the user asks for that, or a major compaction if it's specifically asked for. It might be good to mention that hbase.hstore.compactionThreshold was the old name for hbase.hstore.compaction.min in this section: {quote} + row +entryhbase.hstore.compaction.min/entry +entryThe minimum number of files which must be eligible for compaction before + compaction can run./entry +entry3/entry + /row {quote} Those two are wrong (if 1000 bytes was the max we'd probably never compact :P): {quote} + row +entryhbase.hstore.compaction.min.size/entry +entryA StoreFile smaller than this size (in bytes) will always be eligible for + minor compaction./entry +entry10/entry + /row + row +entryhbase.hstore.compaction.max.size/entry +entryA StoreFile larger than this size (in bytes) will be excluded from minor + compaction./entry +entry1000/entry + /row {quote} It should be 128MB and Long.MAX_VALUE as per: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java#L71 Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14013249#comment-14013249 ] Hadoop QA commented on HBASE-11120: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647508/HBASE-11120-4.patch against trunk revision . ATTACHMENT ID: 12647508 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + xlink:href=https://issues.apache.org/jira/browse/HBASE-7842;ExploringCompaction Policy/link/title +termIf this compaction was user-requested, perform the requested type of compaction./term +compaction is requested manually, HBase always runs that type of compaction. If the +user requests a major compaction, the major compaction still runs even if the are {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9642//console This message is automatically generated. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120-4.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010754#comment-14010754 ] Hadoop QA commented on HBASE-11120: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647031/HBASE-11120-3-rebased.patch against trunk revision . ATTACHMENT ID: 12647031 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:red}-1 release audit{color}. The applied patch generated 32 release audit warnings (more than the trunk's current 0 warnings). {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +Obviously, once a major compaction has run, such behavior will not be the case anymore...footnote +linkend=major.compactions.change.query.results / for a more in-depth explanation. This + xlink:href=https://issues.apache.org/jira/browse/HBASE-7842;Exploring Compaction Policy/link/title + filenamehbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java/filename./para {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9613//console This message is automatically generated. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120-2.patch, HBASE-11120-3-rebased.patch, HBASE-11120-3.patch, HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006215#comment-14006215 ] Sergey Shelukhin commented on HBASE-11120: -- This may need correcting? {noformat} +entry / +entry / +entry1.2F/entry + /row {noformat} {noformat} +entryhbase.hstore.compaction.max.size/entry +entryThe maximum size for a file to be eligible for compaction, expressed in + bytes./entry +entry1000/entry {noformat} This is not a very good description imho... It would be good to insert link to effect of major compaction on read results into the section about deletes. There's description somewhere in the book already. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006480#comment-14006480 ] Jean-Daniel Cryans commented on HBASE-11120: On top of Sergey's comments: This could also use a sentence regarding TTLs, they are simply filtered out and tombstones aren't created: {noformat} bq. titleCompaction and Deletions/title {noformat} Same kind of comment for this para, the old versions are filtered more than deleted: {noformat} +titleCompaction and Versions/title {noformat} One thing bothers me about this para: {noformat} +paraThe compaction algorithms used by HBase have evolved over time. HBase 0.96 + introduced a new algorithm for compaction file selection. To find out about the old + algorithm, see xref +linkend=compaction /. The rest of this section describes the new algorithm, which + was implemented in link + xlink:href=https://issues.apache.org/jira/browse/HBASE-7842;HBASE-7842/link. {noformat} What's written next is much more about how to pick the files that will then be considered for compaction than the actual new policy which is in ExploringCompactionPolicy.java. In other words, the text that follows what I just quoted is ok but not really new in 0.96 via HBASE-7842, and it seems to me that we should explain what HBASE-7842 does. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14006714#comment-14006714 ] Misty Stanley-Jones commented on HBASE-11120: - bq. It would be good to insert link to effect of major compaction on read results into the section about deletes. There's description somewhere in the book already. Do you mean 5.9.2.2. Major compactions change query results? Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005462#comment-14005462 ] Misty Stanley-Jones commented on HBASE-11120: - Integrating review comments from [~jdcryans], cancelling patch review in the meantime. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005472#comment-14005472 ] Misty Stanley-Jones commented on HBASE-11120: - Check to see if we absolutely have to compact in case we have too many store files then the next para: +termIf this compaction was user-requested, try to do a major + compaction./term see the link I posted above, we only major compact when the user asks for it I think the code you saw was more like forcing the major compaction if the user was indeed requesting it Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005471#comment-14005471 ] Misty Stanley-Jones commented on HBASE-11120: - a store can only have so many files, and that's called the blocking store file count so you cannot flush data if you have too many files, you have to wait for a compaciton and you cannot insert if the memstore is over some threshold but then the compaction selection might not select a single file if none of them are good so if you get there, you're stuck that's the accounting here: boolean mayBeStuck = (candidateFiles.size() - filesCompacting.size() + futureFiles) = storeConfigInfo.getBlockingFileCount(); Jean-Daniel Cryans 10:29 am if our files we're supposed to compact, minus the ones we're already compacting, plus 1 since those files will end up being a new store file, is greater or equal to the blocking file count then we force a compacting compaction Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)