[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078881#comment-14078881 ] Hudson commented on HBASE-11316: FAILURE: Integrated in HBase-TRUNK #5353 (See [https://builds.apache.org/job/HBase-TRUNK/5353/]) HBASE-11316 Expand info about compactions beyond HBASE-11120 ADDENDUM TO FIX BROKE TESTS (stack: rev 85853f69c88c08db61af7fb6ef3b5432f926e531) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactionPolicy.java > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316-9.patch, HBASE-11316.patch, addendum.txt, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078779#comment-14078779 ] stack commented on HBASE-11316: --- Pushed addendum. [~jdcryans] now I only owe you $2.1455M dollars. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316-9.patch, HBASE-11316.patch, addendum.txt, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078104#comment-14078104 ] Jean-Daniel Cryans commented on HBASE-11316: +1, thanks for looking at this Stack and sorry I overlooked those failing tests. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316-9.patch, HBASE-11316.patch, addendum.txt, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078013#comment-14078013 ] stack commented on HBASE-11316: --- Interesting. The test failures are legit: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection The hbase-default additions seem to break above tests. Let me isolate which changes are responsible for the break. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316-9.patch, HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077336#comment-14077336 ] Hudson commented on HBASE-11316: FAILURE: Integrated in HBase-TRUNK #5349 (See [https://builds.apache.org/job/HBase-TRUNK/5349/]) HBASE-11316 Expand info about compactions beyond HBASE-11120 (Misty Stanley-Jones) (jdcryans: rev afae1e2583b8b32d828efbadce363247c784a93e) * src/main/docbkx/book.xml * hbase-common/src/main/resources/hbase-default.xml > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316-9.patch, HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077225#comment-14077225 ] Jean-Daniel Cryans commented on HBASE-11316: Comments: {noformat} -Name of the failsafe snapshot taken by the restore operation. +Name of the failsafe snapshot taken by the restore opecompn. {noformat} Typo slipped in there. {noformat} + Being Stuck {noformat} Looking at the output, this line (and the other titles at that level) is comically small. There's probably we can do short of changing the style sheet. And I'm good with the rest. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075773#comment-14075773 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658042/HBASE-11316-8.patch against trunk revision . ATTACHMENT ID: 12658042 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. +HBase since version 1.0 and ask that a column family have both its meta and DATA blocks hosted onheap in the L1 tier by +A StoreFile is a facade of HFile. In terms of compaction, use of + xlink:href="https://issues.apache.org/jira/browse/HBASE-11316";>HBASE-11316. +drop (filter out) deletes or expired versions, because of potential side effects. See hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + will compact larger StoreFiles. However, during reads, HBase will need to seek + through fewer StpreFo;es to accomplish the read. Consider this approach if you + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10197//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316-7.patch, HBASE-11316-8.patch, > HBASE-1
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065941#comment-14065941 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656417/HBASE-11316-7.patch against trunk revision . ATTACHMENT ID: 12656417 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. +A StoreFile is a facade of HFile. In terms of compaction, use of + xlink:href="https://issues.apache.org/jira/browse/HBASE-11316";>HBASE-11316. +drop (filter out) deletes or expired versions, because of potential side effects. See hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + will compact larger StoreFiles. However, during reads, HBase will need to seek + through fewer StpreFo;es to accomplish the read. Consider this approach if you + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10115//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065900#comment-14065900 ] Misty Stanley-Jones commented on HBASE-11316: - The last patch broke everything, don't use it. Stand by for a fix. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased.patch, > HBASE-11316-6.patch, HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065864#comment-14065864 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12656404/HBASE-11316-6-rebased-v2.patch against trunk revision . ATTACHMENT ID: 12656404 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. +A StoreFile is a facade of HFile. In terms of compaction, use of + xlink:href="https://issues.apache.org/jira/browse/HBASE-11316";>HBASE-11316. +drop (filter out) deletes or expired versions, because of potential side effects. See hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + will compact larger StoreFiles. However, during reads, HBase will need to seek + through fewer StpreFo;es to accomplish the read. Consider this approach if you + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} {color:red}-1 site{color}. The patch appears to cause mvn site goal to fail. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10113//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased-v2.patch, HBASE-11316-6-rebased
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057026#comment-14057026 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654913/HBASE-11316-6-rebased.patch against trunk revision . ATTACHMENT ID: 12654913 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10008//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6-rebased.patch, HBASE-11316-6.patch, HBASE-11316.patch, > ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049661#comment-14049661 ] stack commented on HBASE-11316: --- bq. Well, it seems like you and JD are in disagreement on this so I am unsure how to proceed. Sorry for confusion caused. Go storefile and store. JD and Esteban have a point especially given that is how we talk up files in UI (metrics also talk storefile). Suggest talking up how relates to hfile and column family since these are notions that are all over the hbase shop too. Thanks Misty. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6.patch, HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049623#comment-14049623 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653533/HBASE-11316-6.patch against trunk revision . ATTACHMENT ID: 12653533 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. +A StoreFile is a facade of HFile. In terms of compaction, use of + xlink:href="https://issues.apache.org/jira/browse/HBASE-11316";>HBASE-11316. +drop (filter out) deletes or expired versions, because of potential side effects. See hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + will compact larger StoreFiles. However, during reads, HBase will need to seek + through fewer StpreFo;es to accomplish the read. Consider this approach if you + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9936//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316-6.patch, HBASE-113
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049588#comment-14049588 ] Misty Stanley-Jones commented on HBASE-11316: - Well, it seems like you and JD are in disagreement on this so I am unsure how to proceed. :) > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049574#comment-14049574 ] stack commented on HBASE-11316: --- Misty you have enough to go on? Just ask if not. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049476#comment-14049476 ] Jean-Daniel Cryans commented on HBASE-11316: IMO, unless we specifically refer to something related to the HFile implementation, we should always use StoreFile. That's also how we present in the web UI when list the number of Stores and StoreFiles in a region. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049410#comment-14049410 ] stack commented on HBASE-11316: --- In the code, StoreFile is a facade over our file implementation HFile. Store is how we implement the modeling concept column family. We should have named Stores 'ColumnFamily' rather than Store. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049405#comment-14049405 ] Misty Stanley-Jones commented on HBASE-11316: - OK, then how do they relate? Especially Store / Column Family. It seems really strange to me to talk about an HFile being "in" a Column Family. Can you help me out? > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049402#comment-14049402 ] stack commented on HBASE-11316: --- Folks seem to update hfile rather than put in place a new version (it is on version 3 at the moment). hfile seems to be known outside hbase (we bulk import hfiles, not storefiles). hfile is used more often in our doc currently: kalashnikov:docbkx stack$ grep hfile *|wc 97 980 11524 kalashnikov:docbkx stack$ grep -i hfile * |wc 2524413 35218 kalashnikov:docbkx stack$ grep -i storefile * |wc 1882386 22833 If you just use storefile, folks will wonder where hfile fits. You are the boss. Pick one. Just explain how the two relate on first use. Good on you misty. Ditto with column family/Store. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049354#comment-14049354 ] Misty Stanley-Jones commented on HBASE-11316: - So, [~jdcryans], [~esteban], and I talked about this yesterday and he seemed to have the opinion that we should be using StoreFile and Store, since we could support other types of StoreFiles besides HFile later on. Based on that, I changed it all back. Now I will hold off on further changes until I get more consistent advice about how I Should be referring to these things. :) > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049114#comment-14049114 ] stack commented on HBASE-11316: --- bq. Anyway I played around with this and tried to be consistent in using HFile instead of StoreFile, Column Family instead of Store. This is the way to go IMO. Looking at PDF, you have this in spite of the above: "it flushes its contents to a StoreFile. The number of StoreFiles in a Store increases over time. " What you thinking given the above? > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch, ch9_compactions.pdf > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048555#comment-14048555 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653316/HBASE-11316-5.patch against trunk revision . ATTACHMENT ID: 12653316 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. +drop (filter out) deletes or expired versions, because of potential side effects. See hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + will compact larger StoreFiles. However, during reads, HBase will need to seek + through fewer StpreFo;es to accomplish the read. Consider this approach if you + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} +create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9912//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch,
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048503#comment-14048503 ] Misty Stanley-Jones commented on HBASE-11316: - Ready for review (again). > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316-5.patch, > HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048496#comment-14048496 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12653289/HBASE-11316-4.patch against trunk revision . ATTACHMENT ID: 12653289 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + The time for which a region will block updates after reaching the StoreFile (HFile) + can try raising this value. Default: the value of LONG.MAX_VALUE, expressed in bytes. + hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. +compaction. This may be because not all HFiles in the Column Family are available to +less than the sum of the sizes of the other HFiles in the set. +HFile. Its function is to remove deletes and expired versions, and reset locality +HFile to be compacted, a major compaction is run, even if it would otherwise have + compaction. Its effect is to limit compaction of large HFiles. The value of + xlink:href="http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/Store.html#836";>Store +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9908//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: 9.7.pdf, HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > ta
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048483#comment-14048483 ] Misty Stanley-Jones commented on HBASE-11316: - Got clarification from JD that it is most appropriate to talk about StoreFiles and Stores, not HFiles and ColumnFamilies. I'll update that patch to reflect that. Sorry about the churn. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: 9.7.pdf, HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316-4.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14048371#comment-14048371 ] Misty Stanley-Jones commented on HBASE-11316: - Working on this today. Do you prefer HFile or StoreFile? I didn't actually realize they were the same thing when I was originally writing this. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: 9.7.pdf, HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044939#comment-14044939 ] stack commented on HBASE-11316: --- I see 'agiven' in first description section in the hbase-default.xml. I started reviewing and it looks really good -- thanks in particular for working on these hbase-default.xml descriptions -- but I kept tripping over your use of HStore vs Store vs StoreFile vs HStoreFile. I started to edit and was adding in after StoreFile (HFile) in commas because that is how we usually refer to these files and Column Family for Store since they equate but I gave up thinking I should be punting this to you since you are probably trying to have one voice across the doc and one set of terminology you'd like to apply. How would you like to deal with HStoreFile vs StoreFile vs HFile? And Store vs HStore vs Column Family? Good on you Misty. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: 9.7.pdf, HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040307#comment-14040307 ] Misty Stanley-Jones commented on HBASE-11316: - BTW, noticed and fixed a problem (empty parentheses in the first line of the section) but won't regenerate the PDF for that. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: 9.7.pdf, HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040303#comment-14040303 ] Misty Stanley-Jones commented on HBASE-11316: - I can't find either of your typos and am a bit worried you might be looking at an old output. Let me generate a PDF and attach. I don't know about Review Board -- who should I ask? > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038796#comment-14038796 ] Lars George commented on HBASE-11316: - Typo in "agiven" (missing space) and "milliseconds.." (two dots). I need to read the rest still, but that will take a little longer. Maybe this is larger enough to stick it on review board - or else it is getting messy feedback wise. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032068#comment-14032068 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650499/HBASE-11316-3.patch against trunk revision . ATTACHMENT ID: 12650499 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +is a minor or major compaction) is called the compaction policy. + hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + major compactions. This approach alleviates some of the challenges of larger regions. +Stripe compaction is fully compatible with and works in + conjunction with either the ExploringCompactionPolicy or RatioBasedCompactionPolicy. It can be enabled for + You can enable stripe compaction for a table or a column family, by setting its hbase.hstore.engine.class to +alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} +create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} +hbase.store.stripe.compaction.flushToL0 to true. This will {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.compactions.TestStripeCompactionPolicy org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9776//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316-3.patch, HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe com
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14031321#comment-14031321 ] stack commented on HBASE-11316: --- Ugh. This no longer applies. Please rebase [~misty] I took a read through. It is very good. Minor is that a minor compaction could end up with one file when done (you say it can't) and you never edit hbase-default.xml. You add your configs to hbase-site.xml. The writeups you have on each of the compaction configs, should these go into the actual hbase-default.xml too? Excellent Misty. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028915#comment-14028915 ] Hadoop QA commented on HBASE-11316: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650004/HBASE-11316-2.patch against trunk revision . ATTACHMENT ID: 12650004 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+0 tests included{color}. The patch appears to be a documentation patch that doesn't require tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +which resides within a Store. The number of StoreFiles increases over time. Compaction is an operation which reduces the number of +StoreFile. Minor compactions do not drop (filter out) deletes or expired +versions, because of potential side effects. See and compaction policy. + hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java. + The maximum number of files which will be selected for a single minor + There are two different thread pools for compactions, one for large + major compactions. This approach alleviates some of the challenges of larger regions. +Stripe compaction is fully compatible with and works in + conjunction with either the ExploringCompactionPolicy or RatioBasedCompactionPolicy. It can be enabled for {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: {color:red}-1 core zombie tests{color}. There are 1 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/9753//console This message is automatically generated. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11316) Expand info about compactions beyond HBASE-11120
[ https://issues.apache.org/jira/browse/HBASE-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028853#comment-14028853 ] Misty Stanley-Jones commented on HBASE-11316: - File size shrunk because I rebased -- sorry. > Expand info about compactions beyond HBASE-11120 > > > Key: HBASE-11316 > URL: https://issues.apache.org/jira/browse/HBASE-11316 > Project: HBase > Issue Type: Bug > Components: Compaction, documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11316-1.patch, HBASE-11316-2.patch, > HBASE-11316.patch > > > Round 2 - expand info about the algorithms, talk about stripe compaction, and > talk more about configuration. Hopefully find some rules of thumb. -- This message was sent by Atlassian JIRA (v6.2#6252)