[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664806#comment-13664806 ] Liu Shaohui commented on HBASE-7516: Any plan to backport this issue to 0.94? Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.95.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664823#comment-13664823 ] Sergey Shelukhin commented on HBASE-7516: - When we were discussing it there wasn't any; the final trunk design is definitely too intrusive... maybe this JIRA separately may make sense. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.95.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566633#comment-13566633 ] Ted Yu commented on HBASE-7516: --- @Jimmy: Can you attach the latest patch here ? Thanks Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566641#comment-13566641 ] Jimmy Xiang commented on HBASE-7516: trunk-7516_v3.patch is the latest patch. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566656#comment-13566656 ] Ted Yu commented on HBASE-7516: --- Can you rebase that patch ? Reversed (or previously applied) patch detected! Assume -R? [n] Apply anyway? [n] Skipping patch. 1 out of 1 hunk ignored -- saving rejects to file hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java.rej Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566723#comment-13566723
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567171/trunk-7516_v3.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4257//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch,
trunk-7516_v3.1.patch, trunk-7516_v3.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566728#comment-13566728 ] Ted Yu commented on HBASE-7516: --- Can you fix the findbugs warning ? You can use the following as baseline: https://builds.apache.org/job/PreCommit-HBASE-Build/4256/artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.xml Thanks Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566729#comment-13566729 ] Sergey Shelukhin commented on HBASE-7516: - +String className = family.getConfigurationValue(COMPACTION_POLICY_KEY); this is actually not necessary since the family setting is already included in HStore compound configuration Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567052#comment-13567052
]
Hadoop QA commented on HBASE-7516:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567212/trunk-7516_v3.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4262//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch,
trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567056#comment-13567056 ] Ted Yu commented on HBASE-7516: --- +1 Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567104#comment-13567104 ] Sergey Shelukhin commented on HBASE-7516: - +1 Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567265#comment-13567265 ] Hudson commented on HBASE-7516: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #385 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/385/]) HBASE-7516 Make compaction policy pluggable (Revision 1440737) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567350#comment-13567350 ] Hudson commented on HBASE-7516: --- Integrated in HBase-TRUNK #3830 (See [https://builds.apache.org/job/HBase-TRUNK/3830/]) HBASE-7516 Make compaction policy pluggable (Revision 1440737) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/DefaultCompactor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch, trunk-7516_v3.1.patch, trunk-7516_v3.2.patch, trunk-7516_v3.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13563507#comment-13563507 ] Jimmy Xiang commented on HBASE-7516: Since HBASE-7571 is in, I will rebase and post another patch. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562884#comment-13562884 ] Sergey Shelukhin commented on HBASE-7516: - latest patch looks good, except the metadata setting can now be cleaned up, HBASE-7571 went in Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562888#comment-13562888 ] Sergey Shelukhin commented on HBASE-7516: - btw, I don't know if this also needs a committer +1, my +1 is only half +1 :) Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562145#comment-13562145 ] Sergey Shelukhin commented on HBASE-7516: - btw, my review is on /r/ :) Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562153#comment-13562153 ] Jimmy Xiang commented on HBASE-7516: Thanks a lot for the review, I will address your comment in next patch. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562170#comment-13562170 ] Jimmy Xiang commented on HBASE-7516: I posted the second patch on RB: https://reviews.apache.org/r/9044/. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562179#comment-13562179 ] Sergey Shelukhin commented on HBASE-7516: - Can you also attach it here for Hadoop QA run? Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13562252#comment-13562252
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566412/trunk-7516_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4170//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch, trunk-7516.patch, trunk-7516_v2.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13558976#comment-13558976 ] Jimmy Xiang commented on HBASE-7516: I put my patch on RB https://reviews.apache.org/r/9044/. It is very different from Sergey's patch. Instead just making the configuration pluggable, I made the compactor part of the policy. I assume the configuration is tightly related to the compactor for each compaction algorithm. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13558985#comment-13558985
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12565819/trunk-7516.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestCatalogJanitor
org.apache.hadoop.hbase.regionserver.TestBlocksScanned
org.apache.hadoop.hbase.regionserver.TestResettingCounters
org.apache.hadoop.hbase.regionserver.TestScanWithBloomError
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
org.apache.hadoop.hbase.regionserver.TestHBase7051
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
org.apache.hadoop.hbase.filter.TestColumnPrefixFilter
org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection
org.apache.hadoop.hbase.client.TestIntraRowPagination
org.apache.hadoop.hbase.filter.TestDependentColumnFilter
org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking
org.apache.hadoop.hbase.regionserver.TestHRegionInfo
org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter
org.apache.hadoop.hbase.regionserver.TestKeepDeletes
org.apache.hadoop.hbase.regionserver.TestMinVersions
org.apache.hadoop.hbase.filter.TestFilter
org.apache.hadoop.hbase.regionserver.TestScanner
org.apache.hadoop.hbase.regionserver.TestWideScanner
org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4112//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch, trunk-7516.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13558992#comment-13558992 ] Sergey Shelukhin commented on HBASE-7516: - I took a cursory look; this patch makes compactor also pluggable and makes it return multiple files, and moves the default compaction policy stuff off the base class. If compactionPolicy returns, I wonder if (while Compactor is separate for reuse) it makes sense to make compactionPolicy interface simpler, and just let it compact. E.g. if Store has no default selection stuff anymore it doesn't make sense for it to get selection and feed it into compactor, right? Then, does it make sense to change CP interface to be called for all files post compaction instead of one file? I am not sure what use-cases it has, but there's no way to tell apart different compaction algorithms. My main concern is that this patch does not allow us to implement level compaction as described (see txt in level compaction issue). We can implement different algos which will allow for gradual compaction at the cost of IO, but not level algorithm, because that will remove the file ordering by seqNum, break the heuristic for determining mid-point for split, and other things. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch, trunk-7516.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13559151#comment-13559151
]
Sergey Shelukhin commented on HBASE-7516:
-
This code will also have to change depending on compation algo, in particular
for level.
{code}
// exclude all files older than the newest file we're currently
// compacting. this allows us to preserve contiguity (HBASE-2856)
StoreFile last = filesCompacting.get(filesCompacting.size() - 1);
int idx = candidates.indexOf(last);
Preconditions.checkArgument(idx != -1);
candidates.subList(0, idx + 1).clear();
}{code}
I will try to get a prototype patch for store files into the refactor jira by
tomorrow, tomorrow we can discuss the necessary changes
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch, trunk-7516.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556389#comment-13556389 ] Jimmy Xiang commented on HBASE-7516: When I filed this Jira, I meant to support pluggable compactions like level compaction. I think we don't need two levels of plugs for compaction, which may cause confusion. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556435#comment-13556435 ] Sergey Shelukhin commented on HBASE-7516: - Please see comments above... For level compactions even things like store file management will have to be pluggable. For many possible compaction algorithms on the other hand this change is sufficient. We can build on top of it for leveldb; I think that on top of this change, there should be an ability to replace CompactionPolicy/Compactor/etc. together. Including, for example for tiered algorithm, keeping standard compactor and replacing the policy. Or the other way around. First JIRA for that is HBASE-7603. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556933#comment-13556933 ] Sergey Shelukhin commented on HBASE-7516: - Attached notes in the parent, the first section (General notes) is relevant Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556934#comment-13556934 ] Sergey Shelukhin commented on HBASE-7516: - Er, in HBASE-7519 Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13555765#comment-13555765 ] Sergey Shelukhin commented on HBASE-7516: - Hi. Any objections to this? I will create another JIRA for preparation for LevelDB compactions. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553199#comment-13553199 ] Sergey Shelukhin commented on HBASE-7516: - *custom, not default, configuration. How do I get to edit comments? :) Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch, HBASE-7516-v2.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553325#comment-13553325
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564792/HBASE-7516-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestRegionSplitter
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4012//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch,
HBASE-7516-v2.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551732#comment-13551732 ] Sergey Shelukhin commented on HBASE-7516: - I have left some comments in the /r/ and will come up with updated patch today. With regard to level compaction and making more things pluggable. Initially I just thought I will make Compactor pluggable and change its interface a bit (e.g. make it produce multiple files). However, as I see now, with full leveldb compaction algorithm (with levels and overlapping ranges across levels), it will be insufficient - we'd need to change a some logic in HStore. Around compactions, obviously; storefiles list will also be suboptimal (all LevelDB algorithms work with levels, each of which except one have non-overlapping ranges; so selection from the list, while doable, is not pretty; various checks/logic relying on one sequence of files no longer make sense at all); also, StoreScanner file selection for scans will be different if we use ranges (ditto for gets). Other things may change too. We can either fully refactor StoreFile management out or HStore (tempting), or do minimum abstraction when we already have implementation in mind. In either case it will be in separate JIRA. I can try to do the former in another JIRA. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551744#comment-13551744 ] Sergey Shelukhin commented on HBASE-7516: - Link: http://leveldb.googlecode.com/svn/trunk/doc/impl.html Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551771#comment-13551771 ] Sergey Shelukhin commented on HBASE-7516: - Ok, there will be no patch today. I need to sleep on the issue some more, I don't like all these configuration classes. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548769#comment-13548769 ] Sergey Shelukhin commented on HBASE-7516: - TestLocalHBaseCluster appears to fail in all JIRAs. Will remove lines longer than 100 as part of next review iteration. Make compaction policy pluggable Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Assignee: Sergey Shelukhin Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch Currently, the compaction selection is pluggable. It will be great to make the compaction algorithm pluggable too so that we can implement and play with other compaction algorithms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547120#comment-13547120
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563792/HBASE-7516-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3931//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547179#comment-13547179
]
Jimmy Xiang commented on HBASE-7516:
Not a big deal, but wondering, can we call the existing policy
DefaultCompactionPolicy or something else, and make CompactionPolicy the actual
base?
{noformat}
- CompactionPolicy compactionPolicy;
+ CompactionPolicyBase? compactionPolicy;
{noformat}
The compactionPolicy has two abstract methods. How do you expect them to be
used? Where will the actually compaction algorithm logic be if
applyCompactionPolicy returns a CompactSelection? What's a CfgType?
{noformat}
+ protected abstract CfgType createCompactionConfiguration(Configuration
configuration,
+StoreConfiguration storeConfig);
+
+ protected abstract CompactSelection applyCompactionPolicy(CompactSelection
candidates)
+throws IOException;
{noformat}
Some javadoc will be very helpful. By the way, should we use RB for the next
patch?
[~sershe], thanks a lot for taking this issue.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable
[
https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547599#comment-13547599
]
Hadoop QA commented on HBASE-7516:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563870/HBASE-7516-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 8 zombie test(s):
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:833)
at
org.apache.hadoop.hbase.catalog.TestCatalogTracker.testServerNotRunningIOException(TestCatalogTracker.java:250)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3941//console
This message is automatically generated.
Make compaction policy pluggable
Key: HBASE-7516
URL: https://issues.apache.org/jira/browse/HBASE-7516
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Sergey Shelukhin
Attachments: HBASE-7516-v0.patch, HBASE-7516-v1.patch
Currently, the compaction selection is pluggable. It will be great to make
the compaction algorithm pluggable too so that we can implement and play with
other compaction algorithms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable and configurable per table
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546492#comment-13546492 ] Sergey Shelukhin commented on HBASE-7516: - See HBASE-7055, HBASE-7236 Make compaction policy pluggable and configurable per table --- Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7516) Make compaction policy pluggable and configurable per table
[ https://issues.apache.org/jira/browse/HBASE-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546503#comment-13546503 ] Sergey Shelukhin commented on HBASE-7516: - Oh, I see. Pluggable can be done by extracting part of HBASE-7055 patch; configurable per table would need HBASE-7236 Make compaction policy pluggable and configurable per table --- Key: HBASE-7516 URL: https://issues.apache.org/jira/browse/HBASE-7516 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
