[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642682#comment-13642682 ] Hudson commented on HBASE-8299: --- Integrated in hbase-0.95 #163 (See [https://builds.apache.org/job/hbase-0.95/163/]) HBASE-8299 ExploringCompactionPolicy can get stuck in rare cases. (Revision 1475965) Result = FAILURE eclark : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ConstantSizeFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/EverythingPolicy.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ExplicitFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/GaussianFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/MockStoreFileGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/PerfTestCompactionPolicies.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SemiConstantSizeFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SinusoidalFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SpikyFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/StoreFileListGenerator.java ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642796#comment-13642796 ] Hudson commented on HBASE-8299: --- Integrated in hbase-0.95-on-hadoop2 #81 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/81/]) HBASE-8299 ExploringCompactionPolicy can get stuck in rare cases. (Revision 1475965) Result = FAILURE eclark : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ConstantSizeFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/EverythingPolicy.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ExplicitFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/GaussianFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/MockStoreFileGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/PerfTestCompactionPolicies.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SemiConstantSizeFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SinusoidalFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SpikyFileListGenerator.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/StoreFileListGenerator.java ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642831#comment-13642831 ] Hudson commented on HBASE-8299: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #511 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/511/]) HBASE-8299 ExploringCompactionPolicy can get stuck in rare cases. (Revision 1475966) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ConstantSizeFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/EverythingPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ExplicitFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/GaussianFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/MockStoreFileGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/PerfTestCompactionPolicies.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SemiConstantSizeFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SinusoidalFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SpikyFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/StoreFileListGenerator.java ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642581#comment-13642581 ] Hudson commented on HBASE-8299: --- Integrated in HBase-TRUNK #4080 (See [https://builds.apache.org/job/HBase-TRUNK/4080/]) HBASE-8299 ExploringCompactionPolicy can get stuck in rare cases. (Revision 1475966) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/ExploringCompactionPolicy.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/RatioBasedCompactionPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ConstantSizeFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/EverythingPolicy.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/ExplicitFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/GaussianFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/MockStoreFileGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/PerfTestCompactionPolicies.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SemiConstantSizeFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SinusoidalFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/SpikyFileListGenerator.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/StoreFileListGenerator.java ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640758#comment-13640758 ] Sergey Shelukhin commented on HBASE-8299: - +1... I assume the tests above are unrelated? ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640955#comment-13640955 ] Elliott Clark commented on HBASE-8299: -- yeah there was a trunk build break. Everything's back to normal. I'll check this in later today. Thanks for the review [~sershe] ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch, HBASE-8299-3.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639279#comment-13639279 ] Sergey Shelukhin commented on HBASE-8299: - bq. This class will search all possibilities for different unfinished bq. .subList(0, Math.max(0,candidateSelection.size() - 1 - comConf.getMinFilesToCompact())) subList's 2nd argument is exclusive, so -1 is not needed as far as I see. E.g. 5 files, 3 min files, will now remove [0, 1), leaving 4 files not 3. I think test is needed for this path. bq. //To change body of implemented methods use File | Settings | File Templates. Leftovers. There are long lines. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch, HBASE-8299-2.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639582#comment-13639582
]
Hadoop QA commented on HBASE-8299:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12580106/HBASE-8299-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 35 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5407//console
This message is automatically generated.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch,
HBASE-8299-2.patch, HBASE-8299-3.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638722#comment-13638722
]
Hadoop QA commented on HBASE-8299:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579959/HBASE-8299-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 35 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5389//console
This message is automatically generated.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch,
HBASE-8299-2.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636119#comment-13636119 ] Elliott Clark commented on HBASE-8299: -- bq.Would smallest be empty in this case ? Probably not, assuming that there is some set of files that can satisfy the other conditions on a compaction. See the previous discussion on why we want to skip some compactions. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636882#comment-13636882
]
Sergey Shelukhin commented on HBASE-8299:
-
Thanks, great testing.
Do you intend to commit all the policies e.g. EverythingPolicy?
Didn't review them yet assuming you would only commit the best one.
Ok, I actually reviewed one accidentally.
For NoStuckBestRatio, if it gets committed:
When not stuck, the compactions are only judged based on selection size and
file size, foundRatio and bestRatio are never compared.
When stuck, the reverse is true, it chooses best ratio without regard to
selection size or file size. Is it intended?
For both this and the NoStuckSmallest one:
{code}
if (files.size() 2) {
return true;
}
{code}
The min size sanity check should ensure it's 2 or more. Should it do
precondition check?
Also nit^2 in this function - total size computed inside and outside in the
caller, could just pass it in. Too bad Java doesn't have out/pointer parameters
;)
Nit^3 - the loop could actually just find the biggest file (it could be done in
the previous loop too), by definition it would have the biggest ratio. Doesn't
really matter.
There are some blank lines.
One test set was removed but nothing was added; need a couple new tests to
ensure it doesn't get broken?
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637002#comment-13637002
]
Elliott Clark commented on HBASE-8299:
--
bq.Do you intend to commit all the policies e.g. EverythingPolicy?
I don't think so. They are great to get a sense of what compactions could be
like. But they aren't really useful for any read users. So if anyone wanted
to use them as a benchmark then I would move those algorithms into the test
folders.
{quote}
When not stuck, the compactions are only judged based on selection size and
file size, foundRatio and bestRatio are never compared.
When stuck, the reverse is true, it chooses best ratio without regard to
selection size or file size. Is it intended?
{quote}
Yes this is intended. The thought was to keep the policy as close to
ExploringCompactionPolicy as possible. So if not stuck take the same set of
files that Exploring Compaction Policy would. If we are stuck take the set of
files that are the closest to being in ratio. This policy works really well if
there are giant spikes (see the last set of files created in the perf test).
bq.The min size sanity check should ensure it's 2 or more. Should it do
precondition check?
I'd much rather protect against a bug then try and prematurely optimize.
bq.Also nit^2 in this function - total size computed inside and outside in the
caller, could just pass it in. Too bad Java doesn't have out/pointer parameters
You brought this up last time. Without some profiling I'm not up for
optimizing this code. I don't think that it's needed at all. Readability is
worth a lot, especially in something that is as complex as our compactions have
become.
bq.Nit^3 - the loop could actually just find the biggest file
I'm not convinced that removing a division is worth the loss of clarity.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637016#comment-13637016
]
Sergey Shelukhin commented on HBASE-8299:
-
{quote} bq. The min size sanity check should ensure it's 2 or more. Should it
do precondition check? {quote}
bq. I'd much rather protect against a bug then try and prematurely optimize.
It's not optimization, just correctness, we never should be attempting to
compact one file here.
With other stuff in my mind less lines generally means more clarity.
+1 assuming the all of the policies will be removed except the good one.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637067#comment-13637067 ] Jean-Marc Spaggiari commented on HBASE-8299: bq. So if anyone wanted to use them as a benchmark then I would move those algorithms into the test folders. +1 with the idea of moving the algorithms into the test folders. Might be good to keep them for later uses. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637071#comment-13637071 ] Sergey Shelukhin commented on HBASE-8299: - yeah that also works ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635925#comment-13635925 ] Elliott Clark commented on HBASE-8299: -- Then running the perf tests gets this result: h2.Data ||Name||Max Files||Min Files||Ratio||IO||File Diff||IO Per File|| |EverythingPolicy|10|3|1.2|566916287|-113090|5012.97| |RandomPolicy|10|3|1.2|457296661|-109033|4194.11| |NoStuckLargestPolicy|10|3|1.2|30827212|-111271|277.05| |NoStuckBestRatioPolicy|10|3|1.2|25519685|-111439|229.00| |RatioBasedCompactionPolicy|10|3|1.2|25242372|-111622|226.14| |NoStuckSmallestPolicy|10|3|1.2|24200357|-111339|217.36| |ExploringCompactionPolicy|10|3|1.2|23526926|-110921|212.11| h2.Explanations * Random Policy just illustrates what the worst case is * Everything policy compacts all files all the time. * The No Stuck Smallest policy seems like the best all around pick for this. It won't get stuck because when it hits the storefile blocking limit it ignores the ratio. It then picks the smallest set of files to compact. * The No Stuck Best Ratio does similar except instead of trying to pick the smallest set of files it chooses a set of files that are close to being in ratio. This works really well on spiky load but really badly on sinusoidal work loads. h2.Testing [~jdcryans] put this patch up on a 0.94 cluster. He then started a random gets, while looping a bulk load job. For the old compaction algorithm we ended up with max average latency of 136ms. For the new algorithm that doesn't get stuck we ended up with a max average latency of 20ms. Lots of compactions were skipped. The compactions that were chosen all were what I would have expected. No major compactions were needed or triggered. h2.Plan So I'll clean up the code and make the NoStuckSmallestPolicy default. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635967#comment-13635967
]
Hadoop QA commented on HBASE-8299:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579464/HBASE-8299-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5353//console
This message is automatically generated.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636070#comment-13636070
]
Ted Yu commented on HBASE-8299:
---
NoStuckSmallestPolicy needs license header.
For NoStuckSmallestPolicy#applyCompactionPolicy():
{code}
+if (bestSelection.size() == 0 mightBeStuck) {
+ return new ArrayListStoreFile(smallest);
+}
{code}
When bestSelection.size() == 0 !mightBeStuck, we return bestSelection. Would
smallest be empty in this case ?
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch, HBASE-8299-1.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626913#comment-13626913 ] Sergey Shelukhin commented on HBASE-8299: - There will only be infinite loop (I assume you mean no compaction ever selected?) if there are no more files added. I think this is by design, compaction in this case doesn't make sense... if it doesn't make sense as minor compaction at one time, it still doesn't make sense later. For the above case getting stuck: 1) Why isn't major compaction enough to resolve it? 2) Another thing is that the above described pattern looks very uncommon. Imho we should optimize for common pattern... if we introduce something that can hurt normal use case by making unnecessary compactions at least it should be configurable and off by default. Btw, smallest allowed set of files in the 500 1 1 case is also 500 1 1. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626914#comment-13626914 ] Sergey Shelukhin commented on HBASE-8299: - The intuitive reason to compact 10 1 100 (for example) from [ 10 1 100 5000 1 ] is relative size to the 5000 file. Maybe that could be encoded in a heuristic. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627041#comment-13627041 ] Elliott Clark commented on HBASE-8299: -- bq.(I assume you mean no compaction ever selected?) Nope. I mean an infinite loop of compaction selection. If the number of store files doesn't change and you are over the high water mark then after a compaction is done another selection will be queued. So there have been problems in the real world with previous compaction selection algorithms not choosing files to compact. It is a bug if there are no files chosen when a compaction is queued. It's not as rare as you seem to think. Ask Stack about the half size file problem. This change only affects the algorith when it can't find any files sets in ratio and have need of a compaction. Just ignoring the need for a compaction is not an option. bq.The intuitive reason to compact 10 1 100 (for example) from [ 10 1 100 5000 1 ] is relative size to the 5000 file. Maybe that could be encoded in a heuristic. That's a completely different algorithm. If you want to write that then I would feel better doing that in a different issue. Right now we need to fix the policy that's currently there. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627133#comment-13627133 ] Sergey Shelukhin commented on HBASE-8299: - The compaction is not only requested when it's necessary. There's periodic compaction checker, and there's compaction when region is opened. needsCompaction also doesn't check ratio iirc, so number of files would be enough to call compaction selection. It doesn't mean there must be selection. In such cases with one large file and a few new memstore flush files that are not yet by themselves within the ratio, there must be no compaction. If something calls selection non-stop when there's no reasonable selection to be made it's a different problem. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627137#comment-13627137 ] Sergey Shelukhin commented on HBASE-8299: - if it's above blockingFileCount (you mean that by high watermark?) should the force flag be passed for this case? Then it can do no-ratio compaction. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627145#comment-13627145 ] Elliott Clark commented on HBASE-8299: -- bq.If something calls selection non-stop when there's no reasonable selection to be made it's a different problem. Exactly that's the odd thing not the normal. We shouldn't change the algorithm to protect a user from that. We should make sure the normal case is fine. The normal case is that compaction is needed because store says needsCompaction() = true. If there's some times that compactions are requested when not needed then we should fix that not patch around it here. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627150#comment-13627150 ] Sergey Shelukhin commented on HBASE-8299: - Current case definitely doesn't work like that, see needsComaction implementation. Its semantics is more like mayNeedCompaction, a cheap check, and request actually does the heavy lifting. And code expects request to not request anything - that can happen both due to logic of the policy, and due to coprocessor overrides, or concurrent compaction requests from coprocessors. That seems like reasonable logic for me, making it possible to call the method speculatively when region opens/memstore flushes/compaction finishes (for schemes that may require multiple compactions like level or stripe). Force flag (or method) can solve the forcing compaction issue here. Otherwise needsCompaction basically needs to be true to requestCompaction implementation (at least). ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627151#comment-13627151 ] Sergey Shelukhin commented on HBASE-8299: - *concurrent compaction requests, including from coprocessors ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627158#comment-13627158 ] Elliott Clark commented on HBASE-8299: -- I don't understand what you're looking for. You're asking to change the whole algorithm without specific requests. If you want to propose another algorithm for compaction selection then another issue would be best. As it stands now ExploringCompactionPolicy is broken and either needs to be changed so that it can't get stuck (done in this patch) or ExploringCompactionPolicy needs to be removed (breaking bulk loads again). ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13627177#comment-13627177 ] Sergey Shelukhin commented on HBASE-8299: - I get that it needs to be fixed, but this fix affects the cases beyond the ones it's looking to solve, so I am looking for another, or modified, fix. This fix changes the logic of requestCompaction to force compaction every time it's called when needsCompaction is true, which is different from current behavior (and that for old default policy) where needsCompaction is an inaccurate pre-check and real decision is done in requestCompaction, which is called speculatively form many place. Changing semantics of requestCompaction like that will cause bad compactions in normal cases where compaction is not needed. Including bad consequences with many compactions (see compaction checker - how it runs compactions). I am looking to avoid that. The ways to avoid it that can be used are: 1) Keep this patch but add force flag/method which will trigger this behavior of select compaction regardless of ratio. 2) Change semantics of needsCompaction and requestCompaction across the code so that the assumption this patch makes is actually true. 3) Change the heuristic to not get stuck in such case via other means. It seems that (1) is the simplest. ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626007#comment-13626007 ] Elliott Clark commented on HBASE-8299: -- If there are no files that are in ratio the ExploringCompactionPolicy can get stuck. [ 10 1 100 5000 1 ] ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[
https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626107#comment-13626107
]
Sergey Shelukhin commented on HBASE-8299:
-
{code}potentialMatchFiles.size() bestSelection.size(){code} is always true
initially and true for any larger potential match after that.
If no compaction is in ratio (which can be true in normal case), it would
always do the maximum possible compaction, wouldn't it?
50 1 1, minfiles 3 would major compact as far as I see, which seems wrong, it
should wait for more small files to appear.
ExploringCompactionPolicy can get stuck in rare cases.
--
Key: HBASE-8299
URL: https://issues.apache.org/jira/browse/HBASE-8299
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HBASE-8299-0.patch
If the files are very oddly sized then it's possible that
ExploringCompactionPolicy can get stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8299) ExploringCompactionPolicy can get stuck in rare cases.
[ https://issues.apache.org/jira/browse/HBASE-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13626147#comment-13626147 ] Elliott Clark commented on HBASE-8299: -- We are only requested to compact when there is an issue (user requested, or the number of files is getting close). The compaction should do as requested. If we don't do something then there's the chance of getting stuck in a loop. If in your same example the max storefiles is 3 there will be an infinite loop if we don't choose something. Maybe if we don't pass the ratio test it should take the smallest set of files ? ExploringCompactionPolicy can get stuck in rare cases. -- Key: HBASE-8299 URL: https://issues.apache.org/jira/browse/HBASE-8299 Project: HBase Issue Type: Bug Affects Versions: 0.95.1 Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-8299-0.patch If the files are very oddly sized then it's possible that ExploringCompactionPolicy can get stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
